Before the arrival of event streaming systems like Apache Kafka and other Pub/Sub architecture, data processing was typically handled with periodic batch jobs that couldn’t be handled in real-time.
Instead, Apache Kafka has streams of events: they are created to capture the time value of data and create push-based applications that kick in whenever an interesting event occurs in real time.

Being purpose-built for real-time log streaming, Apache Kafka is ideal for applications that need to:
- Native support for data/message playback
- Ability to partition messaging workloads as application requirements change
- Real-time streaming for data processing
- Reliable data exchanges between different components

Nowadays there are a lot of applications that can take a lot of advantages using Apache Kafka, and now we can understand the basic concepts of this technology.
Broker: It’s a single Kafka server that runs on the Java Virtual Machine.

Cluster: It’s a group of brokers that work together, it can contain one or more brokers. Every broker in a cluster has a unique ID.

Topics: They are the categories used to organize messages and each of them has a unique name across the entire Kafka cluster.
Topics are partitioned and replicated across brokers, using the partitions approach, which enables the parallelization of topics, enabling high message throughput.
Each partition has a specific offset and they are sliced into different segments
Every topic can have zero, one, or multiple consumers subscribing to that topic.

(image from: https://www.javatpoint.com/kafka-topics)

Producer: It’s an application that sends messages to a topic, those messages are distributed to partitions according to a mechanism such as key hashing.
A Kafka message consists of different elements:

• Key: It is optional in the Kafka message and it can be null. A key may be a string, number, or any object and then the key is serialized into binary format.
• Value: represents the content of the message and can also be null. The value format is arbitrary and is then also serialized into binary format.
• Compression Type: The compression type can be specified as part of the message. Options are none, gzip, lz4, snappy, and zstd
• Headers: There can be a list of optional Kafka message headers in the form of key-value pairs. It is common to add headers to specify metadata about the message, especially for tracing.
• Partition + Offset: Once a message is sent into a Kafka topic, it receives a partition number and an offset id. The combination of topic+partition+offset uniquely identifies the message
• Timestamp: It is added by the user or the system in the message.

Consumer: It’s an application that reads data from the Kafka cluster via a topic, It reads the data within each partition in a specific order starting from 0 and increasing the offset number.

Zookeeper: It’s a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.
It’s used to track cluster state, membership, and leadership

Now, these are the main concepts (there are a lot more) but now it’s time to see a little bit of code, creating topics, sending messages, and reading them.

Create a basic application

The first thing to do is to create a local environment with Docker, obviously that you can use Confluent Cloud which is perfect for most of your use cases. But for this example, I preferred to use a local docker.

docker-compose.yml

version: ‘3’
services:
  zookeeper:
    image: wurstmeister/zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 

To run the application you can launch this command:

docker-compose up -d

I have used the standard port for each service, now we can try to create our NodeJS app by installing a useful library: kafkajs

npm install kafkajs

After that, you can create a file called topic.js to create your topics

import {Kafka} from ‘kafkajs’

run();
async function run() {
  try {
    const kafka = new Kafka({
      “clientId”: “myapp”,
      “brokers”: [“localhost:9092”]
    })

    const admin = kafka.admin();
    await admin.connect();

    await admin.createTopics({
      “topics”: [{
        “topic”: “Users”,
        “numPartitions”: 2
      }]
     })
     console.log(“Created succesfully”);

     admin.disconnect();
     } catch(ex) {
       console.error(`Something bad happended ${ex}`)
     } finally{
       process.exit(0);
    }
 }

In this file, you are connecting to the local Kafka creating a topic "Users" with two partitions, one from the first letter starting with a-m, and the second from n-z.
To create a topic you can launch:

node topic.js

Now we can send a message into the topic created previously, to do it, you can create a file called producer.js like this:

import {Kafka} from ‘kafkajs’
const msg = process.argv[2];
run();
async function run() {
  try {
    const kafka = new Kafka({
      “clientId”: “myapp”,
      “brokers”: [“localhost:9092”]
     })
     const producer = kafka.producer();
     await producer.connect();
     const partition = msg[0] < “N” ? 0 : 1;
     const result = await producer.send({
       “topic”: “Users”,
       “messages”: [
        { 
          “value”: msg,
          “partition”: partition
        }
        ]
     })
     console.log(`Send succesfully! ${JSON.stringify(result)}`);
     producer.disconnect();
   } catch(ex) {
      console.error(`Something bad happended ${ex}`)
   } finally{
     process.exit(0);
   }
 }

In this file, we are connecting again to Kafka and we take from the CLI command the message argument passed. With that argument, we try to understand in which partition we can send the message and after that, we can send it passing the name of the topic and the message with the value and the partition.

You can send a message from your CLI in this way:

node producer.js test-message

Now we can consume the message with a consumer.
You can create a file called consumer.js with the following code:

import {Kafka} from ‘kafkajs’
run();
async function run() {
  try {
    const kafka = new Kafka({
      “clientId”: “myapp”,
      “brokers”: [“localhost:9092”]
    })
    const consumer = kafka.consumer({“groupId”: “test”});
    await consumer.connect();
    await consumer.subscribe({
      “topic”: “Users”,
      “fromBeginning”: true
    })
    await consumer.run({
       “eachMessage”: async result => {
          console.log(`Received message ${result.message.value} on partition ${result.partition}`)
       }
    })
  } catch(ex) {
     console.error(`Something bad happended ${ex}`)
  } finally{
 }
}

In this file, we are connecting to Kafka and we are reading from the beginning the topic "Users" from each partition.
To read messages you can launch this command:

node consumer.js

Here is an example of a message created and read!

Now you can add another consumer attached to the same topic, if you do it, you are parallelizing the way you are reading messages because a consumer read from one partition and the other one from the other partition.
A consumer can consume from multiple partitions, but a partition can be consumed by only one consumer, this is a concept of the Consumer group!

So, this can be your first application using Apache Kafka locally, it’s very simple and basic.
There are a lot of concepts about Kafka and more and more things to know and understand but with this simple example, you are seeing the basic concept of Kafka!

Pros and Cons

But what are the pros and cons of this architecture?
Pros:
- High performance
- Distributed system
- Event Driven, Kafka can be a Pub Sub system and a queue system
- Scalability
- Parallel Processing

Cons:
- The producer needs to explicit the correct partition to write and it can be a problem
- It’s complex to install configure and manage

There are a lot of advantages and use cases where you can apply Kafka, it's an outstanding architecture with a lot of useful components to solve your problems!

You can find the code of this article example in this repository: https://github.com/AlessandroMinoccheri/apache-kafka-basic

It’s important to share why they are so important for a team and what are their goals:

• Quality control: code reviews can increase the quality because we can find new solutions and improvements
• Sharing knowledge: It’s easy to end up in a situation where one developer deals with a certain part of the codebase because they’re most familiar with it. in short-period seems a win but it’s not for the future. When ownership is shared, teams become more motivated and autonomous.
• Increasing developer's knowledge: we can learn through code reviews, by asking questions about a piece of code that we don’t understand, or by sharing a different method to solve the problem

So after sharing why code reviews are so important it’s time to understand what are some tips and tricks for using them correctly and take advantage of them.

1 Keep Pull requests small

Keeping the Pull requests smaller as possible allows for the easiest review and, usually, more contribution and focus on it.
If you create a bug pull request with 120 files changed, 12000 lines added and 3500 lines removed it’s very difficult to find problems or better solutions. It’s totally a waste of time reviewing such a big pull request and it takes too many hours for doing it.
For that reason, it’s better to split the pull request into different pull requests with a specific code review.

A SmartBear study of a Cisco Systems programming team revealed that developers should review no more than 200 to 400 lines of code (LOC) at a time because the brain can only effectively process so much information at a time; beyond 400 LOC, the ability to find bugs, defects diminishes.

In practice, a review of 200–400 LOC over 60 to 90 minutes should yield 70–90% defect discovery. So, if 10 defects existed in the code, a properly conducted review would find between 7 and 9 of them.

2 Don’t review for more than 1 hour

Usually, it’s not a best practice to review code too quickly, on the other side you shouldn’t review for too long because performance starts to decrease after about 1 hour.
Our brain needs a break to refresh to have good performance and to review well the code.
It’s important to keep the focus on the review to discover and understand every single part of the pull request.
Doing a superficial review it’s not a good practice and decreases the credibility of the process for the whole team.

3 Add a clear description

To help reviewers to do their job it’s important to add a description to the pull request to share:

• which problem are you trying to solve
• why did you choose that solution
• doubts about something
• question about the implementation
• …

It’s also important to add a clear description for the future, for example, after some months you would like to understand why that piece of code was added. There are a lot of benefits to adding a clear description!

4 Use Pull Request template

Many tools, like GitHub, provide a template to create Pull Requests, in this way you have a standard and it’s easier to create a new one and review it. If you want to add the template.
GitHub you need to create a new file .github/pull_request_template.md

Github Documentation: https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/creating-a-pull-request-template-for-your-repository

Using a template allows the team to create a standard pull request to help each member of the team.

5 Number of reviewers

What is the correct number of accepted reviews to merge a pull request? There isn’t a specific rule again but it’s important to decide it in the team and try to understand what is the correct number.
Usually, you don’t need that every member of the team accepts the pull request, except for some critical features.
Sometimes one or two accepted reviews are good enough to merge the pull request.
Why not everyone?
Because you can create a bottleneck and your deployment can be blocked by some reviewer that didn’t see it already for other reasons.

6 Empathy

Empathy is an important skill during the review.
As a reviewer, you need to understand what the author is trying to solve, what was his thought, and why it’s implemented in this way.
You need to be kind when you comment on the pull request without being rude.
Don’t try to give a new solution, instead ask questions about the implementation like: “Is this the only way to solve the problem?” “It seems a little bit coupled with this code, what do you think? Is there a better way?”

If you liked a piece of code or you learn a new thing write a comment like: “this is perfect, I never thought about it” or “I didn’t know about this function! Amazing, it seems very useful!”
Try to foster a positive code review culture!

7 Delegate static analysis

Using a tool to automate static analysis checks it’s important because during reviews you need to focus on behaviors, business logic, and patterns you don’t have to focus on spaces, missing commas, or things that the static analysis tool can check.
All the automatable static analysis checks should be delegated to these tools during a CD/CI process.
This delegation can save you a lot of time and allows the team to focus on the business logic and the high-level approach to the implementation.

8 Run a retrospective

After a month or two of reviews it’s important to understand if the process is good or not and how can you improve it.
To do it I can suggest running a retrospective to understand it.
It’s important to find an external facilitator so all the team can discuss together what was good and what was not.
There is a lot of different type of retrospectives and the facilitator can use one of them.
Every month or two it’s important to run the retrospective, especially in the early stage of the adoption process, to improve it and create a feedback process for all the team.

Conclusion

Code reviews are a fantastic tool but you need to know how to manage them to get all the benefits.
There is always something to learn and improve in this process to discover with your team and discuss it.
I enjoy creating pull requests and reviewing the code of others, I always learn something and understand other points of view.
Never stop learning.

Adding an image with React is very simple and fast, this is an example:

import React from "react";
import imageToAdd from "./../assets/images/logo.png";
function YourComponent() {
   return <img src={imageToAdd} alt="Image" />;
}
export default YourComponent;

This works like a charm in a React project built using CRA or Vite.
but if your project has a custom bundler, created using Typescript + Webpack, with the code above you will receive this error:

Cannot find module  './../assets/images/logo.png'

The first time I saw that error, I thought “It’s a bug!”, but after searching and understanding typescript well, I discovered that receiving an error is the correct behavior.

By default in typescript, the module resolution resolves the import using only the files with extension: .ts .tsx or .d.ts and this is the reason why in the previous case the module couldn't be found.

So, how can we fix the problem?
To solve the problem, usually, you have to:

1. add a directory called types on your project's root

2. create inside of that folder a file called index.d.ts with the following content

declare module "*.jpg";
declare module "*.png";

N.B. the extension depends on the file type you are adding.

3. install the file-loader dependence using npm, yarn or pnpm

npm install --save-dev file-loader

4. update your webpack config file to use the file-loader module like this

...
{ test: /\\.(png|jp(e*)g|svg|gif)$/, use: ['file-loader'], }
...

Then, you can run your application, your build will succeed and your image will appear! 🎩

If you would like more about Server-Sent events, what it means, advantages and limits, you can read my previous article here.

To realize a Server-Sent Events in PHP we found a Symfony library symfony/mercure, I saw this library in some article on LinkedIn and it’s the proof that following the right people can be very important.

How to install it

We can use Flex to install it so, you can write into your CLI::

composer require mercure

After that, you can configure Mercure environment variables into the file: config/packages/mercure.yaml

mercure:
  hubs:
    default:
      url: https://mercure-hub.example.com/.well-known/mercure
      jwt:
        secret: '!ChangeMe!'
        publish: ['foo', 'https://example.com/foo']
        subscribe: ['bar', 'https://example.com/bar']
        algorithm: 'hmac.sha256'
        provider: 'My\Provider'
        factory: 'My\Factory'
        value: 'my.jwt'

Let me explain what are the values to set into the configuration file:

secret: the key to use to sign the JWT (all other options, besides algorithm, subscribe, and publish will be ignored)
publish: a list of topics to allow publishing when generating the JWT (only usable when secret or factory are provided)
subscribe: a list of topics to allow subscribing to when generating the JWT (only usable when secret, or factory are provided)
algorithm: The algorithm to use to sign the JWT (only usable when the secret is provided)
provider: The ID of service to call to provide the JWT (all other options will be ignored)
factory: The ID of service to call to create the JWT (all other options, besides subscribe, and publish will be ignored)
value: the raw JWT to use (all other options will be ignored)

Automatically inside your .env will be updated with environment variables.

If you are using a Symfony version less than 6, the YAML file doesn’t exist and you need only to configure your environment variables into the .env file like this:

MERCURE_URL=http://localhost:9090/.well-known/mercure
MERCURE_PUBLIC_URL=http://localhost:9090/.well-known/mercure
MERCURE_JWT_SECRET=”JWT-AuctionEngine”

Using a specific Docker container

You can use an official docker for your local environment, with a simple configuration like this:

version: "3.5"

services:
  mercure:
    container_name: mercure_sse_poc
    image: dunglas/mercure
    environment:
      SERVER_NAME: ':80'
      MERCURE_PUBLISHER_JWT_KEY: 'JWT-AuctionEngine'
      MERCURE_SUBSCRIBER_JWT_KEY: 'JWT-AuctionEngine'
      MERCURE_CORS_ALLOWED_ORIGINS: '*'
      MERCURE_EXTRA_DIRECTIVES: |-
      cors_origins "*"
      anonymous 1
    ports:
      — "9090:80"
      — "4433:443"

JWT_KEY must be the same as your environment variables, in this little example, I declare that it’s not necessary to be authenticated and you can use Mercure from all domains.
In production, I recommend restricting those parameters to avoid undesired problems.

How to use Symfony/Mercure

To send data to your client you need to instantiate a Mercure object called Update with 2 arguments: the topic name where you want to send your data, and the data encoded.
After that, you can publish your event using an object that implements HubInterface (for this example).
But symfony/mercure has already its implementation so you can use it without writing your implementation.
Here is a little example:

public function send(Request $request, HubInterface $hub): void
{
    $update = new Update(
       'channelname',
       json_encode([
           'foo' => 'bar'
       ], JSON_THROW_ON_ERROR)
    );
    $hub->publish($update);
}

The object Update has different arguments (directly from https://mercure.rocks/spec):

topic: The identifiers of the updated topic. It is RECOMMENDED to use an IRI as an identifier. If this name is present several times, the first occurrence is considered to be the canonical IRI of the topic, and other ones are considered to be alternate IRIs. The hub MUST dispatch this update to subscribers that are subscribed to both canonical or alternate IRIs.
data (optional): the content of the new version of this topic.
private (optional): if this name is set, the update MUST NOT be dispatched to subscribers not authorized to receive it. See authorization. It is recommended to set the value to on but it CAN contain any value including an empty string.
id (optional): the topic’s revision identifier: it will be used as the SSE’s id property. The provided id MUST NOT start with the # character. The provided id SHOULD be a valid IRI. If omitted, the hub MUST generate a valid IRI (RFC3987). A UUID (RFC4122) or a DID MAY be used. Alternatively, the hub MAY generate a relative URI composed of a fragment (starting with #). This is convenient to return an offset or a sequence that is unique for this hub. Even if provided, the hub MAY ignore the id provided by the client and generate its id.
type (optional): the SSE’s event property (a specific event type).
retry (optional): the SSE’s retry property (the reconnection time).

Frontend

The frontend client needs to subscribe to the topic, the following example is a piece of a React component:

const url = 'http://127.0.0.1:9090/.well-known/mercure?topic=auctions-1';
const eventSource = new EventSource(url);

eventSource.onmessage = event => {
  const results = JSON.parse(event.data);
  console.log(results);
}

You can use this code for example in a UseEffect function of React and you are telling that you want to subscribe to the topic auctions-1 and every event sent to that channel you will print it into the console to see what you have received and you can use those data for whatever you want.

What happens?

I have written a little example here to understand well what happens: https://github.com/AlessandroMinoccheri/sse-poc
Now if you clone the repository and follow the instructions to install the backend and frontend you can visit the page: http://localhost:8080/
You can see a simple form to submit a bid offer.

Now, if you open the browser network, select Fetch/XHR (for Chrome), and reload the page you can see a subscription request to a specific topic.

Then, if you insert a number into the text field and click the button to make a bid, you can see in the network tab a new POST request for the bid, but inside the previous request in the tab Events, you can see a new event sent by the server with new information!

So in the same subscription request, we can receive events for the server, for this reason, the Server-Sent events approach is considered a monodirectional communication because is the server that sends data to the client, and the client decides only to subscribe to topics.

If you make other bids you can see in the tab events all data sent from the server in the same HTPP connection.

Subscribe to multiple topics

Is it possible for the client to subscribe to multiple topics? Yes, absolutely, but how?
Well, you can easily do this thing into your frontend app (in this example using React):

const url = 'http://127.0.0.1:9090/.well-known/mercure?topic=auctions-1';
const eventSource = new EventSource(url);

eventSource.onmessage = event => {
  const results = JSON.parse(event.data);
  console.log(results);
}

const anotherUrl = 'http://127.0.0.1:9090/.well-known/mercure?topic=anotherTopic';
const anotherEventSource = new EventSource(anotherUrl);

anotherEventSource.onmessage = event => {
  const results = JSON.parse(event.data);
  console.log(results);
}

You can subscribe to many topics in this way because the server will send data to a specific topic.
You can use this approach to create a private topic for a single user, or group of users to send events only to them.

Asynchronous dispatching

You can write your implementation to dispatch asynchronous events, but this is not recommended because Mercure already sends them asynchronously.
But if you want, you can use Symfony Messenger to dispatch it by yourself.
Let’s see how the previous example could be with your own implementation:

public function send(Request $request, MessageBusInterface $customBus): void
{
   $update = new Update(
      'channelname',
      json_encode([
          'foo' => 'bar'
      ], JSON_THROW_ON_ERROR)
   );
   $customBus->publish($update);
}

As you can see the code doesn’t change, you can use a different implementation.
But remember that Mercure already sends events asynchronously so it depends on you if you want to maintain by yourself the dispatching system.

Use Case Examples

There are many examples of using the Server-Sent events approach.
Imagine that you need to build a stock option client, the frontend application can subscribe to topics to receive data from the server only when a stock quote changes its value.
Or if you need to build a Twitter feed update, the server will send data when to every user (with a custom topic each) with new tweets.
Recently I built a push notification system where clients receive information and updates from the server like the common push notifications app on your phone.

Conclusions

There are a lot of useful resources and examples, in my opinion using symfony/mercure library has simplified my life because in some minutes you can create a system ready to send events to the client without a lot of configurations and code to do.
Using the right tool can be important to the success of the project, but to know it you don’t have to start from the implementation.

Start understanding what is the real problem to solve, I usually do an EventStorming before coding because we need to know the domain and business logic.

After that, you can understand better if Server-Sent Events is the right approach or not.

The reason could be that WebSocket is used more, but not always is the ideal approach.
Sometimes Server-sent events approach is better to solve problems and create good applications.

It’s very important to understand differences, limits, and ideal cases for both to solve your problem with the right solution.
Sometimes we have a cognitive bias that doesn’t allow us to take the right decision.
Cognitive bias is a systematic error in thinking that occurs when people are processing and interpreting information in the world around them and affects the decisions and judgments that they make.

(Image from https://uxdesign.cc/how-to-improve-experience-design-by-managing-cognitive-biases-d7b360d35b0a)

So, for example, if we have solved a problem using WebSocket, next time with a similar problem we could think that, again, it’s the right approach without considering the big picture, all information about the problem to solve.
For that reason, understanding deeply bot approaches can help us to reduce the cognitive bias and make the best decision.

Let’s try to explain both approaches and their characteristics.

Websocket

It’s a computer communications protocol, that provides full-duplex communication channels over a single TCP connection (Wikipedia). The protocol was standardized in 2011.

The WebSocket approach is used in many contexts and projects because has many benefits and pros in its favor.

Bidirectional communication

The client and server can both send data to the other part.
The client creates a TCP connection to the server and keeps it open as long as needed.
Both server and client can close the connection easily whenever they want.
To exchange data there is a handshake process that verifies the communication, and if it’s all ok, they can exchange data using the custom WebSocket protocol.

(Image from: https://www.onlyfullstack.com/polling-vs-server-sent-events-vs-websocket/)

Advantages

- Bidirectional communication in real-time, connections can both send and receive data from the browser(example: chat).
- Generally don’t use XmlHttpRequest and headers are not sent every time, so it reduces the expensive data loads being sent to the server.
- Can transmit both binary data and UTF-8 format.

Limits

- When connections are terminated, WebSocket doesn’t automatically recover them, you need to implement a reconnection system by yourself.
- Browsers older than 2011 don’t support WebSocket connections.
- Some enterprise firewalls with packet inspection have trouble dealing with WebSocket.
- It cannot be placed behind container-based authentication methods with header-based security, it has to be query parameters (or hopefully, a token-based authentication where the token is established outside the WebSocket path).

Ideal cases

- Chat: both client and server exchange data to send and receive messages
- Multiplayer games
- Collaborative editing/coding
- Social feeds
- Sport updates
- … a lot more …

Server-Sent Events

It’s a server push technology enabling a client to receive automatic updates from a server via an HTTP connection.
It was born on September 1, 2006, so it’s been a while since it’s available.
It provides a memory-efficient implementation of XHR streaming.
Unlike a raw XHR connection, which buffers the full received response until the connection is dropped, an SSE connection can discard processed messages without accumulating all of them in memory.

Monodirectional communication

The client sends a request to subscribe to a specific channel and the server will send event/data into that request, so only the server can send data and the client receive automatically those events in the same HTTP connection.

(Image from: https://www.onlyfullstack.com/polling-vs-server-sent-events-vs-websocket/)

Advantages

- Transported over simple HTTP instead of a custom protocol
- Simpler protocol
- Built-in support for re-connection and event-id
- No troubles with corporate firewalls doing packet inspection

Limits

- It’s limited to UTF-8 and does not support binary data.
- When not used over HTTP/2, Server-Sent events approach is subject to limitations with regard to the maximum number of open connections. This can be especially painful when opening various tabs as the limit is per browser and set to a very low number (6).
- It’s mono-directional
- It doesn’t have native browser support. However, there are available workaround with poly-fill (replicate an API) in Javascript that simulates the SSE functionality to solve this issue. All modern browsers support server-sent events: Firefox 6+, Google Chrome 6+, Opera 11.5+, Safari 5+, Microsoft Edge 79+.

Ideal cases

- Stock client updates: the client receives updates from the server when a stock changes its value
- Twitter feed updates: the client receives a new tweet every time one is sent
- push notifications

Conclusion

In conclusion, both approaches are valid and can be used to build specific software.
Websocket is more used and can be used also to develop an application that can use Server-sent events.
Server-sent Events have fewer ideal cases but for them, it’s very useful with less configuration and less time to create the updating system.

I have used both of them and no one is better than another, it depends on the specific context and what you need at that moment.

Knowing WebSocket and Server-sent events can help you in deciding which approach is better for every application.

For this reason for me, it’s important to test my Github Actions without committing something because it’s not necessary.

Photo by https://www.giuseppemaccario.com/it/wp-content/uploads/2021/10/Continuous-Deployment-con-GitHub-Actions.png

I found this library perfect for testing Github Actions: act.

This library is very useful and easy to use, you can run locally your Github Actions and you have fast feedback on your workflows.

How does it work?

Act simulates the actions we would like to perform in the container of our Github Actions.
It creates a container capable of performing all the steps by properly simulating all the steps.
So we can see all the steps of the Github Actions and their results through simple instruction.

Installation

If you are using Homebrew (Linux, macOS) you can launch this command from your CLI:

brew install act

If you are not using Homebrew I recommend checking the Installation section inside the README file of the project because there are many ways and options to add it.

How to use it

To run the tool you need to write into your CLI:

act

This command runs the default (`push`) event.
When you run for the first time act, it will ask you to choose an image to be used as default and it will save that information inside the file ~/.actrc that contains configuration flags.

If you want to run a specific event you can launch:

act pull_request

For the specific event, I mean Github Events: https://docs.github.com/en/developers/webhooks-and-events/events/github-event-types

To run a specific job you can launch:

act -j your_job

You can also launch command in dry mode like this:

act -n

Environment variables and Secret variables

Secrets and environment variables saved in GitHub can’t be used by act, but you can pass them and define them locally.

Secrets

act -s KEY=VALUE — To pass secret from CLI
act — secret-file my.secrets — load secrets values from local.secrets file.

Environment variables

act -n KEY=VALUE — To pass environment variable from CLI
act — env-file my.env — load environment variables from local.env file.

Use Case

I have used act many times in these months because I created many Github actions for my projects.
Every time that I change something I tried the action with act.

So, for me, it’s better to test it locally to save a lot of time.
The process could be:
- create your first Github Actions
- launch act to test changes
- improve or fix your Github Actions
- launch act to test changes

And then you can iterate until you are satisfied with your Github Action.

Example

I have created a simple example to test act and you can find it following this link: https://github.com/AlessandroMinoccheri/act-example

It’s a simple example to test act and it took me 2 minutes to install and launch act.
It’s very useful to see the output of your GitHub Actions to understand if it’s all ok or not.

Limit

There are a few limitations by using act (https://github.com/nektos/act#known-issues):
- Non-Linux Runners: act can works for Linux runners, not for the others, but it’s a Docker limitation on your workstation.
You can work around this by setting DOCKER_HOST before running act, with e.g:

export DOCKER_HOST=$(docker context inspect — format ‘{{.Endpoints.docker.Host}}’)

- Default images do not contain intentionally all the tools that GitHub Actions offers by default in their runners.

Conclusion

In my opinion, act can save you a lot of time to test your Github Actions without launching them every time.
It’s very easy and fast to use.
Usually, when you are running act and all it’s fine your Github Actions will not have problems.

Building a career it’s difficult sometimes because you don’t know what you want next or what you want to achieve.
For this reason, I take some days to understand it and try to design the next steps for my career.
Sometimes I need to change a lot of things but it’s important to follow our dream.

Remember that a career is more than a job and every company (or most of them) is hiring developers.

So my goals for 2021 were:
- read more than 30 books
- learn React
- 5 meetup as a speaker
- write at least 5 articles
- release phparkitect

Read more than 30 books

I have read a lot of articles about:
“You need to read at least 20 books, 30 books 50 books..”

I have taken this advice and I set the goal of 30 books for 2021.
In the first three months, I read about 8 books and I was really happy about it.
But I decided to think about this behavior and what I’m trying to achieve in 2021.

My goal is not to read 30 books, my goal is to improve my skill, get a new point of view. So my goal is to learn not to read books only to check from my list this goal.

I decided to stop for a moment and change this behavior.
After that, I take my books and I continue to reading them more slowly, getting notes and thinking about what they try to share.
I have created a template in Notion to track my books and notes inside.
I started to create a visual mapping with Miro and at the end of the year I don’t achieve my goal of 30 books, I read 18 books but I am really satisfied with what I have learned in a year.

So it’s time to share the 3 best books that I have read this year:
- Antifragile
- Startup, Scaleup, Screwup
- The goal

Learn React

Another goal for my 2021 was to learn React, and to learn React I meant to write a single page application with components and test obviously.
I have reached the goal before June following some articles and tutorials.
I did an application useful to manage my bank account.

I started watching some tutorials and create my first react Kata project.
A Kata is a project to learn something new without thinking about the idea. You can try new approaches, frameworks, and languages.

My first kata project with React was a simple to-do list where I had to create a list, edit it and view it.
After this, I have complicated the to-do list project with some features to understand well hooks and best practices.

5 meetup as a speaker

In 2021 I did exactly 5 talks:
- PugRomagna 2 talks about hexagonal architecture in Symfony and another one about smart working
- JUG Milano about hexagonal architecture in Kotlin
- Ticino Software Craft about hexagonal architecture in Kotlin
- PugRoma about hexagonal architecture

Speaking activities are moments where you share something with other people but you are learning at the same time because you need to study concepts deeply to explain them to others.
It’s important to open your mind to questions and different points of view.
Every time that I did a talk I learned something and I met a lot of clever developers, I recommend to all the developers to try to share their knowledge with others because it’s a moment to network and learn together every time.

Write at least 5 articles

In 2021 I wrote
- Why you should deploy on Friday without fear
- How dotfiles can save you a lot of time
- Push and publish Docker images with GitHub Actions
- Learning a new programming language
- PHPArkitect: Put your architectural rules under test!

It was very interesting to write these articles because explaining to others in a fluent and clear article sometimes could be difficult.
I tried to write small articles and answer all their questions.
Sometimes people contact me in private asking for advice or doubts about the article and their projects, and I am very happy to answer them and ask for a different point of view if it’s possible.

Release PHParkitect

The last goal was to release PHPArkitect in 2021.
It’s a library that helps you to keep your PHP codebase coherent and solid, by permitting you to add some architectural constraint check to your workflow.
The team is made up of a couple of people and I am one of them.
We developed the tool all year and we released it to all the world in May.
Nowadays we are working to improve the library day by day and at the moment we are using it in some projects.
I think that there are a lot of features to be done to improve the library.

Feel free to try it and give me feedback, please!

Goals 2022

For 2022 I take some hours and I walk alone to understand what I would like to achieve next year.
Learning sometimes could be easy but focusing on the right things it’s difficult.

I thought about my career path and what I can do in the next years and I understand that this year I would like to:

- learn typescript because I think that it could be a good solution for many projects
- read books that can help me to improve my hard skill and soft skill. I wrote a list of books that can help me, and these are the first two books that I will read: Clean Craftsmanship, Learning Domain-Driven Design
- do at least 3 talks to continue my career as a speaker
- write at least 10 articles to share my knowledge and to understand deeply some concepts to share

At the moment these are my goals for 2022, I have scheduled some moments to check every point, and write down a path to achieve them during the year.
It’s important not only to understand your goal but it’s important to measure them and create a path to understand how to achieve them.

To do it you need to ask yourself:
- What is the definition of done to understand if this goal is achieved or not?
- How can I achieve this goal?

Answering these questions you can understand well your next steps to improve yourself and your career.

Creating your career it’s one of the most difficult things to do because it depends on you usually.
You need to understand where do you want to go and what are next steps to achieve your goals.

It’s a never-ending path but could be amazing, it depends on you!

I have seen a lot of projects in my career with many different types of rules.
Other examples could be when a team tries to apply the hexagonal architecture and inside the domain, you shouldn’t call external libraries or framework classes (there are exceptions but this is out of scope).

Usually, a team shares those rules during retrospective or dedicated moments to improve the software and to establish rules to follow.
But after some months usually, those rules could be violated by new members of the team or someone that doesn’t remember them.
So the next step is to write down into a README file or in a file all the rules but again they can be violated.

Another good solution is to write ADR: Architectural Decision Records that explain the choice and the reason, but still, they can guarantee that will not be violated.

You need a tool to validate your architectural rules.
PHPArkitect can do it for you.

PHPArkitect is a tool that helps you to keep your PHP codebase coherent and solid, by permitting you to add some architectural constraint check to your workflow.
It was born in 2020 to satisfy the need to respect architectural rules in various projects.
In Java there is a similar tool from which we took inspiration: archunit.

Now PHPArkitect is public and you can find it here

Which problem is trying to solve PHPArkitect?

As I explained before, PHPArkitect was born to help teams to define architectural rules in a project.
With this tool, you can add a new step with it into your continuous integration and you have all your architectural rules in a file, so new developers can understand easily the constraints of the project.

How to install it?

To install PHPArkitect you can use composer launching this command into your CLI:

composer require — dev phparkitect/phparkitect

You can also use a .phar file because you can have some conflicts with one or more of Phparkitect’s dependencies.
The Phar can be downloaded from GitHub:

wget https://github.com/phparkitect/arkitect/releases/latest/download/phparkitect.phar
chmod +x phparkitect.phar
./phparkitect.phar check

How can you use it?

To use PHPArkitect it’s necessary to create a file inside your root called phparkitect.php with your rules written inside it like this:

<?php

declare(strict_types=1);

use Arkitect\ClassSet;
use Arkitect\CLI\Config;
use Arkitect\Expression\ForClasses\HaveNameMatching;
use Arkitect\Expression\ForClasses\ResideInOneOfTheseNamespaces;
use Arkitect\Rules\Rule;

return static function (Config $config): void {
 $classSet = ClassSet::fromDir(__DIR__ . ‘/src’);
 $r1 = Rule::allClasses()
 ->that(new ResideInOneOfTheseNamespaces(‘App\Order\Infrastructure\Controller’))
 ->should(new HaveNameMatching(‘*Controller’))
 ->because(“we want uniform naming”);

$config->add($classSet, $r1);
};

And now you can launch from your CLI the command:

./vendor/bin/phparkitect check

If you want to use a different name for your file you can specify it when you launch the command like this:

./vendor/bin/phparkitect check — config=/project/yourConfigFile.php

What can you do with PHPArkitect?

With this tool, you can write your own rules for your project.
At the moment you can check if a class:

- depends on a namespace
- extends another class
- not extend another class
- have a name matching a pattern
- not have a name matching a pattern
- implements an interface
- not implement an interface
- depends on a namespace
- don’t have dependency outside a namespace
- reside in a namespace
- not reside in a namespace

Let’s make some examples:

Assume that if you want those classes inside a namespace that doesn’t depend on classes outside that namespace you can write this rule:

$rule = Rule::allClasses()
 ->that(new ResideInOneOfTheseNamespaces(‘App\Catalog\Domain’))
 ->should(new NotHaveDependencyOutsideNamespace(‘App\Catalog\Domain’))
 ->because(‘domain should not have external dependencies’);

If you want to be sure that classes inside a namespace are following certain naming rules you write a rule like this:

$rule = Rule::allClasses()
 ->that(new ResideInOneOfTheseNamespaces(‘App\Order\Infrastructure\Controller’))
 ->should(new HaveNameMatching(‘*Controller’))
 ->because(“we want uniform naming”);

If you want to be sure that all classes that extend a class are following certain naming rules you write a rule like this:

$rule = Rule::allClasses()
 ->that(new Extend(AbstractType::class))
 ->should(new HaveNameMatching(‘*Type’))
 ->because(‘we want uniform form type naming’);

You can also exclude some classes from the rules for example:

$rule = Rule::allClasses()
 ->exclude([‘App\Order\Application\Service\Foo’])
 ->that(new ResideInOneOfTheseNamespaces(‘App\Order\Application\Service’))
 ->should(new HaveNameMatching(‘*Service’))
 ->because(“we want uniform naming”);

How to contribute

PHPArkitect it’s young at the moment and a lot of things could be developed.
If you are interested you can:
- try the tool
- improve the documentation
- try to write your own rule
- open pull requests to fix bugs or to discuss new features with a draft
- open issues to ask something or to reveal a bug

You can help with a little effort to improve this project and help other developers.

Well in my opinion there isn’t the perfect recipe to learn a new programming language because it depends principally on how the developer can learn new things.
Some people need to learn from books or others that prefer coding directly etc.

SHU HA RI

Before going deep into the argument, I would like to share the concept of SHU-HA-RI, because I think that it’s important to know when you start to learn something:

Shu-ha-ri is a concept of Japanese martial arts and describes the stages from learning to mastery, therefore a path that involves the presence of a teacher accompanying a student.
This concept was taken into Toyota to indicate the skills development model.

(image from: https://leadagilesolutions.com.au/a-mental-model-for-the-phases-of-mastery/)

Shu-Ha-Ri is a way of thinking about how you learn a technique and can be divided into three steps:

* Step one: Shu. The Sensei (the coach) protects the student by showing how to do it, without leaving room for errors and distortions. The practice must be performed as per the manual.
* Step Two: Ha. The student, independently, applies the practices respecting the standard. Here the student will be wrong, but he will understand even more.
* Step Three: Ri. The student applies the practice in complete autonomy, with strong creativity. He has understood the spirit and no longer feels “bound” by the rules. He might even revise the rules themselves.

This approach can be used in many different scopes: music, martial art, coding…

Usually, when I would like to learn a new programming language I try to follow the Shu-ha-ri approach trying to divide my learning time into different steps:

* Read the documentation
* Watch videos
* Follow people on Twitter that shares context about that language
* Find a mentor
* Learning by Doing (kata)
* Contributing to open-source projects
* Teach What You Learn
* Be Curious

Read the documentation

Sometimes it’s very annoying to read the documentation of something but don’t under evaluate this step because reading the documentation is the first approach to a new language, to understand how it works, how you can start, what you need in terms of software, tools, libraries.
Reading the “Get started” chapter, usually, helps you to enter into a new world and for me, it’s fantastic to see new things and how their work.

You don’t have to read all the documentation, my advice it’s to read the first parts, to understand how can you start to write something and step by step learn new things.
Sometimes the documentation is not updated or is superficial, feel free to open a pull request or contact the support to improve that part.
This is another good step to participate in something and learn new things.

Watch videos

Nowadays there are a lot of videos about all things.
So it’s very common to find developers that share videos about the language that you would like to learn.
I usually search for many videos or playlists and watch the first minutes to understand if the level of the video is correct or not for me at that moment.
For example, six months ago I studied for a new language and I found an amazing video by an advocate of that language and I stopped that after 3 minutes because the level was too high for me at that moment.
I marked the video as “Watch later” and after two months I saw all the videos understanding all the things.
It depends on your level and skills to find the best videos for that moment for you but believe me, you have a lot of options on the web nowadays to find the best video for you!

Follow people on Twitter that shares context about that language

Twitter is a social network where you can find a lot of developers that share powerful context about everything.
My advice is to search for some developers that frequently share interesting content, and follow them.
The next step is to interact with them to ask something to clarify your doubts.
I have found a lot of kind developers that help me a lot and vice versa.

Find a mentor

Finding a mentor sometimes seems difficult and impossible, but I can guarantee that it could be very easy.
When I have to find a mentor, usually I am thinking of my colleagues or ex-colleagues first because I know already their skills.
If no one has the required skills I am searching on Twitter for example or LinkedIn or participating in community meetings.
When you will find your mentor it’s important to share with him your goals to set up your expectations.
With him, you can establish a study plan to archive your goals.
Once you have specified those things you need to schedule a meeting every one or two weeks with your mentor to check your progress.
Don’t forget to ask a lot of things and thoughts in your mind to understand well what you are doing.

Learning by Doing

Learning by doing it’s one of my favorite steps because I love to touch directly the code and find problems, things not clear and interesting points of view about this new language.
It’s very important to practice a lot to become more confident with the new language.
I usually create Kata projects to make a lot of practice.
Kata means “way of doing” and they are projects that you create yourself to learn something: new coding language, different approaches, architectures, etc.
You can create a new little project and experiment with what you would like to explore.
There are a lot of Kata projects and you can create your own, this is a list of possible kata:
https://project-awesome.org/gamontal/awesome-katas

For example. I did twenty times the Tic-tac-toe game because every time I tried new things and a new way of doing it.

Contributing to open-source projects

Contributing to open-source projects could be another good solution to learn something about what you are studying, you can always ask questions creating issues or pull requests and you can try to create your first pull request starting from fixing some part of the documentation for example.
Open source projects can generate a lot of value and a new strategy to implement something.
You can write to contributors and maintainers to learn new things every day.
Don’t be scared to try something, open source can open to you a lot of doors.

Teach What You Learn

After studying something it’s important to share your knowledge with other developers. Try to teach what you have learned through a little talk with a few people or you can find a local meetup to try your new skills.
When you try to explain something to others you can realize if you know the argument or not, you can realize what you can study and see things from another point of view by answering questions during your talk.
Every time that I did a talk, after that, there were a lot of questions and we discussed those, understanding new approaches.

Be Curious

Every day you can learn something new, there are a lot of resources nowadays.
Now it’s up to you what you want to learn from today.
Be curious about new approaches, technologies, and all the things about your work.
Curiosity helps a developer with learning in two ways. First, it keeps you motivated during learning so you don’t get bored. Second, it highlights new learning opportunities that you would overlook.
Curiosity is what makes great engineers stand out from regular ones.

Usually, when I want to publish my docker images to DockerHub, I need to do it manually by the command line, like this:

Building the image

docker image build -t organization/project:0.1.0 .

Publishing to DockerHub

docker push organization/project:0.1.0

It’s not a lot of work, but every time you fix or add a new feature you need to remember to build a new image and publish it.
Usually, I try to avoid manual operations because human error is possible, and automating what is repetitive for me is a best practice everywhere.

So for this reason, in one open-source project Arkitect where I’m contributing nowadays, we have a Dockerfile that needs to be published every time there is a push on master, or a new release comes out.

I can build and publish the Docker image manually every time, but I prefer to avoid this and I have tried exploring GitHub Actions to automate this process.

Exploring GitHub’s actions to automate the process of publishing a docker image to DockerHub was interesting because I found a lot of other interesting GitHub actions and many projects that do the automation that I like.

First of all, I have inserted inside my GitHub project into Settings->Secrets, two important repository secrets:
- DOCKERHUB_USERNAME: this is your username on Dockerhub or the name of your organization
- DOCKERHUB_TOKEN: this is the token and you can get it going on DockerHub in Account Settings->Security. Here you can generate a new Access Token. You can take the value and put it on GitHub.

Next step, I have created a file for the workflow inside GitHub and named it docker-publish.yml

The file is something like this:

name: Arkitect
on:
  push:
    branches:
      — '*'
    tags:
      — '*'
 pull_request:

jobs:
  publish_docker_images:
    runs-on: ubuntu-latest

    steps:
    — name: Checkout
      uses: actions/checkout@v2
    — name: Docker meta
      id: meta
      uses: crazy-max/ghaction-docker-meta@v2
      with:
        images: phparkitect/phparkitect
        tags: |
          type=raw,value=latest,enable=${{ endsWith(GitHub.ref, 'master') }}
          type=ref,event=tag
        flavor: |
          latest=false
     — name: Login to DockerHub
       if: GitHub.event_name != 'pull_request'
       uses: docker/login-action@v1
       with:
         username: ${{ secrets.DOCKERHUB_USERNAME }}
         password: ${{ secrets.DOCKERHUB_TOKEN }}
     — name: Build and push
       uses: docker/build-push-action@v2
       with:
         context: .
         push: ${{ GitHub.event_name != 'pull_request' }}
         tags: ${{ steps.meta.outputs.tags }}
         labels: ${{ steps.meta.outputs.labels }}

But it’s not enough because this build will only start if the tests pass, so I have moved the content of this file inside my CI process to another workflow called: build.yml. This is the test suite.

So we need to create a new job that depends on the test job, thanks to the keyword “needs”, like this:

name: Arkitect

on:
  push:
    branches:
      — '*'
    tags:
      — '*'
    pull_request:

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
    — uses: actions/checkout@v2

    - name: Install PHP
      run : //stuff to install PHP

    - name: Test
      run: ./bin/phpunit

  publish_docker_images:
    needs: build
    runs-on: ubuntu-latest

    if: GitHub.ref == 'refs/heads/master' || GitHub.event_name == 'release'
    steps:
    — name: Checkout
      uses: actions/checkout@v2
    — name: Docker meta
      id: meta
      uses: crazy-max/ghaction-docker-meta@v2
      with:
        images: phparkitect/phparkitect
        tags: |
          type=raw,value=latest,enable=${{ endsWith(GitHub.ref, ‘master’) }}
          type=ref,event=tag
        flavor: |
          latest=false
    — name: Login to DockerHub
      if: GitHub.event_name != 'pull_request'
      uses: docker/login-action@v1
      with:
        username: ${{ secrets.DOCKERHUB_USERNAME }}
        password: ${{ secrets.DOCKERHUB_TOKEN }}
    — name: Build and push
     uses: docker/build-push-action@v2
     with:
       context: .
       push: ${{ GitHub.event_name != 'pull_request' }}
       tags: ${{ steps.meta.outputs.tags }}
       labels: ${{ steps.meta.outputs.labels }}

In this example, as I said before, I have created a new job that depends on the job named “build”.
In this way, if the job named “build” fails, I don’t create a new docker image, because I want to create it only if the tests pass.

I have used these GitHub Actions:
crazy-max/ghaction-docker-meta@v2 : it extracts metadata (tags, labels) for Docker.
docker/build-push-action@v2: it builds and pushes Docker images with Buildx with the full support of the features provided by Moby BuildKit builder toolkit.

A condition that I have added to my docker push job is:

if: GitHub.ref == 'refs/heads/master' || GitHub.event_name == 'release'

Because I want to execute this job only:
- when there is a push on master
- when a new tag is created

This is necessary for me because I don’t want to create a new docker image every time a pull request is created.

When there is a push on master the workflow, create a new docker image with the tag “latest”.
When a new tag is released, the workflow creates a new docker image, with the tag equal to the tag of the project and the tag “latest” is recreated.
In this way, I am sure to publish the last version of the docker image every time with new features or bugs fixed.

In conclusion, using GitHub Actions saves me a lot of time and repetitive work every time I need to publish my docker image for my project.
There are a lot of other processes that can be automated that I would like to explore and try in my projects, so as soon as possible I will publish other articles about this argument.