In the dynamic world of Artificial Intelligence, the tools and concepts we use are continually evolving. Language models, an integral part of this landscape, have grown in complexity and capability, resulting in the creation of Large Language Models (LLMs) such as GPT-4. Alongside LLMs, the concept of semantic embeddings has emerged, unlocking new ways to represent and utilize information. This article serves as a guide, detailing the essentials of LLMs and embeddings, and showcasing how they can be leveraged to extend the power of AI models, providing them with a form of unbounded long-term memory.

What is a Large Language Model (LLM)?

A Large Language Model is a type of AI model that is designed to generate human-like text. These models are trained on vast amounts of text data, enabling them to predict the next word in a sentence, given the context of the previous words. The “largeness” in LLMs refers to both the training data’s size and the model’s complexity, which often involves billions, or even trillions, of parameters (Each of the individual connections inside of the model). GPT-4, for example, leverages transformer-based architectures to process and generate text in a contextually and semantically rich way.

Thanks to the expansive and high-quality training data they employ, these models have acquired a certain level of problem-solving and reasoning capabilities. These skills are inherited from the vast array of language and literature they’ve been trained on. Consequently, they offer innovative ways to process and comprehend documents and data. Moreover, they provide more user-friendly and intuitive interfaces for human interaction.

What does mean to “train” a model?

Training a model, such as GPT-4, is an intricate and resource-intensive process. Drawing inspiration from the workings of biological brains, a large AI model is a network comprised of millions of nodes (or artificial neurons), connected by billions of weighted links. The training process necessitates the calculation of a specific set of weights to make the neural network function in the desired manner.

To illustrate, the fundamental objective of an LLM like GPT-4 is, quite simply, to predict the most sensible next word given a piece of text. To accomplish this, OpenAI engineers utilized a substantial corpus of text examples, presenting each document to the network and systematically adjusting the weights until the correct words began to surface.

Training a large language model is a remarkably complex undertaking, necessitating months of computation on specialized hardware and finely honed algorithms. Hence, it’s not a venture that just any organization can embark upon. To harness the power of LLMs using your own data — a prevalent use case — it’s more common not to train a custom model, but to utilize an existing model coupled with a database of embedding-indexed data.

What are the Embeddings?

Embeddings are numerical representations that encapsulate the semantic meanings and relationships among words, paragraphs, or even entire documents. While that may sound complex, consider this analogy: The word “apple” is semantically closer to the word “fruit”, signifying that an apple is a type of fruit, a relationship our brains naturally discern. However, we view the word “apple” as semantically distant from the word “car”. Embeddings provide a mathematical depiction of these meanings, maintaining the relative semantic distances as we cognitively perceive them. But how? By representing them as vectors or points in high-dimensional space. In this context, “apple” would be positioned closer to “fruit” and further from “car”, thereby reflecting the semantic relationships among these words. Specialized models are used to generate these embeddings.

In mathematical terms, a vector can be understood as a collection of coordinates that represent a point in space, corresponding to the size or dimensions of the vector itself. For instance, a vector of three numbers could denote a specific point in space, such as latitude, longitude, and altitude. Embeddings function on a similar principle, but typically involve thousands of dimensions, thus creating a high-dimensional space. Even though we cannot visually represent these in a way that’s comprehensible to humans, we can still manipulate them mathematically. For example, to calculate the distance between two text pieces, which would be a numerical measurement of how much their semantics are related.

Generating Embeddings for custom data

Embeddings can be created with specialized AI models that are specifically designed for this. OpenAI provides its own API endpoint to generate embeddings, but there are other models that can be used, the important thing is that you use the same model for all the documents you’re handling. The embeddings model processes a given text and outputs the corresponding high-dimensional vector.

This operation is not reversible, given an embedding, you can’t get the original text again, so the original text must be kept in order to retrieve it later. The interest in generating embeddings is that once we have them for two documents, we can compare how close or far they are from each other from a semantics point of view.

To enhance the efficiency of our embeddings and circumvent the size constraints inherent in embedding models, it might be necessary to partition large documents into smaller segments. The more distinct these segments are, the better. For instance, if we were to generate embeddings for this article, we would likely create one for each individual section.

Specialized Databases for Embeddings

When dealing with extensive text collections in Language Learning Models (LLMs), the large embeddings created require a special operation for retrieval, known as Nearest Neighbours Search. This operation is not supported by most conventional databases, necessitating the use of specialized databases adept at handling high-dimensional data.

There are many solutions designed for efficient storage, search, and management of vectors like Faiss by Meta, Milvus, Qdrant, Chroma DB, or Pinecone, among others​.

For more traditional setups, it is possible to use PostgreSQL with the pgvector extension, which could be convenient if you’re planning to keep all the data in the same place​, or use the advanced vector search features of indexing engines as ElasticSearch or OpenSearch.

Using Embeddings for Semantic Search

Once we have generated embeddings for all our documents and inserted them in a database that supports vector searches, they can be used to perform semantic searches, a type of search that understands the searcher’s intent and the contextual meaning of terms to generate more relevant results.

Because embeddings capture the semantic meaning of words and sentences, they can be used to find the “closest” (in terms of vector distance) entries in a database to a given query or question.

For example, if we input a query like “juicy fruit”, a semantic search powered by embeddings might return entries containing “apple” or “orange”, even if the exact phrase “juicy fruit” is not present. This is because the embeddings for “juicy fruit” and “apple” are close in the high-dimensional space.

In-Context Learning and Extending the LLM’s Long-Term Memory

A key aspect of LLMs like GPT-4 is their ability to generate text that is contextually relevant. However, these models traditionally do not have long-term memory, meaning they don’t remember information beyond the current context window (a few thousand tokens for GPT-4).

To expand this capacity and give the model a form of “long-term memory,” we can perform semantic searches for any user input, select the closest documents, and inject them as part of the model prompt to augment its knowledge of the matter before answering. This technique is known as in-context learning, because instead of training the model with new data, we inject relevant information as part of the prompt within the model’s context window.

For instance, if we’ve previously generated an embedding for a phrase like “Paris is the capital of France”, we can feed this embedding into the model when we ask it about French landmarks. Even though “Paris is the capital of France” is not in the immediate context, the model will still have this information available thanks to the injected embedding.

This method enables the model to recall and use information beyond its standard context window. This approach is particularly useful in applications where the model needs to maintain consistency or recall information over a large set of documents like a company’s Wiki or the documentation of a technical tool.

Conclusion

The development of LLMs like GPT-4 has been a major stride in AI and NLP with important implications at many levels, but when organizations want to use them with their own data, training a dedicated LLM from scratch is probably not the best solution. With the power of embeddings, we can store and use semantic information in previously unfeasible ways, allowing us to “teach” commercially available LLMs to work with data outside of their training corpus without having to re-train them.

This technique enables the possibility to any organization to build specialized LLM-powered tools that take advantage of private or very specific or private data and knowledge.

Do you want to know more? Visit our ‘Open Kitchen’ page to find everything we’ve learned on our AI journey. Don’t hesitate to contact us if you need help!

LLMs and Embeddings 101: Unleash the Power of GPT-4 with Unbounded Long-term Memory was originally published in The Agile Monkeys’ Journey on Medium, where people are continuing the conversation by highlighting and responding to this story.

This is a personal opinion piece with my current view on what’s coming. I might be plain wrong or change opinions tomorrow, so I invite you to respond to this post and share your thoughts and perspectives on this topic too.

Introduction

Large Language Models (LLMs) such as OpenAI’s GPT-4 are revolutionizing the software development industry by democratizing access to advanced knowledge and programming expertise. GitHub Copilot, for example, brings LLM capabilities directly into developers’ Integrated Development Environments (IDEs), and the upcoming release of GitHub Copilot X promises even more advanced features.

These tools will make us faster and produce higher-quality code, thus producing below-average code will soon become unacceptable, and developers must adapt to stay competitive.

The Evolving Role of the AI-enhanced Software Developers

In this new landscape, the day-to-day work of a software developer could shift towards a role more similar to that of a lead developer, with less emphasis on writing code and more focus on ensuring quality, reviewing code, deeply understanding business requirements, and agreeing with their team on code style, architecture, and boundaries. One of the most critical skills that software developers might need to acquire soon is their ability to build prompts and evaluate, fix, accept, or reject code and documentation proposed by LLMs.

Additionally, LLMs can significantly accelerate the learning process for developers. Previously, learning a new tool or language could take weeks or months of practice, with developers often becoming stuck in the process. The learning experience typically involved long periods of confusion followed by small moments of realization. LLMs can absorb the documentation of any tool and, with browsing capabilities, inspect the entire internet to find solutions to blockages in seconds. This drives developers through a series of realization moments, one after another, without getting stuck in obstacles. As a result, there’s no reason not to produce not only good but also idiomatic code for any technology you can imagine.

The post-GPT software engineers will barely write code themselves but still hold responsibility for any code pushed.

As I see it, the software engineer of the post-GPT world will need more knowledge of quality standards, design patterns and architecture, security, understanding of business processes, and soft skills. All of them already desirable qualities for excellent developers, but now made essential, because knowing how to write code for a specific stack won’t be enough anymore. They will need to master transferable programming skills that make them effectively polyglot, being able to use any programming language or framework in no time, and there will be no excuses for undocumented or poorly tested code.

Thriving in a Post-GPT Software Development World

Many developers are afraid of the upcoming changes, and there are good reasons to be concerned. We come from a world with a massive demand for anyone who could contribute to creating software. Knowing how to code could easily get you a great job offer with great perks. Now we are headed to a world where one developer with the proper set of skills could produce as much work as an entire team does today.

To remain competitive in such a landscape, developers must rethink their strategies and adapt to the new situation. Embracing continuous learning, specialization into hard-to-master topics, focusing on creative problem-solving, honing soft skills, and staying up-to-date with industry trends and emerging technologies will be essential. By cultivating their ability to evaluate others’ code and effectively guide its improvement critically, developers can navigate the evolving demands of the job market and thrive in a post-GPT software development world.

I think the world is gonna find out that if you can have ten times as much code at the same price, you can just use even more — Sam Altman in the Lex Fridman podcast (1:36:52)

But is it worth the effort? Many are worried that having AI-enhanced developers who do 10x the work will result in 10x fewer developers needed. I believe that at least during the next few years, even if productivity is increased by 10x, the amount of work required and what we expect from software might easily outgrow the productivity increase. AI is not here to compete against competent software developers but to allow us to work on all those ideas that never left the backlog because we had to focus on delivering the ones from the top.

As a personal opinion piece, I welcome your thoughts, experiences, and perspectives on this topic. Please feel free to share them and contribute to the conversation!

LLMs Raise the Bar in Software Development: Adapting to a New Era of Democratized 10x Productivity was originally published in The Agile Monkeys’ Journey on Medium, where people are continuing the conversation by highlighting and responding to this story.

Event-driven machine learning will enable a new generation of businesses that will be able to make incredibly thoughtful decisions faster than ever, but is your data ready to take advantage of it?

When you look at the speed of innovation in the AI world, it seems like each business out there will eventually fall within one of these two categories: the ones that took advantage of AI and the ones that went out of business because they could no longer compete. And the key differentiation factor is going to be the amount and quality of their data. Here, let me especially highlight the word quality because knowing what, when, and why something happened in your company will be crucial.

But how can you be certain that you can recover this information since you turned your system on for the first time? This is exactly the core value of event sourcing: Instead of storing only the current state of a system, everything that happens is stored permanently as a long series of immutable events. Then, the state shown to the users is calculated on demand, but the events are never discarded, they can always be inspected, reprocessed, or forwarded to other places.

Not all developers are willing to adopt this way of storing data though. Event sourcing and CQRS, an intimately related pattern, require a slightly different way of thinking. Software developers that are not used to them tend to fall back into implementing RESTful solutions instead, as they’re perceived simpler, but at design time it’s hard to predict who and when will need the extra data, so why going the extra mile? Why making the extra investment on development time and data storage?

In the era of AI, a company can no longer afford to discard a whole dimension of their data, especially when it is as relevant as time.

That is exactly the risk, once you’ve discarded the time-series data, it becomes unrecoverable. We’re sacrificing a dimension of the data that could become of high relevance in the mid-term for a perceived short-term benefit.

Data science teams mitigate this problem using Change Data Capture (CDC). Most databases keep an operations log that provide a reliable source of differential changes. But CDC misses another important variable: the semantics of each change, that is, why the change happened. It’s not the same to say “the order has been updated” as saying “the order has been paid,” the second statement is way more relevant from a business analytics point of view.

A development team that is aware of the importance of data will always make an effort to generate enough metadata to infer the context. Still, humans will make mistakes, and more frequently, we will easily decide to deprioritize this when there’s a hard deadline. A team that uses event sourcing generates high-quality data by design, without the team needing to make a conscious decision or extra effort.

Event sourcing removes the responsibility from the development team to decide whether a change must be stored, because the only way to change the system state is by creating an event that encapsulates what changed, when, and why.

But the advantages of storing events don’t end there. Events are also structurally different, they can be easily sorted and consumed in progressive chunks, and as they’re immutable, there’s no risk that a piece of data you’ve already processed gets out of sync because other process updates it. All event consumers can work assuming that they only need to process the next event in the queue and won’t miss a byte of data. Hence, they enable, among other things, what some authors are starting to describe as event-driven machine learning:

• Collecting data becomes easier with a real-time flow of events available to data analysts, as opposed to relying on static data dumps that provide a snapshot of a database at a particular moment. The event stream is akin to a live sports broadcast, which can be paused to view specific details in a scene, replayed to analyze interesting moments from the past, or monitored in real-time to observe the most current data.
• Continual learning becomes possible, this is a technique that emulates the human ability to adapt their behavior in a dynamic environment. The models are re-trained with recent events to continuously adapt them to changing environments (i.e. the recommendation system of Netflix requires constant fine-tuning in response to the introduction of new shows and evolving viewer preferences).
• Fine-grained data manipulation enables data-driven decision-making. Analysts can view data from different angles by reprocessing events and uncover previously inaccessible information. Each event represents a change that can be analyzed at an individual level, such as individual transactions in a bank account, or aggregated to see a broader picture at a particular time, such as the account balance. Data can also be processed to create new perspectives, like categorizing expenses by type.

With nowadays tools, using the data in this way can be expensive, and still require the expertise of specialized developers or data scientists, but we’re expecting advancements in large language models capabilities to provide natural language interfaces for manipulating and visualizing data. These new data manipulation models will make these techniques available for anyone in an organization, fostering actual data-driven business cultures.

New advancements in large language models capabilities will provide natural language interfaces for manipulating and visualizing data, and make data-driven decisions at every level in an organization.

In short, with all the amazing AI products we’ve seen recently, such as DALL·E or ChatGPT, we’re barely scratching the surface of what’s to come. It’s clear that AI is here to stay and will deeply affect how we conduct business. To stay competitive in this new AI-driven world, companies will need to collect high-quality data of everything that happens in their systems.

Event sourcing captures high-quality data by design, packing the what, why, and when for every change, recording events that can be compared to the individual frames of a high-definition security camera for your data.

This constant stream of events provides extra benefits. Data can be analyzed or manipulated in real-time, and previous states can be replayed. Event-driven machine learning models enable accurate predictions and automatic reactions to complex scenarios, working as a restless security guard that continuously monitor the cameras and find inefficiencies and risks.

For all these reasons, we believe that companies that adopt event-driven designs and learn to take advantage of event-driven machine learning models will have a strong competitive advantage and lead the market in this new AI-driven generation.

You can no longer afford time amnesia in your software systems. was originally published in The Agile Monkeys’ Journey on Medium, where people are continuing the conversation by highlighting and responding to this story.

So, you’re probably asking, why are these guys trying to create their own model instead of using tools like ChatGPT or GitHub copilot? Well, we believe that we can improve the performance of these state-of-the-art tools for code generation by reducing the scope of the model. In order to achieve this, we are using Booster framework, a framework that allows us to build event-driven microservices with the minimal amount of code possible. This will allow us to work on a constrained problem, reducing its complexity. The final result should be an AI assistant that empowers a new kind of development experience that doesn’t require you to know about a specific set of tools.

First steps

We believe that code generation models offer a wide range of benefits to software developers, including saving time, reducing errors, and improving overall productivity. However, we also recognize that creating an effective code generation model can be a complex and challenging task. For that reason, we have carefully planned and executed a series of steps:

1. Analyze the state-of-the-art models and select the most promising model to create an effective code generation model.
2. Collect all the necessary data that will be used to help our model learn the intricacies of our particular framework.
3. Explore the possibility of using in-context learning techniques to enhance the model capabilities.
4. Investigate if we can achieve further improvement by fine-tuning our model.
5. Measure the results of our code generation efforts

By carefully following these steps, we believe that we can build a highly effective and efficient code generation model that will be a valuable asset to our team, and Booster’s developers.

State-of-the-art model

We have invested considerable time analyzing the state-of-the-art models as they can significantly impact the performance and accuracy of the code generation system. After analyzing these models, we decided to use OpenAI’s text completion model, which is a Large Language Model (LLM) that will provide us with the necessary capabilities to build a highly effective code generation model. In addition, OpenAI provides all the necessary infrastructure and tools that will allow us to perform experiments in a very fast way to validate our hypothesis.

In fact, OpenAI has emerged as a leading provider of state-of-the-art models in machine learning and natural language processing. By making their models available through an API, OpenAI has democratized access to cutting-edge AI and enabled researchers and developers around the world to build on their work. However, it is more complex than just selecting the model, training a code generation model requires labeling enough quality examples of our code algorithms. So, what do we need to create a model that generates code with our framework?

Data data and data

Collecting the data is a crucial step to creating a good machine learning model. In this sense, we also want to encourage you to contribute to our mission. If you are interested in helping us to generate data to train this model, you are more than welcome to join the booster discord and start collaborating with us.

As we all know, the foundations of machine learning are built on data. It’s like trying to make a sandwich without bread — no matter how many delicious ingredients you have, without the basics, you don’t have much. But what makes data so critical for training models? The importance of data is to provide context to your AI. Deep learning models are really good at predicting and inferring patterns. They need to know the situation they are going to handle to make an accurate prediction. It’s like having your own Sherlock Holmes and asking him to solve a case without giving him any details.

However, simply having a large dataset is not enough, the quality of the data is also crucial. For a model to learn effectively, the data must be clean, well-labeled, and representative of the real-world scenarios the model will encounter. However, obtaining such quality data can be challenging, particularly in certain domains.

Based on our experience, training a model to generate code can be a challenging task that requires high-quality input data in the form of code snippets and associated descriptions or prompts. The main issue is that this type of data is often proprietary and not widely available, which can limit the amount of quality data that can be obtained. In our case, we tackled this problem by collecting all the Booster projects available and then manually labeling them. However, despite our efforts, we found that the lack of examples was still an issue due to the current limited availability of Booster projects.

If you want to help us create the next generation of software development experience, as mentioned before you can join us in the discord channel. In addition, a very simple solution, but an effective one is to start using GitHub tags for your own Booster projects. This way, we’ll be able to filter projects on GitHub and find Booster’s projects quickly. However, to make this idea work, there must be a commitment from the developers/community that they will use the tag. We want you to use the Booster framework tag!

Advice time

Based on our experience, we want to share some tips that will improve the performance of the OpenAI model:

• Use a separator at the end of the prompt, e.g. \\n\\n###\\n\\n. It’s important to remember to use the separator when we’re performing inference.
• Use an ending token at the end of the completion, e.g. END. In addition, it’s important to add the ending token as a stop sequence during inference, e.g. stop=[" END"]
• It would be ideal if the dataset has at least ~500 examples.
• Ensure that the prompt + completion doesn’t exceed 2048 tokens, including the separator.
• It’s supposed to use a lower learning rate and only 1–2 epochs the model tends to work better for conditional generation.

It’s also important to keep in mind that the quality of the data is more important than the quantity. In other words, it’s better to have a smaller number of high-quality examples than a larger number of low-quality examples.

In-context learning

For our initial experiments, we decided to use the base model of GPT-3 and conduct some in-context learning tests. The basic concept of in-context learning is to provide a list of input-output pairs as input to the language model to demonstrate how to solve a particular task.

This approach allows us to append a test input and enable the model to make predictions based on the prompt and predicted next tokens. The interesting idea behind using this method is that it will allow us to quickly assess whether the model shows signs of being able to learn to generate Booster code. Once we validate this, we can proceed with other techniques such as fine-tuning that will improve previous results.

Based on the results of the experiment, we can assume that the model can learn how to generate Booster’s code quite well. Therefore, we believe that if we were able to generate a sufficiently large dataset that covers most of the possible cases, we should obtain a model with good performance for generating entire Booster microservices from natural language.

As mentioned before, it’s time to move to other ways of training our model. In-context learning is good for quickly testing this idea but it has some limitations. The biggest one is the number of tokens allowed as input. We will be limited in the number of cases we are able to provide as context to the model. Infrastructure costs will also increase since we have to provide context for every code generation. We believe that it’s better to proceed with other training methods at this step of the process.

Fine-tuning model

Fine-tuning is the process of retraining a pre-trained network on a new dataset to improve its performance for a specific task. The advantage of performing fine-tuning is that the network can fit the new data while preserving the knowledge it acquired from the pre-training.

We have already validated the hypothesis that booster code could be generated using GPT-3 through in-context learning. Therefore, for this step, we will be using fine-tuning, so there is no need to provide examples in the prompt anymore. This saves costs and enables lower-latency requests. For that reason, we decided to fine-tune our model (The monkey developer). This should allow us to have a model that we can explain with natural language how we want the code of our framework to be, and obtain the desired code as a result. These are our conclusions after fine tuning our model:

• The results obtained are promising; if we compare the base model of GPT-3 with the results obtained with the fine-tuned model, this model is infinitely better.
• We need to increase the number of examples provided to the model in the training phase. Through this, we believe that we could achieve better results.
• We should perform one experiment using a bigger training dataset, in order to measure the correlation between the performance and the size of the dataset.

However, we have not been able to carry out these experiments, as we were unable to collect more data.

Evaluating the results

Evaluating the model’s performance is important to determine how well it’s working and whether it’s providing accurate results. In order to track all the experiments, we recommend using a platform like weights & biases, which allows you to manage and version your data, collaborate with your team, and focus on building the best models.

In addition, to compare different experiments, we used the Bleu score. This measures the similarity between the generated code and the reference code, and gives a good idea of how close the generated code is to the reference code. By doing this, we could compare different techniques as in-context learning. As we can see in the following image, by using in-context learning and fine-tuning, we got the best performance in generating Booster’s code.

The question is, are these results enough to use these code generation models in a real application?

Conclusion

The results obtained are not good enough to be able to use this tool in a real application. Although the performance of the base model has been improved and, in some cases, the expected code has been generated, there are other cases, perhaps less general and more complex, where the output of the model has been completely wrong. However, these experiments have helped us understand the following points::

• The Fine-tuned GPT-3 model shows improvements over the base model.
• Increasing the number of examples provided to the model is important. In addition, there is a correlation between performance and dataset size.
• We predict that the model may be able to learn how to generate Booster’s code well, using a large dataset covering most cases.
• Community involvement and a data platform are potential options for generating more of Booster’s code or code for your framework.

We need your help! We would like to call on the community to help us build a more complete dataset to improve our code generation tool. If you want to be part of this amazing community you can get in touch using booster discord.

Building an Automated Coding Model for Our High-Level Abstraction Framework was originally published in The Agile Monkeys’ Journey on Medium, where people are continuing the conversation by highlighting and responding to this story.

Storage is one of the essential features for most projects. This tutorial will show you how to easily add cloud file management to your Booster backend.

As a reminder, our previous articles showed you how to create a cloud-based backend application using Booster. We also demonstrated how to connect this backend with an iOS application easily. All this process was documented in these two articles:

• Backend part: “Build Scalable Low-Code Backends with Booster”
• Frontend part: “From iOS Dev to Full-Stack in No Time with Booster”

To showcase this new file feature, we will use the Live Questions app documented in those articles to allow users to set a profile picture that will be uploaded to the Booster backend.

Rockets are cool 🚀

Booster utilizes Rockets to extend your backend infrastructure and runtime with features not included in the framework core. Think of rockets as reusable and composable plug-ins that anyone can create to extend Booster functionality.

In this case, for adding file management to our Live Questions Booster backend, we will integrate the File Uploads Rocket already created and available to the Booster community.

Backend part

This rocket allows file storage in AWS or Azure as the cloud providers. We are going to use AWS for our backend and examples. Here’s a quick overview of the things you can do with the 1.0.0 of this rocket, as you can see in the Readme file in the repository:

• Get the pre-signed upload URL to which you will do the multipart file upload.
• Get the pre-signed download URL of an existing file.
• List all the files in a directory.
• Delete a file.
• Add authorization to those operations easily through your Booster commands and user roles.
• When a file is uploaded to the cloud, the rocket will generate a new Booster event and entity. This way, you can create a Booster ReadModel in your backend to provide a view of the files uploaded to a directory.

As a reminder, let’s ask ChatGPT what a pre-signed URL is and why we need them. Answer:

A pre-signed URL is a URL that has been signed with authentication credentials, allowing anyone with the URL to upload or download the associated object in your Amazon S3 bucket, without requiring AWS security credentials or permissions.

Pre-signed URLs are useful when you want to give time-limited permission to upload or download a specific object. For example, you might use a pre-signed URL to allow a user to upload a file to your S3 bucket, or to download a specific file that they have been granted access to. The pre-signed URL contains all the information required to perform the upload or download, including the bucket name, object key, and a signature that verifies the authenticity of the request. The signature is generated using your AWS access key, which is used to calculate the signature for the URL.

Pre-signed URLs are valid for a limited period of time, which is specified when the URL is generated. Once the URL has expired, it can no longer be used to upload or download the associated object. This helps to ensure that your data remains secure, as it cannot be accessed after the pre-determined expiration time.

Update dependencies

To include the AWS Rocket in your project, go to your package.json file and add the following packages in the dependenciespart:

  "@boostercloud/rocket-file-uploads-aws": "VERSION",
  "@boostercloud/rocket-file-uploads-core": "VERSION",
  "@boostercloud/rocket-file-uploads-types": "VERSION",

And the the infrastructure package in the devDependencies part:

"@boostercloud/rocket-file-uploads-aws-infrastructure": "VERSION"

Note: If you are using Azure as your cloud provider, make sure to use the appropriate packages (-azure instead of -aws).

Configure the rocket in your Booster backend

As you can see in our config.ts file:

 const rocketFilesConfigurationDefault: RocketFilesUserConfiguration = {
         storageName: '<STORAGE_NAME>', // The name you want for your AWS S3 bucket.
         containerName: '', // You can leave this empty, not needed in AWS.
         directories: [<DIRECTORY_NAME_1>] // Root directories for your files.
 }


Booster.configure(environment.name, (config: BoosterConfig): void => {

[...]

config.rockets = [new BoosterRocketFiles(config, 
                      [rocketFilesConfigurationDefault]).rocketForAWS()]

Note: The containerName parameter is not used in AWS, so the final structure created will be storage > directory > files.

Create a Command for getting the Pre-signed Upload URL

As we will use this rocket to upload and download profile pictures, we created a command that abstracts the clients without specifying parameters like the file name and the file directory. Our profile-picture-upload-url.ts command make that decision internally (and extra validations). The image will be stored in “files/<userId>/profilePicture.jpg”. Our command interacts with the rocket through the FileHandler API:

@Command({
     authorize: [UserRole],
     before: [CommonValidations.userValidation]
 })
 export class ProfilePictureUploadURL {
   public constructor() {}

   public static async handle(command: ProfilePictureUploadURL, register: Register): Promise<PresignedPostResponse> {
     const boosterConfig = Booster.config
     const fileHandler = new FileHandler(boosterConfig, ConfigConstants.rocketFilesConfigurationDefault.storageName)
     return await fileHandler.presignedPut(ConfigConstants.rocketFilesConfigurationDefault.directories[0], `${getUserId(register)}/${profilePictureKey}`) as Promise<PresignedPostResponse>
   }
 }

The PresignedPostResponse contains the upload URL and the metadata fields needed to perform the multipart upload from the client:

export class PresignedPostResponse {
  public constructor(
    readonly url: string,
    readonly fields: { [key: string]: string }
  ){}
}

Create a Command for getting the Pre-signed Download URL

The command for getting the download URL is similar, but the response will be just the download URL string this time. As you can see inprofile-picture-download-url.ts:

@Command({
    authorize: [UserRole],
    before: [CommonValidations.userValidation]
})
export class ProfilePictureDownloadURL {
  public constructor() {}

  public static async handle(command: ProfilePictureDownloadURL, register: Register): Promise<string> {
    const boosterConfig = Booster.config
    const fileHandler = new FileHandler(boosterConfig, ConfigConstants.rocketFilesConfigurationDefault.storageName)
    return await fileHandler.presignedGet(ConfigConstants.rocketFilesConfigurationDefault.directories[0], `${getUserId(register)}/${profilePictureKey}`)
  }
}

You can also check the files-list.ts and delete-profile-picture.ts commands.

And that’s it. That was the backend part. Easy, right?

iOS part

Mutations and client API generation

Once you have deployed your Booster changes and the new GraphQL schema is generated, we can start adding the mutations needed to communicate with our new commands in the app.

ProfilePictureUploadURL.graphql

mutation ProfilePictureUploadURL {
    ProfilePictureUploadURL {
        url
        fields
    }
}

ProfilePictureDownloadURL.graphql

mutation ProfilePictureDownloadURL {
    ProfilePictureDownloadURL
}

You can also check FilesList.graphql and DeleteProfilePicture.graphql.

As we previously mentioned in the other article, you can use Apollo’s Code Generation tool to download your schema and generate the Swift API for the types defined in it. Execute the following command in the terminal on the root folder of your iOS project:

./apollo-ios-cli generate -f

Profile image upload

To upload the profile picture, you will first need to call the ProfilePictureUploadURL mutation to obtain the necessary information for the file upload, such as the URL and the fields required for the pre-signed multipart request:

func uploadProfilePicture(data: Data) async throws {
        let mutation = BoosterSchema.ProfilePictureUploadURLMutation()
        
        guard let result = try await networkClient.mutate(mutation: mutation)?.profilePictureUploadURL,
              let fields = result.fields.value as? [String: String] else { throw FileError.presignPostFailure }
                   
        let presignedPost = PresignedPost(url: result.url, fields: fields)
        let uploadEndpoint = API.uploadFile(data: data, metadata: presignedPost)
        let uploadRequest = try URLRequest(endpoint: uploadEndpoint)


        _ = try await URLSession.shared.data(for: uploadRequest)
 }

Profile image download

You need to call the ProfilePictureDownloadURL mutation to get the pre-signed URL of the file you want to download:

 func downloadProfilePicture() async throws -> Data {
        let mutation = BoosterSchema.ProfilePictureDownloadURLMutation()
        
        guard let urlString = try await networkClient.mutate(mutation: mutation)?.profilePictureDownloadURL,
              let url = URL(string: urlString) else { throw FileError.presignGetFailure }
        
        let fileRequest = URLRequest(url: url)
        let (imageData, _) = try await URLSession.shared.data(for: fileRequest)
        return imageData
 }

You can also see examples of the removeProfilePicture() and filesList() methods in FileService.swift.

Conclusion

Integrating file upload and download into your Booster app was easy with this rocket. With a few new mutations and requests, you now have file management capabilities!

Booster offers a convenient way to add file management with access control to enterprise-grade apps without cloud expertise. Reach out to our supportive developer community on Discord for any questions or help. Join us on Discord and become part of our thriving community!

This article was co-authored by Damien Vieira and Juan Sagasti.

Cloud File Management with Booster was originally published in The Agile Monkeys’ Journey on Medium, where people are continuing the conversation by highlighting and responding to this story.

On March 24, 1826, Matilda Joslyn Gage was born in New York. A prolific writer and activist who contributed to many social causes such as abolition, feminism, Native American rights and religion, but best known for or her contributions to the suffragette movement. These contributions began in 1869 with significant accomplishments including:

• Funding, along with Susan B. Anthony and Elizabeth Cady Stanton, the National Woman Suffrage Association.
• Publishing the National Citizen and Ballot Box, NWSA’s official newspaper from 1878 until 1881.
• Writing The History of Woman Suffrage, along with Stanton and Anthony.
• Funding the Women’s National Liberal Unity.

She encouraged her son-in-law, L. Frank Baum, to write down stories. Gage’s words and writings about how governments and churches have persecuted innocent women throughout the centuries by accusing them of heresy and witchcraft (Woman, Church and State, 1893) inspired his stories, in particular, his most famous work: The Wizard of Oz. Many of Gage’s ideas can be recognized in the themes of the book. He used his mother-in-law’s vivid descriptions of witch-hunting as inspiration for world building, where he created a world where men and women are treated as equals, basic needs are met, contributions are made according to ability, moral principles exist independent of religious institutions, cultural differences are valued, and conflicts are not tolerated. Gage’s views influenced Baum’s portrayal of the strong and independent female characters in the book, as well as his critiques of organized religion. Some authors say, Baum’s character of “the good witch” may have been inspired by her mother-in-law, being Glinda a contraction of good and Matilda.

Matilda has also been the inspiration for the NoMoreMatildas movement. In 1870 she published an essay entitled Woman as Inventor (1870) in which she denounced a social situation in which women scientists receive less credit and recognition for their scientific work than men.

Matilda’s figure not only inspires us but also has taught us the importance of giving credit to those who deserve it. Most of the time we don’t think about the importance of giving credit, do you ever use an image for an article or presentation without credit to the artist that created it? This could be seen as a simple thing, I find something on the Internet so I use it. But this scenario can be extrapolated to other areas such as science or software. Giving credit to someone works means you appreciate that work you can use, you’re giving value to someone else's work so they can gain confidence in that work and receive feedback so they can grow stronger as professionals.

References:

• https://msmagazine.com/2021/03/29/wizard-of-oz-matilda-joslyn-gage-suffrage-feminist/
• https://matildajoslyngage.org/gage-biography
• https://matildajoslyngage.org/about-the-gage-home
• https://www.historynet.com/matilda-josyln-gage-the-unlikely-inspiration-for-the-wizard-of-oz/

Matilda Joslyn Gage was originally published in The Agile Monkeys’ Journey on Medium, where people are continuing the conversation by highlighting and responding to this story.

Are you a backend developer looking to significantly boost your productivity, or an iOS (or mobile) developer wanting to become a full-stack developer in no time? Keep reading!

The Problem

When choosing a backend stack for a new app, developers often use no-code or low-code platforms like Firebase because they offer a variety of features. However, these tools might not be suitable for more complex data or situations requiring scalability and robust business logic.

On the other hand, using serverless solutions from scratch can be time-consuming and require a lot of effort to set up and manage.

You just want to move fast and care most about your business logic, right? The good news is that there’s a new framework called Booster that can make backend development a breeze, even for those who are new to it, thanks to its high level of abstraction.

Why this tutorial?

We are two developers from The Agile Monkeys with a strong background in iOS development, but very limited experience with backend production projects. Despite this, we have successfully used the Booster framework in several production projects! We want to share our experience with you, and for that we have created an open-source iOS app and an open-source Booster backend, as well as written three articles detailing the process.

• This tutorial covers the Booster backend, and the Booster code can be accessed here.
• A tutorial on the iOS integration can be found here, and you can view the iOS code here.
• An optional tutorial on how to integrate a Booster Rocket for file storage can be found here.

In addition, the iOS application built for this tutorial is available on the App Store. The app, called Live Questions, is a tool for live events, keynotes, and Q&A sessions that allows hosts to create a topic and gather questions from other users in real time.

To be transparent, Booster is actually a framework developed by another team within our company. Our backend and event-sourcing experts have been working on creating this high level of abstraction for years. As iOS developers, we have used Booster as if we were external clients of the tool, and have been extremely impressed with the developer experience and support we have received.

WTF is Booster?

In short, Booster is a Low-Code TypeScript framework that allows you to quickly and easily create a backend application in the cloud that is highly efficient, scalable, and reliable. It follows an event-driven, event sourcing, and domain-driven design (DDD) to achieve these goals and it reimagines the software development experience to maximize your team’s speed and reduce friction on every level.

The event-driven design of Booster means that the server is designed to process and respond to events rather than traditional requests, making it highly efficient as it can process multiple events concurrently and asynchronously.

The event sourcing design of Booster stores all changes to the state of the backend as a sequence of events, like an event log, rather than in a traditional database. This allows the backend to reconstruct the current state of the system at any point in time and makes it easier to track and debug changes to the system. It also allows for the easy implementation of new features by replaying stored events.

Finally, the Domain-Driven Design of Booster involves breaking down the system into small, cohesive components that represent different business entities and their relationships. This helps to create a backend that is more scalable, maintainable, and adaptable to changing business needs.

A backend project using Booster is organized, on-rails, into the following components:

• Commands: Actions that users can take within the system (e.g. create a topic).
• Events: Simple records of facts (e.g. user X reacted to question Y on topic Z). As Booster is an event-driven framework, events are central to the design of a Booster project.
• Entities: The source of truth, representing data about the subjects discussed within your company (e.g. topics, questions, users).
• ReadModels: Public projections of Entities. While Entities are private within the system, ReadModels are publicly accessible (e.g. get topics).
• Handlers: Code that processes commands, reacts to events to trigger other actions, or updates Entities based on new events.

Modeling a reactive backend is made easy with these building blocks! The high level of abstraction makes it intuitive, even for non-technical people. As an example, here is a subset of the backend we created for the Live Questions app:

Let’s create our Booster backend!

Before starting, it is recommended that you open our open-source repository and refer to it as a guide during this process.

1. Install the Booster CLI

Booster is supported on macOS, Windows, and Linux. You can follow the installation steps on this page.

2. Create the Booster project

Navigate to the root folder of your new project, and then run the following command in the Terminal:

boost new:project [Project-Name]

Follow the prompts and select one of the available cloud providers as the main provider for your project. We recommend AWS for this project. You first need to set up a free AWS account in order to deploy your project.

After running the command to create your project, you will have a properly configured Booster project that you can open in your preferred code editor (such as Visual Studio Code or VSCodium).

The project folder will contain configuration files and empty folders for you to add your commands, entities, events, etc:

For more detailed information about creating a project with Booster, refer to the official documentation.

3. Create Commands

Commands are input for our system that are typically initiated by the users of our application. Upon receipt, we can validate the data, perform any necessary business logic, and register one or more events.

To create a new command, you can use the command line generator in the root directory of your project (you can also create the file manually). Here is an example of how to do this:

boost new:command CreateTopic --fields title:string timeToLive:number

This command will create a TypeScript file called ‘create-topic.ts’ in the ‘src/commands’ directory. The ‘fields’ flag is optional, and you can also add attributes to your command through code after the file has been created.

You can specify who is authorized to execute the command by adding the authorized roles. For now, you can use ‘all’ to authorize anyone. We will discuss user authorization in more detail later.

@Command({
authorize: 'all'
})
export class CreateTopic {…}

To know more about commands, refer to the official documentation.

4. Create Events

Rather than creating, updating, or deleting objects, Booster stores data in the form of events, which are records of facts and serve as the source of truth. To generate an event called ‘TopicCreated’ with initial topic information, run the following command:

boost new:event TopicCreated --fields topictId:UUID title:string

This command will create a new file in the ‘src/events’ directory. Once you have created all your events, return to your commands to register them. To know more about events, refer to the official documentation.

5. Create Entities

Entities represent the internal state of our system and are responsible for combining all events with the same entityID. To create an entity, run the following command:

boost new:entity Topic

This command will create a new file in the ‘src/entities’ directory. Once you have added all your entities, you can use them to reduce events and update the entity state. To know more about entities, refer to the official documentation.

6. Create Read Models

In a real application, it is often necessary to limit access to the full domain model (entities), and different users may have different views of the data depending on their permissions. Read models provide a solution to this problem by projecting one or more entities into a new object that can be accessed through a GraphQL query or subscription.

To create a read model, run the following command:

boost new:read-model TopicReadModel --projects Topic:id

The ‘ — projects’ flag allows you to specify the entities that the read model will observe for changes to project onto the read model. The ‘:id’ after the entity name is the join key.

As mentioned earlier when creating commands, you will need to specify who is authorized to query your read models. For now, you can use ‘all’.

To know more about read models, refer to the official documentation.

7. Local Testing

We have:

• Created a publicly accessible command
• Emitted an event as a mechanism for storing data
• Reduced the event into an entity to represent the internal state of the system
• Projected the entity into a publicly accessible read model

To ensure that your code is error-free and compiles correctly, run the following command:

boost build –verbose

To start the application locally:

boost start -e local

This will execute a local Express.js server and try to expose it on port 3000. Booster applications come with a GraphQL API out of the box, which you can use to send commands through mutations and retrieve read model data through queries or subscriptions.

For testing in local or development environments, we suggest using the Altair GraphQL client. Simply enter the endpoint URL for your application, which is located at http://localhost:3000/graphql. It will also load the documentation for your API schema!

If you are testing a deployed backend (not on your local machine), the endpoint URLs will be displayed in the console after a successful deployment to AWS.

Keep in mind that subscriptions are currently not supported in a local environment. For that, you need to deploy the app to AWS.

In order to ensure that only authorized users can access data or perform actions on the system, it is important to include an ‘Authorization’ header with the value ‘Bearer <user_valid_access_token>’ in your requests using Altair for commands or read models that require authorization. Don’t forget to do this when making requests:

Note that if you are testing some Commands locally and don’t want to deal with the authentication part (we explain authentication later in the article), you can just keep the Command public and authorize it for ‘all’:

@Command({
authorize: 'all'
})
export class CreateTopic {…}

8. Configuration

The config.ts file is where you can configure the environments you will be using in your project. If you take a look to our config file:

function buildEnvironment(environment: Environment): void {

  Booster.configure(environment.name, (config: BoosterConfig): void => {
    config.appName = environment.appName
    config.providerPackage = packageOfProvider(environment.provider)

    switch (environment.provider) {
      case EnvironmentProvider.Local:
        config.logLevel = Level.debug
        break

      case EnvironmentProvider.AWS:
        config.logLevel = Level.debug
        config.assets = ['.env'] // Needed for secrets. Fill your secrets in the .env file.
        config.tokenVerifiers = [
          new JwksUriTokenVerifier(
            environment.auth0Issuer,
            environment.jwksURL,
            "custom:roles"
          )
        ]

        break
    }
  })
}

buildEnvironment(productionEnvironment)
buildEnvironment(developmentEnvironment)
buildEnvironment(localEnvironment)
dotenv.config()

It is generally best practice to store sensitive information, such as passwords or API keys, in environment variables rather than hardcoding them in your code. This helps to prevent accidental exposure of sensitive information, such as when committing code to a public repository. The dotenv module can be used to separate environment-specific configuration from your code, making it easier to manage and deploy your application. It loads environment variables from a .env file, as seen in the line config.assets = ['.env']. This file can be used to store your Auth0 secret keys, which will be discussed later in the authentication portion of the tutorial.

To add the dotenv module or other dependencies to your project, you can use the npm install command, for example: npm install dotenv. This will add the dependency the project's package.json file, as shown in the example below:

{
  "name": "livequestions",
  "description": "Backend of the questions gathering app",
  "version": "0.1.0",
  "author": "The Agile Monkeys",
  "dependencies": {
    "@boostercloud/framework-core": "^1.3.0",
    "@boostercloud/framework-provider-aws": "^1.3.0",
    "@boostercloud/framework-types": "^1.3.0",
    "auth0": "^2.30.0",
    "dotenv": "^16.0.0"
  },
[...]

To install all dependencies already defined in the package.json file, you can simply run npm install.

Additionally, you may want to take a look at the environment.ts file we created to abstract the details of each environment:

[...]

export const developmentEnvironment: Environment = {
  name: 'dev',
  appName: 'livequestions-dev',
  provider: EnvironmentProvider.AWS,
  auth0Issuer: "<AUTH0_ISSUER>", // Example: "https://blablabla.eu.auth0.com/",
  jwksURL: "<AUTH0_JWKS_URL>" // Example: "https://blablabla.eu.auth0.com/.well-known/jwks.json"
}

[...]

9. User Authorization with Auth0

To ensure secure access, Booster uses the OAuth 2.0 protocol for authentication. This protocol involves the use of JSON Web Tokens (JWT) to identify and authorize users. These JWT tokens, also known as access tokens, are issued by an authentication provider such as Auth0, which we are using in this project. Other popular providers that support OAuth 2.0 include Firebase and others.

As mentioned in our iOS tutorial, we are utilizing Auth0 for user authentication in our system. In addition, we will be implementing a role-based authorization mechanism in the Booster backend to ensure that only authenticated users with the appropriate role can perform certain actions. To begin, let’s create the roles.ts file and define the UserRole:

import { Role } from '@boostercloud/framework-core'

@Role()
export class UserRole {}

While we could define additional roles, such as Admin, for the purposes of this tutorial we will only be adding one role to keep things simple.

To begin using Auth0 for your experiments, create a free account and then go to the Applications section to create a new API called ‘Booster API’.

Remember the environment.ts file? There we can add the issuer URLs that Booster will use to validate the token:

export const developmentEnvironment: Environment = {
  name: 'dev',
  appName: 'livequestions-dev',
  provider: EnvironmentProvider.AWS,
  auth0Issuer: "<AUTH0_ISSUER>", // Example: "https://blablabla.eu.auth0.com/",
  jwksURL: "<AUTH0_JWKS_URL>" // Example: "https://blablabla.eu.auth0.com/.well-known/jwks.json"
}

Now we need to add a custom claim to our Auth0 token that Booster will use to get the allowed role (in this case our role is UserRole). To do this, we will create a custom action called ‘Assign User Role’ that will be executed every time a user logs in and before the token is issued:

Inside the code for the Auth0 action, we will add a key-value claim that Booster will look for within the token:

exports.onExecutePostLogin = async (event, api) => {
  if (event.authorization) {
    api.idToken.setCustomClaim("custom:roles", "UserRole");
    api.accessToken.setCustomClaim("custom:roles", "UserRole");
  }
}

If you remember our config.ts in Booster, we had that key defined at the token verifier level:

[...]
case EnvironmentProvider.AWS:
        config.logLevel = Level.debug
        config.assets = ['.env']
        config.tokenVerifiers = [
          new JwksUriTokenVerifier(
            environment.auth0Issuer,
            environment.jwksURL,
            "custom:roles" // <-------- 
          )
        ]
[...]

Now Booster and Auth0 are working together!

You can now add proper role-based authorization to your commands and read models!

@Command({
  authorize: [UserRole]
})

@ReadModel({
  authorize: [UserRole]
})

But we are going to go a step further and allow Booster to trigger user accounts deletions in Auth0 for the deletion feature. For that, we need to grant Booster access to the Auth0 Management API.

Go to the Applications section in Auth0 and create a new Machine-to-Machine application called “Booster Users Management.” In the APIs tab, ensure that this application has access to the Auth0 Management API and enable the “delete:users” permission.

Now, in Booster, you need to add the Domain, Client ID, and Client Secret values of this Machine-to-Machine application to the .env file that we previously mentioned:

AUTH0_DOMAIN_DEV=TODO // Example: blablabla.eu.auth0.com
AUTH0_CLIENT_ID_DEV=TODO
AUTH0_CLIENT_SECRET_DEV=TODO

Install the auth0 dependency in Booster:

npm install auth0

You can check how we call the Auth0 Management API from the delete-user.ts Booster command:

[...]

@Command({
  authorize: [UserRole],
  before: [CommonValidations.userValidation]
})
export class DeleteUser {
  public constructor() {}


  public static async handle(command: DeleteUser , register: Register): Promise<void> {
    const userID = getUserId(register)
    const environment = currentEnvironment()

    /// Delete user from Auth0
    const management = new ManagementClient({
      domain: process.env[`AUTH0_DOMAIN_${environment.name.toUpperCase()}`]!,
      clientId: process.env[`AUTH0_CLIENT_ID_${environment.name.toUpperCase()}`],
      clientSecret: process.env[`AUTH0_CLIENT_SECRET_${environment.name.toUpperCase()}`],
      scope: 'delete:users'
    })

    await management.deleteUser({ id: userID })

[...]

It’s important to note that we have already initialized the dotenv library with the dotenv.config() function in the config.ts file.

To know more about authorization, refer to the official documentation.

10. Deployment

Now that we have our environments and roles are properly configured, we can deploy our application to the cloud!

boost deploy -e dev --verbose

The ‘-e <environment name>’ flag specifies which environment to deploy.

By running the ‘deploy’ command, Booster will handle the creation of all necessary resources in AWS, such as Lambdas and API Gateway, and configure the necessary permissions, events, and triggers to connect them. It even creates a fully functional GraphQL API. Forget about infrastructure code!

Worth mentioning

Logging

You can check the application log groups in CloudWatch, in your AWS account:

To know more about logging, Refer to the official documentation.

Error handling

Data deletion

As you know, Booster uses event sourcing to store and manage changes to an application’s state by recording them as a sequence of events in an event store. Deleting data can be challenging in event sourcing because events are typically append-only and may have dependencies on other events. Currently, the Booster framework only supports soft deletion. Hard deletion, which permanently removes events from the event store, will be supported in the near future!

With soft deletion, you can mark entities as deleted by adding a new event indicating this, rather than deleting the events themselves. This allows you to maintain the integrity of the event stream while still effectively deleting a read model. For example, in our application, we can update a topic or question entity to a deleted state. We also support deletion of users. We encourage you to check these use cases.

Migrations

When implementing a Booster application, you may need to update your entities and perform some schema migration. While data migrations are supported in Booster, they can be complex and require careful planning and testing.

During the development phase, when the state of your application is not critical, it may be easier and faster to delete all of your data directly from your cloud provider and redeploy your application because having incompatible versions of events can prevent Booster reducers to work. This will help you avoid any inconsistencies in the system and make the migration process simpler.

Conclusion

By using Booster, it is possible to create an enterprise-grade, scalable application without needing extensive expertise in cloud technologies or event-sourcing. We found the reactive nature of Booster to be very intuitive and effective in boosting productivity.

Booster is constantly improving. The team is working on new features that will make our lives even easier in the near future. For instance, they are developing a native authentication solution, the ability to delete events and data permanently, and simpler storage options for file uploads. Stay tuned for updates!

Remember that we’ve also written an article on how to use a Booster backend with an iOS app, and an article on how to extend Booster with a rocket to add the file storage feature.

If you have any questions about these articles or want to learn more about using Booster, or if you’d like to contribute to this open source framework, come and join us on Discord! Our community of developers is always happy to help and share their knowledge, no matter your skill level. We hope to see you there!

This article was co-authored by Damien Vieira and Juan Sagasti.

Build scalable Low-Code backends with Booster 🚀 was originally published in The Agile Monkeys’ Journey on Medium, where people are continuing the conversation by highlighting and responding to this story.

Are you an iOS (or mobile) developer who wants to become full-stack in no time? Keep reading!

The Problem

When developing a new app, it’s important to consider which backend stack to use for data synchronization and storage. No-code or low-code platforms like Firebase are often popular choices for app developers because they provide a range of useful features out of the box. While these tools can be helpful for quickly creating an MVP or getting a simple application into production, they may not be suitable for more complex data sets or situations where robust business logic and scalability are required.

On the other hand, setting up and using complex serverless solutions from scratch can be time-consuming and often require a significant amount of effort. There are many different tools and components to manage, and it can be error-prone and involve a lot of boilerplate code. You want to move fast and care most about your business logic, right? Then keep reading.

The good news is that there’s a new framework called Booster that can make backend development a breeze, even for those who are new to it, thanks to its high level of abstraction.

Why this tutorial?

We are two developers from The Agile Monkeys with a strong background in iOS development, but very limited experience with backend production projects. Despite this, we have successfully used the Booster framework in several production projects! We want to share our experience with you, and for that we have created an open-source iOS app and an open-source Booster backend, as well as written three articles detailing the entire process.

• This tutorial covers the iOS integration of a Booster backend, and you can view the iOS code here.
• The tutorial on the Booster backend can be found here, and the Booster code can be accessed here.
• An optional tutorial on how to integrate a Booster Rocket for file storage can be found here.

In addition, the iOS application built for this tutorial is available on the App Store. The app, called Live Questions, is a tool for live events, keynotes, and Q&A sessions that allows hosts to create a topic and gather questions from other users in real time.

To be transparent, Booster is actually a framework developed by another team within our company. Our backend and event-sourcing experts have been working on creating this high level of abstraction for years. As iOS developers, we have used Booster as if we were external clients of the tool, and have been extremely impressed with the developer experience and support we have received.

In this tutorial, we will guide you through the process of integrating a Booster backend with an iOS app. We assume that you already have a basic understanding of iOS development.

WTF is Booster?

The world, businesses, and modern software are all driven by events. As app developers, we understand this concept well, as our apps are designed to respond to user actions and various system and environment events. With Booster, you can build reactive backends that follow this event-driven model, while also enjoying a faster development process thanks to its high level of abstraction. In short, Booster allows you to create reactive cloud-based apps efficiently and effectively.

A backend project using Booster is organized, on-rails, into the following components:

• Commands: Actions that users can take within the system (e.g. create a topic).
• Events: Simple records of facts (e.g. user X reacted to question Y on topic Z). As Booster is an event-driven framework, events are central to the design of a Booster project.
• Entities: The source of truth, representing data about the subjects discussed within your company (e.g. topics, questions, users).
• ReadModels: Public projections of Entities. While Entities are private within the system, ReadModels are publicly accessible (e.g. get topics).
• Handlers: Code that processes commands, reacts to events to trigger other actions, or updates Entities based on new events.

For more in-depth information about Booster and its architecture, we recommend checking out the official documentation.

Booster automatically generates a GraphQL API based on your Commands and ReadModels, which is how your app will communicate with the backend. This API also supports real-time updates through subscriptions using a web socket. Using a GraphQL API is easy and offers numerous benefits over traditional REST APIs, making it an excellent choice for app development!

How to use Booster from your iOS app

To make requests to the GraphQL API exposed by our Booster backend, we need to use a GraphQL library. The best option for iOS is the Apollo client.

Once you have completed the process of creating a Booster backend application (we have a tutorial on this here), you can follow these steps:

1. Add the Apollo dependency to your Xcode project

• Open your Xcode project, go to the File tab in the macOS bar, and click “Add Packages”.
• In the Add New Package window, search for “github.com/apollographql/apollo-ios.git” and select “Add Package”.
• In the new window that appears, select “Apollo”, “ApolloAPI”, and “ApolloWebSocket” under “Package Products to add”.

You can check all the other options available to import Apollo into your project here.

2. Set up GraphQL in your project

The Apollo iOS client comes with a command-line interface (CLI) that makes it easy to run code generation, which simplifies the process of working with GraphQL. The CLI will download your Booster GraphQL schema and generate Swift code for your GraphQL queries, mutations, and subscriptions.

Here’s a brief guide on how to use the CLI:

• Download the code generation CLI tool. Go to the list of Apollo iOS releases on GitHub and find the latest release.In the Assets list for each release, you’ll find a pre-built CLI binary called “apollo-ios-cli.tar.gz”. Download and unzip this file, then move the “apollo-ios-cli” binary file into the root directory of your project.
• Initialize the code generation CLI. Open a terminal window at the root folder of your project, and run the following command:

./apollo-ios-cli init — schema-name [SchemaName] — module-type embeddedInTarget — target-name [TargetName]

Be sure to replace “[SchemaName]” with the namespace used to scope the generated schema types, and “[TargetName]” with the name of your iOS app as set in your Xcode project. This will create a new configuration file, “Apollo-codegen-config.json”, in your root folder.

• Add the schema download method to the configuration file. You’ll need to include the “schemaDownloadConfiguration” dictionary in the config file and specify the URL of your schema, which is the Booster GraphQL endpoint:

• Create GraphQL files with your queries, mutations, and subscriptions. These are the requests your app will make to the backend. The Apollo code generation tool will look for these files with the .graphql file extension to create the Swift API code.

The public interface of a Booster application consists of Commands and ReadModels. In GraphQL terms, you use mutations for Booster Commands, and queries and subscriptions for ReadModels.

To see examples of queries, mutations, and subscriptions in action, you can check the iOS project. You’ll also find examples of how to filter ReadModels.

• Run the Apollo Code Generation tool. Execute this command in the terminal. It will download your schema and generate the Swift code for the types defined in it.

./apollo-ios-cli generate -f

• Import the generated files to your Xcode project. Drag the updated “graphql” folder into your project (make sure to remove the reference first if you’re re-adding the folder). You should be able to compile your project and make successful GraphQL calls!

It’s crucial to re-generate the files whenever you make changes to your Booster schema by adding or removing new Commands or ReadModels, or by adding new GraphQL files to your iOS project. This ensures that the frontend and backend are in sync and can communicate effectively.

3. Set up the Apollo client

On this project, we needed to execute GraphQL queries and mutations and receive real-time updates via web socket using subscriptions. This way, we can update our UI instantly with any change to our read models. Additionally, we wanted to have an in-memory normalized cache to avoid making unnecessary queries.

Based on those requirements, here is how you could set up the Apollo Client in NetworkClient.swift:

If you want to learn more details about Apollo configurations, you can check the official documentation.

The custom NetworkInterceptorProvider class created in step #2 can be found here. Depending on your application’s requirements, you may choose to implement a custom network transport or use Apollo’s DefaultInterceptorProvider. In this app, we needed to restrict access to Booster’s data to only authorized users, so we implemented a UserManagementInterceptor that adds the user’s access token to the authorization header in each request. However, if your application has different security needs, you may not need to create a custom NetworkInterceptorProvider class.

4. Execute a Query

For example, if we try to use the GetTopic query to fetch a topic by id, we could do the following:

GetTopicQuery matches the query we defined in GetTopic.graphql. Notice that we are taking advantage of the new async/await pattern supported in Swift. As it is not supported yet on Apollo 1.0 by default, we added an extension! You can check it out here.

5. Execute a Mutation

Remember that mutations are calls to our Booster Commands. The request is similar to the example above:

6. Subscriptions!

Booster provides support for real-time updates using subscriptions via WebSocket. We are taking advantage of Swift’s asynchronous sequences for this scenario to receive ReadModels updates.

In this example, we subscribe to any question update that belongs to a specific topic:

User authentication with Auth0

In our project, only authorized users are allowed to access and interact with the Booster backend. To ensure secure access, Booster uses the OAuth 2.0 protocol for authentication. This protocol involves the use of JSON Web Tokens (JWT) to identify and authorize users. These JWT tokens, also known as access tokens, are issued by an authentication provider such as Auth0, which we are using in this project. Other popular providers that support OAuth 2.0 include Firebase and others.

You can create a free account with Auth0 for your experiments. In our article on configuring the Booster backend, we provide detailed instructions on how to set up user roles and custom claims in the Auth0 token, which Booster uses to authorize operations. For the iOS counterpart, you just need to follow these steps:

• Create a new native application under Applications in your Auth0 account. Then, follow the official steps to configure the client and obtain the Auth0.plist file for your project. It’s important to note that we have a Build Script Phase called “Auth0 Plist Selection” in our Xcode project, under the LiveQuestions target. This script selects the appropriate Auth0.plist file based on the type of build (development or release). If you only have one environment, you can place the same plist in both folders, or simply remove the script and use a single plist file.

• In AuthService.swift, you can see how we have implemented easy sign in and sign out functionality using the Auth0 framework. It is crucial that you set the audience parameter when signing in, as a missing or incorrect value can result in an incomplete or malformed token. You can use a tool like jwt.io to check the contents of JWT tokens.

The audience parameter refers to the identifier of the custom Auth0 API that you will create for your backend. Don’t worry if you’re not sure what this means — we provide a detailed explanation of this step in the authentication section of our backend article.

Environment Configuration

Whenever you deploy your Booster development or production environments, you’ll see the backend’s endpoints URLs printed in the console. Populate the EnvironmentConfiguration.swift file with these URLs:

And that’s it! You can now easily interact with your Booster backend from iOS. You can find more details about this app and other Booster use cases by exploring the iOS code on Github.

Conclusion

By using Booster, it is possible to create an enterprise-grade, scalable application without needing extensive expertise in cloud technologies or event-sourcing. We found the reactive nature of Booster to be very intuitive and effective in boosting productivity, and we truly believe it has great potential as a solution for building modern, feature-rich mobile apps.

Booster enables mobile engineers to quickly become full-stack developers and manage the backend of their application, making them more versatile and efficient.

Booster is constantly improving. The team is working on new features that will make our lives even easier in the near future. For instance, they are developing a native authentication solution, the ability to delete events and data permanently (currently only logical deletions are possible), and simpler storage options for file uploads. Stay tuned for updates!

If you have any questions or suggestions, don’t hesitate to reach out to us on Discord!

This article was co-authored by Damien Vieira and Juan Sagasti.

From iOS Dev to full-stack in no time with Booster 🚀 was originally published in The Agile Monkeys’ Journey on Medium, where people are continuing the conversation by highlighting and responding to this story.

As a distributed and manager-lesser team, we want to make sure that each of us has the autonomy to make decisions and execute them without needing to ask the boss™ for permission.

Read boss™ here as a manager, lead engineer, VP, CEO, CTO, the Oracle of Apollo at Delphi, $DEITY, your cat, a couple of dice, flipping a coin, or any other authority figure of any kind.

But this is easier to tell than to put into practice. Some decisions should be taken with extra care and involve other team members and bosses™ to ensure that they match the overall project and company strategy.

It’s hard, especially for the less experienced people, to differentiate when they can just execute an idea or escalate it. Lack of confidence can lead you to become too conservative and end up asking your boss™ about the smallest decisions, introducing an extra overhead that will slow things down. On the contrary, if you lean towards the other side and act in a lone-ranger-reckless way, you can become a potential threat to your project’s continuity or even your company.

Jeff Bezos coined the terms “One-way” and “Two-way” doors in his AMZN 2015 Letter to shareholders. He was explaining that to make Amazon become an “invention machine,” they would need to avoid “one-size-fits-all” (conventional) decision making and learn to differentiate between two kinds of decisions:

• One-way doors: decisions that are consequential and nearly irreversible; should be made methodically, carefully, slowly, with great deliberation and consultation.
• Two-way doors: decisions that are changeable and reversible; should be made quickly by high judgment individuals or small groups. If you’ve made a suboptimal decision, you can reopen the door and go back through.

It’s easier to decide if you should escalate a decision by thinking about whether it’s a one-way or a two-way door. If you can undo an action with no consequences, it’s better to execute it. Otherwise, it’s better to make sure you’re doing the right thing and check with your team members and leaders.

Most decisions are two-way doors, so following this simple framework will make your team faster and more confident.

Making and escalating decisions as a distributed team was originally published in The Agile Monkeys’ Journey on Medium, where people are continuing the conversation by highlighting and responding to this story.

Things that matter

Booster is a highly innovative open-source framework for backend development with Event-Sourcing architecture that allows developers to focus on what really matters: the business logic that makes your application unique. Its high-level abstractions save you from dealing with the necessary serverless infrastructure and it deploys all the necessary resources for you on AWS (more providers coming soon). You don’t have to work directly with these resources at a low level (databases, connections, functions…) and it establishes a well-defined framework that is easy to understand and maintain through Commands, Events, Entities and Read Models. All this with an expressive and secure language like TypeScript.

Booster is like a serverless Ruby On Rails! The learning curve is simple: you no longer need to be an expert in cloud tools to be productive!

In addition, Mayday is a free messaging app for iOS and Android that aims to be an exclusive communication channel for sending and receiving important or urgent messages while respecting your privacy and security. This means you can silence the chaos of other notifications that we are subjected to on a daily basis and that steal our attention so that you can focus on the important things in your life. Its philosophy and functionality are very different.If you want to learn more about it, check out this article about the app or its official website.

You can download it now for iOS and Android.

From Firebase to Booster 🚀

For the first beta version of Mayday we used Firebase’s mobile SDKs and Cloud Functions in order to create a working prototype that would validate certain hypotheses as soon as possible. Firebase is a fantastic platform that saves a lot of work in many aspects, but as the backend code became more complex and we lost some performance, we decided to migrate to a more mature and versatile platform like AWS. Booster made that task a lot easier for us:

• Applications built with Booster are very fast as they benefit from the maturity, scaling, performance and potential of AWS.
• By building your business logic using a very high-level abstraction through a series of well-defined and interconnected components (Commands, Events, Entities and Read Models), the codebase becomes convenient, predictable, expressive, secure (also thanks to TypeScript) and ultimately easy to understand and maintain; key aspects for any development team. Any programmer who joins your project can start being productive in a very short time, even if they have no previous serverless experience.
• We completely forget about infrastructure configuration, which usually requires dealing with YAML files or the AWS console.
• The Booster team is very open and quick to add new functionality to the framework. They are always accessible on their Discord server, and being open-source you can easily modify and extend the framework yourself and adapt it to your needs if necessary. Furthermore, it supports a system of plug-ins (Rockets) that allow you to expand functionality without much effort.
• The creation of synergies and feedback between both of the company’s projects was another of our objectives. This allowed us to accelerate the development of Booster, which now supports functionality as a JWT token-based generic authentication system, time-scheduled commands, Read Model deletion…
• This Booster authentication system allowed us to continue to maintain Firebase for user login and save time. Firebase is still in charge of obtaining and renewing the token which is then used by Booster for all its operations. But Booster’s token system is generic and we could use any authentication provider that uses JWT tokens (Auth0, for example).
• The GraphQL API that Booster exposes to apps makes it possible to deal with data in a very convenient way and to build real-time and reactive applications, such as Mayday’s mobile apps. Subscriptions are very easy to create and Mayday uses them, among other things, to keep the Mayday list updated in real time.
• Scheduled Commands allowed us to easily meet one of Mayday’s key requirements: deleting Read Models from Maydays that were resolved after 24 hours.
• We have created a very fast contact detection system between the apps and Booster that checks from time to time if any of your contacts are already in Mayday. All this without ever storing the user’s address book on the server. Privacy is another key point.

Thanks to all this, Mayday has been able to become the first production application to use Booster. The team is still working on new features for the framework, as well as on growing the Booster ecosystem with new applications and services.

Cloud application development has become very intimidating and it has a very complex learning curve. Booster was born during our team’s process of learning about the Serverless paradigm, which faced a number of frictions related to the learning curve of the Cloud world.And it is an important step in that direction, proposing an event-driven and reactive future without gaining in complexity thanks to a higher level of abstraction.

Mayday Messaging is the first app using Booster was originally published in The Agile Monkeys’ Journey on Medium, where people are continuing the conversation by highlighting and responding to this story.