What is the best framework around here? Which one should I use to build my next project, startup, or to make sure my mission at my next customer goes well?
Those are questions many developers are often asking themselves. A framework is the toolbox, the foundation we are using to build a project, and it heavily defines the quality of our work.
It often makes the difference if we will or not be able to deliver, and it’s so easy to get lost among the many choices we now have.
Before being able to answer this, we shall ask ourselves, what makes a great framework?
In my opinion, a great framework is first a framework where you can be productive. It shall take care of the most common operations so the developer can concentrate on the specific needs of his project.
Also, you have to wonder, will this framework be able to scale. Will you be able to build your next multi-million users startup with it?
The ultimate framework shall be good at both productivity, to make a proof of concept in just a few days, and scalability, to improve this proof of concept later without having to refactor everything. Many frameworks out there are good only at one of them.
Finally, the last point is often overlooked but is nevertheless really important : will you be able to create autonomous teams with this framework? I mean, when the project will begin to scale, will you be able to split the responsibilities into several teams, and still be able to make them correctly communicate and work together.
Being able to split the project into several teams is incredibly important, this is what unlocks the real flexibility in a company. It’s what brought success to many world-scale companies at Silicon Valley, and what we really mean when we speak about microservices concepts, two pizza rule, etc…
The ERP frameworks
Before going further into the subject, we should have a look at one kind of framework which is especially interesting : The ERPs. For those who are not familiar, an ERP (Enterprise Resource Planning) is the common name for an integrated software managing the most critical data of a company processes, like accounting, stocks, expenses etc….
The bigger ERP selling company is SAP, bringing decades of best practices for managing the most common processes in companies.
Another popular ERP is Odoo (ex-OpenERP), this one is open-source and written in Python with a PostgreSQL database. Odoo, of course, is far from being as mature as SAP features but there is one point where Odoo is often underestimated : its framework is one of the best in the world for what it does. Entirely open-source, with a strong and well-organized community, Odoo is a solid choice to build the heart of your architecture, your company’s processes, where your employees do their work.
What we are expecting from an ERP framework differs from a regular web framework :
- Scalability is not really necessary, we usually only have hundreds of users (the employees). Of course, it’s important to be able to spawn several servers to horizontally scale if needed, but you shall not have to build a whole farm of servers just for your ERP.
- Since employees only use it, the user interface should be pragmatically and intuitively designed so each resource can easily be found. We’ll call such unified interface ‘admin interface’ in the following of this post.
It differs entirely from what a web framework usually deliver, where you want to have complete liberty of what you can design. In an ERP framework, you want to spend the less possible time coding the interface. - The framework must manage all security aspects. The list of the users, how they log in, the access groups, and which resources they are allowed to access, following complex rules.
- And in general, you shall be able to program new features the fastest possible way. Developer productivity and maintainability are everything here, because it’s directly linked to the flexibility of the company itself. The ERP framework must bring all the tools needed for the developer.
On all those aspects, this is where ERP excels, but this is not enough anymore. Maintaining an ERP is not that easy, and I believe the best practices which emerged these past years with containers, clouds, microservices architectures will completely change how we architecture them. But for this, we will need to leave behind the old frameworks and build new ones.
The challenges of autonomous departments
Don’t get me wrong. If you need to build your internal process quickly ERPs are still some of your best choices, but all ERP from this generation completely overlook one the points I spoke earlier : being able to have autonomous teams. And unlike scalability, this point is crucial for a software managing the internal process of a company. One book especially opened my eyes on this, Building Microservices from Sam Newman.
Despite many successful projects, ERP projects are well-known for their cost (millions), their delay (years) and for not delivering what the users hoped. An ERP is usually one software, one codebase, it must be maintained by one team. When you install an ERP for all the company, it means that all departments become dependent of the team which maintains it, whether it’s the IT department or the contractor which is integrating it.
Often the developer is not even physically working inside the company. This means over-specifications, too many meetings, and in the end unsatisfied users because there is no way all the information can get through all this.
In an age where everything goes faster and faster, the company flexibility becomes a question of life and death, to adapt to a market ever changing and challenging.
And this flexibility is primarily defined by how fast you can evolve your internal processes, and by extension, your internal applications. This is how important ERP challenges are, and I can tell integrating all features into a single software is not the right way to architecture them, the codebase goes bigger and bigger, difficult to maintain, and dependent on too few people.
IT is important for everyone now, the departments needs to be able to manage their tools if they want to, so they don’t have to depend on other people for the features they need. Many departments already started using cloud services, which are great, bringing a lot of values, but outside of the control of the IT department they also bring back old challenges, compatibility, data duplication, the lack of security etc…
Sometimes they may want to develop their software themselves, but they should not be left alone.
The job of IT architects, working in the IT department, shall be to carefully select the technologies the other developers are allowed to use, provide the libraries for them so they can have the same productivity you have with an ERP framework, and ensure they can easily communicate with other services. They shall provide an environment where the intra-department developers build the features they need by themselves, in a reliable, secure, hosting platform with tracing, monitoring, and a full DevOps process.
Many companies want to go into this direction, but right now, you still have to build everything by yourself. Granted, all the technologies are here and open-sourced, but all the integration work still needs to be done, there is no turnkey solution which integrates everything you need to deploy this environment.
This is what needs to be built. The definition of an ERP is not a single, integrated software anymore. It’s the technology you use to develop your internal processes, and I firmly believe that a new generation of ERP frameworks will soon appear.
I don’t know what new fancy name we will give it, next-gen ERP, cloud native ERP, but I know what they will bring : autonomous development teams in each department. It will not be a framework anymore, but a stack, straightforward to deploy, which will provide a lot of libraries to help the architects and the departments developers.
What about web applications
At this point you may think, ok, this post is about internal management applications, I can’t build a website with it. This is where you’re wrong, the goal of this post is not about discussing the best next ERP system, it’s about how to build a stack with the development productivity of an ERP framework, but with a good design I’m sure it’s possible to keep it scalable! There is not that much difference between an ERP framework and a web framework.
Let’s take Ruby on rails for instance, which is probably still the most well-known web framework. There is a lot in common between an ERP framework and the Ruby on Rails framework, especially they are both extremely good at managing CRUD (create/read/update/delete) operations, with Active Record for Ruby on Rails. And this is important, they are 80% of backend operations, when you have a good CRUD framework you can easily prototype the foundation of your project, whether it’s an ERP or a web application.
An ERP framework already manages CRUD features beyond the best of the web frameworks. We would only need to add scalability and a fully flexible frontend.
For the frontend, the answer is to use headless architecture. This means separating backend and frontend, so the frontend developers can use their technologies of choice, and make them communicate through API.
As for scalability, the choice of language will be important. If we compare to Odoo which is written in Python, or even Ruby on Rails in Ruby, they use script languages, very slow at execution. Of course a lot of great web applications are written in Python and Ruby, but nonetheless at scale this makes a big difference in the number of servers you need. Javascript with Node.js is better but it’s a single-threaded language, having his own issues for scalability.
This is why I advocate for some new generation language, like Golang which is statically typed, good at multi-threading and extremely fast (C-like speed). Also, and that’s a big deal, most of the new cloud-native tools built those past years are written in Golang, and I’m sure this is the community where we want to build such an ambitious stack.
Batteries included
In the end, both Odoo and Ruby on Rails became popular because they were “batteries included” frameworks, above all willing to provide all the best ideas to ease the life of their developers. So, given everything we said, what would be the batteries we want in our stack ?
In common with an ERP framework:
- A solid ORM which makes CRUD super easy and intuitive.
- Database schema migrations
- A full featured and secured API, automatically built on the ORM
- A unified administration interface for the employees. This one is a challenge because we want to have independent services, but still want to have a centralized interface to access their data.
- Everything we need to securely log in and define the access rights to data.
In common with a web framework:
- Being able to have a backend one can scale horizontally and confidently expose to the Internet.
- An API which can be used by any client, like a web frontend, a mobile application, IoT etc… Using well-known standards which can easily be used by external developers.
- Provide libraries to help build and design those web frontend, mobile application, IoT etc…
And finally, what we need to make it a real microservice, cloud-native framework/stack:
- Use other services as potential data sources.
- Register and publish asynchronous events with CRUD operations, to implement audit, event sourcing, and saga.
- Provide libraries to interact with external cloud services easily, so a department can choose between internal development or the cloud service of their choice.
- Centralize public access to services through API composition.
- Provide a centralized authentication service, managing users and access rights.
- Use a fast and secure protocol and architecture for inter-service communication.
- Shall be deployed on a container orchestrator, like Kubernetes.
- Provide a full DevOps process, including CI/CD with contract-driven testing.
- And finally a microservice chassis, to provide the observability we need to find the source of a problem in such a complex architecture. From logging, to metrics, health checks, and distributed tracing.
I know there is no silver bullet in IT, but still this would be my dream, the perfect framework/stack capable of managing and scaling every kind of project. With this you could manage both the architecture of the largest company, thanks to the loose coupling, and the smallest personal project, thanks to the productivity and for being a turnkey solution.
Being able to build a whole IT architecture alone is essential. This is how promising students can learn the best practices, how a startup can test an idea before raising money, how NGO and associations can maintain their IT cost at a minimum because they can find volunteers with real skills.
More and more open-source technologies appear every day, but at the same time state-of-the-art IT architecture has become more and more complex, we need to build a turnkey, opinionated, stack so the best practices stay accessible to any developer.
In this first post, I presented my opinion on the actual state of frameworks and stacks, from the frameworks being too monolithic to the actual stacks being still way too difficult to integrate and use. In the second post, I’ll go deeper into the technical side and say how we can build this stack and which technologies we could use.
Some prototypes have been built, but right now this is mainly just a vision. No money behind, just what I dream to work with. If we want to have a chance to make it happen we have to create opportunities, you can contact me directly, join the chatroom on http://empower.sh, or star the project on https://gitlab.com/empowerlab/example.
Follow me for part 2.
