The case for Semantic Programming
There’s an incredible shortage of Software Engineers, let alone skilled and experienced Software Engineers. Looking at the number of new graduates, or even kids and young adults that are truly interested in Software Engineering, this isn’t going to change. On the contrary.
Software is starting to permeate more and more of our lives. Nobody had a smart phone 15 years ago, and now everyone does, and everyone has access to bazillions of app. Software is in our cars, all of our money relies on software, and most of our businesses too. Companies are turning to software to provide digital services, so that they can move up in the value chain meet the consumer’s needs instead of just selling them a product.
But where will they find the developers?
This problem is exacerbated by the fact that developing software, good software that is reliable and well-designed for the user, is very hard. Very, very hard. There are books, educations and methods, but we don’t have a method yet that results in guaranteed success. So next to all that software that has to be produced, it takes a long time to get it right.
We expect that in a decade, maybe two, the demand for Software Engineers is so high that everyone should be a Software Engineer to meet it. Which is not realistic of course. The only way to solve this is to make developing software better. And not better in the sense of creating a new web framework that might offer slightly better performance or developing efficiency, but orders of magnitude better.
The only sensible and realistic conclusion we can reach on how to achieve this is through Semantic Programming. To introduce this properly however, let’s first take a look at regular programming practice.
Regular programming practice
In regular programming practice, a Software Engineer (or a team) usually takes roughly the same steps to develop a system. These are the following:
- Talk with the end-user, domain expert and/or product owner, and figure out what his problem or vision is;
- Given the problem or vision, think of a solution or realization;
- Design a system that solves the solution or realization in your preferred way (be it through mock-ups, flow diagrams, BPMN, etc., or through quick prototyping);
- Iterate over the design with the end-user or product owner;
- Translate the design to an implementation, which is a programming language that a machine can understand;
- Continuously iterate over the implementation to verify that it fits the design;
- Iterate over the design and implementation with the end-user or product owner, to see if it still solves their problem or realizes their vision.
In terms of tooling, most attention is usually paid to the implementation part of developing a system. In education, a disproportionate amount of time is also spend on implementation instead of on requirements engineering and problem-solving. But research indicates that most of the issues occur exactly there, in the human-to-human translations.
This shouldn’t be a surprise to anyone who has ever played Chinese whispers (or telephone in the United States). For social animals, we are terrible at listening and comprehension. We have to be taught to reflect thoughts back to the speaker, to summarize their points, and to point out inconsistencies as we go. And these are not simple systems we are designing either; typical software systems contain dozens of processes and data entities, for which the human mind is not properly equipped to keep track of.
But the problem is worse than this, because designing a system and solving a problem isn’t simply a matter of listening properly. What we do when we listen and design is we model the reality and thought-processes of the end-user into programming concepts. We translate things from what for him is logical in his mind to what is logical in the programming world, and perhaps more importantly, what is logical for us. Because our mind almost always works differently from the people we are talking to, things that are logical for us are not for the end-user, and perhaps don’t even fit in their domain.
This problem is typically tackled by adding a design step (which I’ve included above but in reality is often omitted). This step exists to verify with the end-user that the design solves their problem, before resources are committed to the implementation. It is a way to gather feedback quickly, so that potential communication issues are found as early as possible as well.
The main drawback of the design however is that it is not and will not be a reflection of reality. In the end, reality is created by the programmer, who has to perform yet another translation. There is really no guarantee that the actual implementation will fit the design, and while there is tooling for the creation of a design, there is no tooling to create safeguards for this. But even if the implementation fits the design, the design is merely a model of what the implementation might look like, not the actual implementation. During implementation (and during use!) new issues will appear that might affect the design or even the problem that it was designed to solve. There is no way to detect all of this without having the full implementation.
So to design a good system that solves the end-users’ problem, you should not only be able to build the system for the end-user, you ideally have to be the end-user. Furthermore, to verify that the system solves your problem efficiently, you have to be able to use it nearly instantaneously after the designing it. So now the question is, how we can we make the distance between the end-user and the implementation as small as possible?
Enter Semantic Programming.
Semantic Programming
In Semantic Programming we don’t program for machines, but we program for humans. A Semantic Programming Language should only contain grammar from the domain of the domain expert. So we don’t talk about technical concepts like database tables and fields, but we do talk about the data required to perform a process step. We also don’t talk about how data is modified, merely about what it means when data is in a certain state.
For instance, we can say that an item is in stock if it has a lot number. We can also say that there is a process where we receive a message from the warehouse containing a lot number, and that this lot number should be stored for the item with the given reference. And then we say what it means for an item to be in stock: for example, we can say that our web shop only shows items that are in stock.
These are all terms and concepts that an end-user will understand. Therefore, the end-user should be able to design a system himself with these terms. The ultimate goal for Semantic Programming is to allow the end-user to model a system by himself, through tooling that is easy to understand and use, in the language that fits his domain.
And from that model, it should be possible to generate a working system that can immediately be used to solve the problem of the end-user.
This is a very ambitious goal. There are a lot of challenges that have to be overcome to reach that goal, and the last 30% of getting there are the hardest. And for some of those challenges, no solution is in sight yet. So let’s make our goal a little more realistic instead.
What we want to achieve is to make the feedback between end-user and system as small as possible. We expect that it is going to be very hard to get to the point that an end-user can do this himself, and we accept the fact that we’re going to need, at least for the foreseeable future, people whose specialty it is to design semantic systems that solve an end-users problem. Furthermore, we also accept, for now, that it will not be possible to generate the entire system; instead, we will generate a sequence of forms for every process, as well as a skeleton implementation and tests for the final implementation.
So our intermediate goal is that to be able to have a moderately trained user be able to create a working system modeled as a sequence of forms using a Semantic Programming Language from the domain of the end-user, that can immediately be used to verify all processes, that can be used as a skeleton for the implementation and that generates sufficient tests to verify the final implementation. Finally, it should be possible to modify the design, which modifies the implementation as well.
This will lead to the following, modified process:
- Talk with the end-user or product owner, and figure out what his problem or vision is;
- Given the problem or vision, think of a solution or realization;
- Together with the end-user, design a system that solves the solution or realization using a Semantic Programming Language;
- During Semantic Programming, verify the processes implementation immediately;
- Generate a basic implementation plus tests from the Semantic Programming Language;
- Continuously iterate over the Semantic Programming to verify that it still solves the problem (and generate a modified implementation if it doesn’t);
- Implement the functionality that is not yet generated, taking into account the constraints from the earlier steps.
The main difference with the current programming process is that we will move away from using the implementation as a model for the system. Instead, we use the Semantic Programming Language as a model, which is much cheaper to create and modify, and which can be read and understood by the end-user as well. This will allow us to gather feedback on the actual implementation much faster, and it will speed up development by a large factor as well.
Conclusion
We are currently working on tooling within the Web11 group to support Semantic Programming. We are also putting this tooling into practice with our current customers, and we’ve received great feedback already. Our initial findings is that while the up-front effort increases (which we’re trying to limit by better tooling), it decreases issues in the long run and leads to systems that actually do what the users want it to do, solving their actual problems and supporting their real processes.
Now, we are very much aware that this is an ambitious goal. Surprisingly, however, it is also an extremely controversial goal. Whenever we talk about these ideas, we immediately get a lot of push-back. We are told that it is impossible, and it will never happen. It is too difficult, and things are fine the way they are. History however has taught us that this might just mean we are on the right track.
