Two years ago I was doing a startup with my friend Ming building mobile apps. We hired contractors in Ukraine to work on the UI, and wrote a lot of UI specs. The specs had to be very detailed to avoid emails back and forth. We used to joke that they were so detailed, that you can almost generate code from them.
Of course things are not that easy. A lot more questions would surface during development. How should certain transition be animated? Is it OK to trade occasional data inconsistency for much improved performance? The fact is as detailed as the specs are, there is still a lot of information missing.
The difficulty in generating code from spec is not in generating code, but in knowing people’s intention. If we force the author to give all the details, then we are back to what we are doing today. For the process of software engineering is an iterative process that converts the high level abstract spec, into a low level detailed spec (programming language code) that can be compiled into actual machine code with no ambiguity.
Several months ago, I looked at the problem again. I still want to find a use case where we can potentially generate software directly from an existing spec. This requires the following.
- In this use case people’s intention must be clear.
- A spec exists today so there is no extra work.
- The spec has all the information.
For the first requirement, software testing fits the bill. People’s intention is very clear in QA.
For most of QA use cases however, there is no formal spec. I’m not interested in those cases. There are automated QA methods such as property-base testing, but you have to combine those with some example-based testing to exercise common scenarios. My interest is in replacing the example-based testing with automated methods.
There is one area that I have experience in during app development that has a formal spec - REST API. Because a REST API is a common integration point between teams, it’s usually well documented.
I used OpenAPI (Swagger) to document API before. If used properly, an OpenAPI spec has all the information on how to interact with the API. However, not every piece of information is formally structured. For instance, a parameter can come from another object’s property, and this information would be either documented in the description field in free text, or in the parameter name itself. An example would be an online Order requires the Id of the item the user is ordering. The Order may have a field called item_id, with no description, or it may have a field called source_id, with a description “the item we are ordering”. In both cases the intention would be clear to a developer reading the spec.
Now it seems generating QA software from an OpenAPI spec would be feasible. It satisfies all three requires listed before. The only challenge remaining would be how to really understand the information inside the spec. The NLP technology today can parse out sentence structures, but can not understand the meaning of sentences.
The saving grace is again that in this limited context, we know the intention. The description field of a parameter would be trying to describe the relationship of the parameter relative to the other named entities in the same document. In this context, the problem is much simpler. I used an open-source NLP library called spacy to break down and normalize parameter and path names (e.g. orderId is really two words: order id), and leveraged the existing formal structures in the spec to help understand the unstructured part of the spec.
I published my work here under MIT License — https://github.com/meqaio/swagger_meqa. It’s very much a proof-of-concept. Not all features of OpenAPI are implemented, just the most commonly used ones. The NLP processing is still very simple and didn’t leverage the advanced features today’s NLP libraries are capable of. Still, the result is kind of interesting. For the few public APIs I tried it on, it gets things about 70% right.
I’m hoping that you can give it a try, and let me know whether it works or doesn’t work, and where it breaks down. I’m also interested in knowing your thoughts on how to generate from a spec in general, and the other areas you see this feasible.
