[AI] From I Code to AI code

Part 3: The dawn of a new species

One thing you quickly learn as a programmer, is to ‘think’ like a machine. Functionality is often explained in a few sentences, but it takes hundreds of lines of code to turn this into an application. It’s pretty hard to think like a machine, but in the near future there might better ways, to let the machine figure it out themselves. From I code to AI code.

The traditional workflow

It doesn’t happen too often in IT, but sometimes you hear a story about the old days. Then I hear colleagues near their pension date tell about mainframes, storage tapes and punch cards. They also speak about programming languages they used like Cobol, FORTRAN or Pascal.

What interest me is the workflow they used. They didn’t have the internet at their disposal. No online documentation, or code sharing. How did they do it? How did they got their skills, without any online resources? “We just had class”, they tell me.

After the course it was up to them. It’s amazing that there was no sharing of knowledge and they had to figure everything out by themselves. Even if knowledge was shared, it was likely within the company. Most of all they had to reason for themselves and create a lot of logic in their programs. Often they had a book beside them with explanations and examples. No copy paste here. Just a command line editor like VI or emacs, where they learned all the commands by heart (and sometimes still can outperform younger programmers).

When finished, the code needs to compile. Currently this takes a few seconds for a small project and we get annoyed when this takes a few minutes for larger projects. In the old days it could take multiple hours. To avoid mistakes and recompilation code was often printed on paper and reviewed by the development team.

So the developer workflow was something as follows:

1. Intent → Design → Programming Guide → Learn → Write pseudo-code →
2. Write code → Print code on paper → Check code → Compile code → Wait a few hours →
3. Run program

There are three phases. First phase is thinking, second phase writing and checking and the last phase running the code.

…

Today the developer workflow is completely different, because of shorter compilation times, use of shared libraries and online resources. Even when just using notepad the workflow is more comfortable.

1. Intent → Design → Write some code →
2. Ask Google → Read StackOverflow → Adjust Code → Compile code →
3. Run program

So it shifted somewhat from thinking a lot, to doing a lot. Then while doing we adjust the code with the help of online resources. It’s like people used to map out their whole route before going on a trip. Now we just go and ask the navigation while driving.

What remained in the modern workflow is that we need to think like a machine. For this we need to divide the functionality into many steps. Then we need to code every step into a procedure. Its thus still about procedures, not yet about the original intent.

You can’t yet say a computer what you want. At the end the most annoying thing about computers is that they exactly do what you tell them to do. An example I recent had was to select some names. This name could consist of

• firstname
• middlename
• suffix
• lastname
• partnerSuffix
• partner lastname

So you can have a simple name as John Doe, but also Mary Lee da Silva-Jones. It easy to make mistakes here and find yourself debugging cryptic messages.

The toolchain workflow

Fortunately there are a lot of tools to assist programmers. These are partly build in integrated developers environment (IDE) or available online.

The IDE firstly started to bundle editor, compiler and file explorer in one tool. Today they assist coders without even being aware of it. Common tasks are

• Refactoring: Restructure or rename files or pieces of code by scanning through the code project and keep the rest of the program in sync.
• Auto format: Format code to keep it readable and in line with standards.
• Code completion: Gives the user list of contextual suggestions. Firstly this was done by analyze the class you are using, but nowadays this is extended with AI suggestions based on what other coders did in this context.
• Code quality: Check for errors while writing and analyze it for best practices with linting tools
• Live reloading: Auto compilation and rerun after saving a file.

There are some additional ways to assist the coders workflow

• Version control: With versioning developers can keep multiple versions of the same file. This makes it possible to revert or merge code.
• Build tools: Build tools not only automate code, but also download and resolve the correct version of libraries from (online) repositories.
• Online resources: This goes from online documentation, finding libraries or code examples to question and answering websites like StackOverflow.
• Try it online: Used for code sharing and trying out small pieces. Think of online environments like Plunker or Codepen.

The best tools work quietly in the background. Most programmers today take these things for granted, but it’s good to be aware how the developer workflow changed.

The cloud workflow

The toolchain workflow has its disadvantages. Firstly an IDE is often heavy. It’s start slow and takes a lot of system resources. Secondly people nowadays need to learn far more than just programming, but also complete toolchains. Especially with the initial setup, but also when a tool is not working one is debugging the IDE or toolchain instead of the program.

An upcoming developer workflow is using cloud IDE and functions. The coder doesn’t need to setup the toolchain. This is already done by the cloud provider. It doesn’t even matter which compilation, build tools and deployments are used. Mostly the application is made of microservices or functions that live in the cloud. They can call each other through the internet.

This makes development easier and the end result more scalable, but it’s hard to keep the program coherent. Last, but not least it didn’t solve the problem that we need to think like machines. We still need a lot (cloud) programming skills. And the cloud are just systems.

From this premise IT specialists and computer scientists try to make programming easier by creating new programming languages that are easier to read and write. Things you used to write a whole function in language like C++ or Java can now be done with onliners in Swift or Kotlin. Considers this program in Java:

class bottles
{
  public static void main(String args[])
    {
    String s = "s";
    for (int beers=99; beers>-1;)
      {
      System.out.print(beers + " bottle" + s + " of beer on the wall, ");
      System.out.println(beers + " bottle" + s + " of beer, ");
      if (beers==0)
        {
        System.out.print("Go to the store, buy some more, ");
        System.out.println("99 bottles of beer on the wall.\n");
        System.exit(0);
        }
      else
        System.out.print("Take one down, pass it around, ");
      s = (--beers == 1)?"":"s";
      System.out.println(beers + " bottle" + s + " of beer on the wall.\n");
      }
    }
}

And the same program in Kotlin:

fun main(args: Array<String>) {
    for (i in 99.downTo(1)) {
        println("$i bottles of beer on the wall")
        println("$i bottles of beer")
        println("Take one down, pass it around")
    }
    println("No more bottles of beer on the wall!")
}

Besides totally new programming languages there also other approaches. Sometimes languages are made for a special domain (Domain-Specific Languages). These are for example query orientated like SQL or table-driven like 4GL.

From these higher abstractions also code generators were made. Mostly this is done with help of template engines. With help of these engines you can describe your program in a more declarative way and then the program is generated. This mostly takes some data and a template as input with code in for example Java as output:

There are even projects that try to combine IDE, toolchain, higher-level languages and templates engines. An example is the Jhipster project. This takes the most common backend and frontend frameworks and than let the user choose which ones to use. You can also provide a data model. The end result is a web application in plain (Java) code.

The problem with this approach (and code generation in general) is that it‘s very hard to update software. It’s hard because code gets overwritten and lots of boilerplate code is generated. Last, but not least those template engines are only targeted at certain types of applications (mostly web applications).

The last problem also counts for low-code platforms. And they even add the problem that they are proprietary platforms with a vendor lock-in. Still they are very interesting because they even operate on a higher level, by reducing the amount of code needed the write an application.

Low-code platforms, like Mendix and Outsystems, do this by offering modeling tools for certain aspects of the application. For example an Entity-Relationship diagram for creating the database. A graphical editor to create the UI and flows to create the business logic. They still offer a way to create custom pieces of code.

Let AI write your code

So low-code is a solution for specific cases, but it’s not versatile. You need a model for every part of the application. Good for a web application, but not if you want to write a game, an AI algorithm, an integration and so on. AI gives more room to the developers as they react to the users input. One low-code ‘AI way’ example is Microsoft’s Sketch2Code. This transform handwritten design sketch into HTML:

Another, already mention use case, is to let AI assist you for code completion like for example Codota or Bayou. These are completion tools for the IDE that predicts the code you will write and find examples accordingly. Another way that enhances this idea is to improve code quality. Most ‘linters’ are rule based, but for example DeepCode uses open source examples of bug fix commits.

“Instead of individual “rules” created by experts, DeepCode learns from millions of open-source commits. This creates a constantly evolving base of all discovered bugs. We use AI to present logical conclusions in the DeepCode engine. What it means for you: We not only show you what is logically wrong, but also how others have fixed the same problem.”

These approaches are more flexible than rule and model based approaches as they:

• React on input
• Support multiple languages

How good they are depends of course on the number of code that is online available. So Java will probably work better than a new language like Julia. Ironically (like low-code), though they depend on open source, they are not open or freely available.

These approaches are still more assistants than that they really write code. Also they are specifically made for coding, so more narrow machine learning algorithms than AI. The next level of abstraction might be to drop these altogether and just use natural language.

Recently a new AI platform was released GPT-3. This is a language model developed by OpenAI. This organization was originally funded by Elon Musk and Sam Altman but now also backed by Microsoft. Instead of using just code as input it uses millions of (unstructured) texts. The complete Wikipedia is just 3% of its corpus.

Normally a lot of these texts used for machine learning is labeled. Instead OpenAI uses neural networks to label itself. The GPT-3 has currently 175 billion of parameters to optimize the output. This output can be all kinds of texts like a poem, a mail, a book, a translation and code. It could even write an article about itself.

Let’s return to our original image of some robot that writes on the keyboard:

This image illustrates that this new technology is getting out of hands. Not in the apocalyptic sense of movies like “I, robot”, but in a literal sense that we as programmers don’t need a keyboard anymore and let programmers do their work. This is the direction that GPT-3 makes possible. Something a lot of IT people won’t thought was possible. Yes, IT and AI will automate work and make jobs obsolete, but there sure much be someone who creates this software. Indeed, but it may not take the 55 million IT people like this is the case today (and will rise to 62 million in coming years).

Code is just one of the many possible outputs. This makes GPT-3 very very versatile. It’s not attached to programming or a specific programming language. Because it has no real knowledge about the world the output feels more like a ‘trick’ than targeting at a good result.

It seems like we could better give the AI systems some rules, models and labels, like we use to did. However Rich Sutton disagrees that at the end things in AI will be solved by adding more computation:

“The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin. The ultimate reason for this is Moore’s law, or rather its generalization of continued exponentially falling cost per unit of computation.

….

Seeking an improvement that makes a difference in the shorter term, researchers seek to leverage their human knowledge of the domain, but the only thing that matters in the long run is the leveraging of computation.”

The Bitter Lesson

There are multiple reasons why GPT-3 is more interesting than previous attempts:

1. It really produces code (not just assist)
2. It’s very versatile and can be used for specific domains
3. Microsoft/OpenAI have a lot of computation power (currently one Exascale was used to create GPT-3).
4. Microsoft owns Github which host most open source code repositories (more than 100 million) this give very direct access to this source.

It’s still does not make programmers obsolete, but it is good to see where AI for coding is going. It may need some mixture between real world input, but also validation rules like rules of games so that it can optimize output like for example AlphaGo does. For GPT-3 you can apply for their closed beta program.

Part of the series:

[AI] Homo Sapiens Technosis

[AI] AI Universalis