Pix2Code: Automating Front-End development

Théo Szymkowiak
May 29, 2017 · 3 min read

Certainly not the end of Front-end developers, but it is a good start.

Paper link: https://arxiv.org/abs/1705.07962

Several days ago, Tony Beltramelli published pix2code. Pix2Code aims at automating the mockups-to-code process using Artificial Intelligence.

Training example from the Paper.

The generated code is not really compilable, but it can be converted into Web, iOS, Android or anything that can be used in an application.

Training

  • A LSTM (Long-Short Term Memory) network that takes a sequence of tokens as input (tokens being terminals in the language chosen)
  • a CNN (Convolutional Neural Network) used to engineer features out of the mockup.
  • A LSTM’ that takes (q_t, p) as input and outputs a Softmax vector of tokens. The result (x_t+1) is then fed into the next iteration. The output vector is a mapping of probability to a terminal token.
Description of the training network from the paper.

The restricted language complexity of the DSL (Domain Specific Language) allows hot vector encoding at the input and a Softmax vector at the output. The LSTM network is fed with a window of tokens of size T (the paper specifies that T=48 is a good window size following empirical tests).

Sampling

Inference network from the Paper

The empty state is an array of 48 vectors with the last one being <START>. The generating process ends when the generating network outputs the <END> token.

The generated DSL code can be compiled into a target code such as HTML (Web), Objective-C (iOS), Java (Android).

Generating code from a GUI mock up

Results

Loss curves
Ill-generated code.

Restrictions of the algorithm so far:

  • No dataset of GUI to code is available online
  • The code is not provided (yet) by the author
  • The code generated is restricted to a subset language (DSL)
  • The colors and text are not captured by the AI

This paper is a good start and an interesting use of Convolutional Neural Nets for writing software. This echoes with some other recent papers (DeepAPI, DeepCoder) that hope to automate some parts of software engineering is the near future.

Interestingly enough, to my knowledge, nobody has been able to generate front-end code using Adversarial Neural Network so far. Surely we’ll see more of those in the short term.

Théo Szymkowiak

Written by

@McGillU CS Grad. Founder, McGill AI Society (https://mcgillai.com). Full Stack Dev at Clay (https://base.run). Entrepreneur.