How Software Engineering and Data Science are Different

Credit: Double-M

An excerpt from AI for Business Leaders.

Your AI team needs a combination of software engineering and data science skills. Because data science is relatively new many tech leaders do not appreciate the differences between the two.

I’ve hired and worked with both. Here are some of my general observations — of course there are many exceptions.

Data Scientists

Data scientists usually have a hard sciences background such as physics, bioinformatics, economics, or computer science. They almost all have college degrees.

Data scientists excel at answering hard questions using the scientific method: generating and testing hypothesis, evaluating results. Many teach themselves programming R, Python, or MatLab to run experiments.

Data science work (1) is self-directed, (2) relies on experiments, (3) has fewer fixed deadlines than product teams. Data scientists excel at coding quick hacks to implement mathematical functions or transform data. Most can take abstract concepts and quickly code data models — a skill which requires brains and years of practice to master.

Data scientists produce models, data sets, papers or answers to business questions. Data science looks like magic to many software engineers.

Software engineers (professional programmers)

Programming is not computer science. I’ve worked with great programmers who have studied every subject in school. Some of the best didn’t go to college.

Professional programmers excel at making products — it is a social skill and great programmers can build products as part of a system. Their work is (1) interdependent with the rest of a product team, (2) deadline-driven, (3) results-oriented.

“It works” isn’t good enough — experienced programmers create robust, testable, readable code which makes the whole system better. They constantly search for ways to refactor their own code to make a better system.

Programmers make real products — a process which looks like magic to many data scientists.

Build your AI team with professional programmers who are self-taught data scientists

Your AI team is building enterprise software systems — not doing research. Rely on transfer learning so your team only needs basic data science skills to get started. Programmers with the aptitude and desire can learn basic data science through independent study and side projects.

Software engineering skills must be learned on the job. Programmers only appreciate software engineering best practices by experiencing product failures: unreadable code, brittle systems, or bugs.

You don’t want your team learning these hard lessons on your AI product. Start by finding programmers who have demonstrated initiative and commitment to become AI engineers.

Then selectively add advanced data science/AI modeling expertise as your product becomes more sophisticated.

Examples from my work

Here are two different examples of implementing the same function. My goal was to re-create the results in Perceptual Losses for Real-Time Style Transfer and Super-Resolution.

I built a neural network which creates a higher-resolution image from an original low-resolution one. For example, I took the blurry breakfast on the top and created a higher resolution version:

Low res image

High-res output of my neural network

Data science approach

I needed to implement the Gram matrix function described in the paper:

Here is the Python function I used in Jupyter notebook:

def gram_matrix(x):
  feat = K.batch_flatten(K.permute_dimensions(x, (2, 0, 1)))
  return K.dot(feat, K.transpose(feat)) /   
    x.get_shape().num_elements()

As you can see from results above … it worked. I was able to successfully achieve my goals of recreating the result. The Gram matrix function is quick, compact, and I understand it.

Software engineering approach

Unfortunately the Gram matrix code above has a number of problems which make it unacceptable for a production system:

1. It is too hard to read,
2. It is hard to test,
3. It contains “magic” numbers like (2, 0, 1),
4. It isn’t reusable.

Yuck.

A better (although far from ideal) approach is following the approach in https://github.com/titu1994/Fast-Neural-Style. This repository is designed for reuse and readability.

Get a free early copy of my new book

Wondering “what do I need to know about AI?” but don’t have time for courses? Get simple, practical AI advice for product & corporate innovation teams. Get your free copy here.