On Software

Incorporeal

The thing with software is that, it is incorporeal. It is thought made flesh. You cannot touch it. If software was like a painting, then anyone could see it and appreciate it. If it was a car, then you could take it out for a test drive.

Now consider software. You build a website. It is binary information which the computer reads and executes. The computer does not care if your software is written in Ruby on Rails or React. Nor do most of your users. Most people eat food, without concerning themselves with how it is made. It is the same with software.

With software, users never see the whole thing, unless something goes wrong. It is an iceberg, only a small tip is visible.

We developers think of our code as Art! But how many users are looking at your beautiful Ruby DSL, semantic HTML, nicely organised CSS or checking if your API is RESTful? Not many, or mostly nobody.

Needs to change

And the second problem is that, software needs to change. It will change while it is being written and after it is written. What needs to change itself will change. Change is a constant. It is relentless. And frankly, sometimes it is also tiring.

To continue our previous example. If a car was like software, then the blueprint will change while a robot is building the car. And once the car is finally built, then we will need to silently upgrade the engine and change a flat tyre, and all this without interrupting the user who is driving.

This change is possible at all, because software is not physical. It is thought made flesh.

Then ideally you would think that, software can be changed at the speed of thought as well. But that never happens.

For software to change, it needs to be understood and we developers need to be confident, that the change we make is correct and will not break anything.

I think software is hard to change, mostly because, programming is a human activity and humans are imperfect. Programs have to be grokked and changed by humans. We have to keep a mental model of the program in our heads and change it. And our working memory is good for only so such.

And because programming is a tower of abstractions built on imperfect foundations. Software needs to run on older browsers or future browsers! And take into accounts browser bugs. Sometimes it is a wonder that it runs at all.

And lastly because the future is unknown. We make choices while programming and these choice constrain us. It is a bet against the future. And mostly bit-rot wins the future.

Unique snowflakes

Each piece of software is a unique snowflake.

Software can be built from scratch but it is mostly assembled from existing components. And then customised and changed. There is a specific requirement or goal in mind. Otherwise users would use an existing software. This makes developing software equally rewarding and terrifying.

A commonly heard counter-example is Architecture, concerning physical buildings. This is all hearsay, as I am not an architect. But evidently, architects construct buildings from well-tested old blueprints. It needs to be vetted and stress tested, so that it can bear a certain amount of load. Generally architects will not start completely from scratch. Example, imagine building an underwater city, populated by people from Mars. But this can be built in software!

Buildings have a physical cost, if they collapse people will die. Also they take physical effort to build and are difficult to change. A skyscraper does not change into a villa.

But not so with software. Because software is easy to construct and change. People expect all sorts of unique requirements and changes. Your website is probably different from the other websites in all sort of unique ways.

There is no physical cost for software. Change is easy and can be fast. And requirements also change. This makes estimating and building software difficult.

Generally we never goto a store, buy a piece of software like we buy a wall socket and expect it to work with all our devices. To be fair, even then voltage and sockets may differ from country to country. But at-least there is some standardisation.

In software, on paper for example, web technologies like HTML, CSS and Javascript, are standardised. But they do not work consistently, across all the browsers your users are using. This is not just the web. This is true for all software. Your Oracle specific SQL will not work as-is on MySQL.

Change is constant. Computer languages change and go in/out of fashion. And so do language frameworks. The essence remains the same, but the painful part is that, it’s essence is not captured.

For example, a typical web form will:

1. accept input
2. validate input
3. show error messages and warnings
4. submit the data and wait for response

There are a lot of subtle gotchas and best practices. eg. Always post a form using HTTP post/put and not get. How to serialize the fields ? Use HTML5 number input for a better experience on mobile. etc.

But there is no blueprint for building a form. We cannot even consistently validate a HTML5 form across all the browsers!

All this becomes muscle memory and does not carry across the various languages and frameworks we use. There is no common blueprint or library for even the most common essential ideas. The programmers 10 years back were doing the same things as today. But if you were using PHP back then and React now, then none of that code can be reused. And even now, programmers have no common abstractions across the various frameworks and languages. A form built in Rails, is different than a form built in React.

This is common across all software. Not just for something as simple as creating a form. Not that building a good form is simple. Far from it.

This makes software difficult, because there are no common primitives which have remained unchanged over a long period of time, let’s say 10 years. And even when there is a standard, it is changing and the implementations are quirky.

So we build our bespoke custom requirements from un-reliable primitives. And all this while, our requirements and primitives are changing rapidly.

It is chaos. The best we can do, is to have fun along the way and to keep up with the changing technologies.

People and code of ethics

How do you become a professional software developer ?

Maybe you join a university or a vocational course and decide to formally learn programming. Or you decide to study engineering and learn how to build physical bridges. Or you take up english literature and learn to appreciate the well written word. Or you pick up the guitar and learn how to play music.

Whatever. It does not matter.

Most universities do not have a clue, about how software is actually built. Also they never and cannot teach the craft of programming.

Lab exercises never need to change, because of seemingly random requirements. We never communicate with actual users or our co-workers. We never clarify requirements. Or push for a changed requirement because your change is evidently easier or bug free. Or fix bugs reported with incomplete or buggy information by a real user.

They never teach you how to communicate or sell your idea to others. Or how to understand, debug or change a 10 year old codebase. They will never teach you how to ask questions on the internet. They will never teach, that you need to be aware of office politics. They generally never teach about how to write good code. And will never teach you how to use version control and why version control is important. Can you imagine writing an exam on git and being quizzed on its’ command-line options!

More importantly, they will never teach you how to learn a technology which is not popular. Ruby and Rails, Nginx, React and Webpack are popular now. But somebody had to try them out, when it was all new. When it was a risky choice. They could not Google it or search on Stackoverflow. They probably had to ask on the mailing list or read the source code. More importantly they had to evaluate the software and decide to use it in production. Then more importantly, they popularised it by writing blog posts and books about their experience and giving talks at conferences.

They can never teach the joy of programming. Feeling proud of something you built, enjoying the company of your co-workers, being responsible for some software. This cannot be taught. Only experienced.

They also cannot, teach you the bad parts of programming, slogging through the weekend, loss of motivation, etc.

They may teach an ideal but the real world is mostly violently different.For example, I remember being taught the waterfall model of software development in class! Good luck with that.

Also they have to teach a wide variety of subjects, most of which will never be used by everyone. I remember learning and being tested by writing an essay on Bresenham’s line algorithm. An essay! And on something which I and none of my classmates have used since passing out of college.

Universities arguably teach how to learn itself. But in my experience it is mostly bookish outdated knowledge and being tested using outdated methods. For example, an exam problem might be, write C code to update a doubly linked-list, on paper! Never mind that, I have never written code on paper in my job. I would use an editor like emacs, a linter and a compiler to check the code I write and maybe tests. The code I write on paper will be non-standard C code anyways and will never compile. Also, I have never written production C code, after passing out from college. And I never had to implement a linked list after college as well. That is probably because, nobody in their right mind will ask me to write a list routine for their new computer architecture. And like a responsible developer, I would use an existing language library anyways.

By the way, all my teachers are good people and I would be friends with them, if I met them socially. I am sure they all tried their best. And it is not as if we students were kind, understanding or polite everytime.

So how do you become a professional programmer ?

Weirdly enough. You become a programmer by programming. How meta! By studying and practicing by yourself and on the job. And if you are lucky, apprenticing with someone who has travelled on the road you are on. And is willing to guide you.

Also there is no universal code of ethics for software and programmers. You are free to release your todo app written in React Native. Without worrying that it might cause harm, by maybe losing the users’ data or say exposing their personal information to hackers.

There are “best practices” and cultures like TDD. Uncle Bob and others have spoken extensively about it. But it is not universal. I am forced to admit, that I do not use TDD or write tests in my current project, that I am consulting on. Mostly because, it is a legacy project without any existing tests. And writing tests for an existing project is just hard.

There is no organization which trains and tests programmers. Everyone has to figure it out and experience it, on their own.

A good example of a certification process, is Chartered Accountants or Doctors. They have a formal list of things to be learnt, regular tests and code of ethics. They recognise members of their profession, award excellence and disbar offending members. You can call yourself a CA only after passing their exam. Maybe they have trade unions, pensions and conventions, not sure. And evidently some organisations grade on a curve as well, so that the market is not flooded with new members who will compete with the older members.

Personally, I am not sure if I would pass such a certification! What if maths or statistics is a requirement ? It is scary. Or even if, I would enjoy undergoing such a certification process ? It seems formal and goverment-ish.

There is no book or set of courses which will teach you how to build software and to use all the libraries and frameworks. And all the courses will be outdated in six months anyways. You cannot just take a course, pass a test and call yourself a software developer.

But there is a huge demand for new software, for customising software, for adding features and fixing bugs to existing software. Software is evidently eating the world. So they say.

So there is no formal organisation or code of ethics. What this means is that, not everyone you work with, or even your past and future self’s will understand or enjoy the whole stack and the product. The software you write/maintain and the software you depend on, will be of varying quality. We will need to learn new things and it will be painful.

Technology is basically unknowable in it’s entirety. This is true for any human field. But it is exacerbated here, technology being what it is. It is, thought made flesh and changes fast, not limited by any physical laws.

It is also very un-evenly distributed. We might all be programming in Rust, React and GraphQL in the future. There are people who are already experts in these technologies because they invented it or started working with them much before it was released to the public. Who knows what the future will bring ?

It would be ideal if we can invent the future, but personally I am not there yet.

And mostly, it is impossible! Can you tell all your users to only use Chrome 61 or above ? Can you fix bugs on an Intel CPU and make sure all your servers use it ? We have to make-do with our reality.

What will you do when you are faced with these changes ? What best-practices can be carried over to the future ? Will we learn the essence of programming or will we continuously learn the next new & hip framework which comes up tomorrow ? How will building software be any different tomorrow ?

We need to exercise taste, biases and continuously update your Values about good code. This is why all best-practices boil down to empathy and kindness towards your co-workers and your future selfs. People worry about their website’s UX. Good clean code is the UX for source code.

I wanted to clarify, at-least to myself (in writing), what qualities I value in the code I write and read. But I ended up writing this blog post instead.

This is the lens through which I can hopefully express later, what good code mean to me.

• Incorporeal: Software is not physical. So we need to enforce structure. Structure helps us to understand and communicate intent. This structure helps because software is read more than it is written. This is why software is like writing a novel. You need to divide it into chapters. Put a full stop and comma. Hopefully provide a table of contents and an index. We can write everything in one line, but that will be horrible for everybody. This is why it is a best-practice to use meaningful names, use inheritance or composition where it makes sense. etc.
• Needs to Change: Structure is good. But not, if it cannot be changed. This is why best-practices like duck typing, clean code, TDD and CI/CD are popular. The structure is there for our understanding, so that we can grok the code. So that we can keep a mental model of the code in our minds, change the mental model and then update the code.
• Unique snowflakes: The christians believe that their God built the world in seven days. But we are mortals. Our software is unique but we cannot rebuild reality for each and every software product. So most of us use libraries and frameworks. But we have to pick and choose our dependencies. And sometimes, as a best-practice insulate our code from our changing or quirky dependencies.
• People and code of ethics: Whatever structure you choose needs to be understood by freshers, maybe after some training. It should be easily testable. Things should be as complicated as needed and no more. Simple things should be easy and complicated things should be possible. This also means that you are paying a tax for each new library, dependency or abstraction. Choose wisely.

All best-practices boil down to empathy and kindness towards your co-workers and your future selfs.

Before deciding on the framework to use and the best-practices to follow, we need to be clear about what Values we hold dear, in the software we write and use.

Values are funny. Everyone and every project has them, consciously or sub-consciously. And it is interesting in contrast.

We need to be clear about what Values we hold dear, in the software we write and use.

Ruby on Rails is an interesting social experiment. It is written in one of the slowest languages. It is a framework and not a library. And the framework is pretty opinionated and convention driven. So why do people use it then ?

You know where you stand with Rails. It is not because you need the fastest website. It’s Values are pretty clear.

Another example is React. There are faster and lighter alternatives like Inferno and Preact. There is even Elm, which has a better developer experience. Sometimes it feels as if, Elm is what React wants to grow up and become. So why do people use it then ?

What are the Values held dear by React ?

What values do you hold dear in the software you write and maintain ? Please blog, reply or tweet me at https://twitter.com/_dkannan

I was thinking about my Dad while writing this post. He always encouraged me to read books. After reading a book, he would make me summarise it, in my own language. We would edit it thrice. Each time, the objective was to capture the essence and to make it better. A presis, to be exact. But this is a late night, stream of consciousness post. I have not edited or summarised it too much.

Writing something down clarifies. But it is very hard. I almost gave up multiple times while writing this down.

Personally this post seemed a bit pretentious. I was in two minds about it. But this is my personal blog after all.

What values do you hold dear in the software you write and maintain ? Please reply or blog or tweet me at https://twitter.com/_dkannan

FIN.