Web development… is it all just mail merge?
To be honest, I didn’t come up with this phrase, it was said in passing by a colleague I worked with a few years ago during one of my many contracts. It has always stuck with me because of its base simplicity, cutting right to the nub of what we do 90% of the time as web developers i.e. the act of taking a piece of data and merging it into a template to create a page instance. When framed this way it brings me back down to earth with a good dose of reality, no matter how complex it all gets with teams of specialists and expansive sites: really, it’s all just mail merge, with bells on.
When I started thinking of how I could use this concept to start to introduce our work at Hi-Fi, my starting point was to go to Wikipedia and get a definition of mail merge. I was surprised by how directly relevant it was to our industry.
The mail merging process generally requires the following steps:
1. Creating a Main Document or and the Template.
2. Creating a Data Source.
3. Defining the Merge Fields in the main document.
4. Merging the Data with the main document.
5. Saving/Exporting.
source Wikipedia
I decided I would have a little fun in this blog post and see how far I could take this analogy. I hope that by the end, there will be a point where I can illustrate some of the issues I have come across when building modern websites, especially in large inter-disciplinary teams.
How does it fit…
1. Creating a Main Document or and the Template. View / Partial files
2. Creating a Data Source. Database / CMS
3. Defining the Merge Fields in the main document. PHP, Razor, Handlebars, Angular templates, JSX, VSX
4. Merging the Data with the main document. PHP, ASP (MVC, Webforms), Handlebars, Angular, React, Vue
5. Saving/Exporting. Web Server / Browser
I don’t know about you, but I think it looks like a good fit, simplifying a very complex process down to its essentials. Note that the inclusion of client-side javascript frameworks shows that you can’t just escape from the simple reality of this fundamental process. It even looks like older frameworks still have to jump through the same hoops. Maybe there is no escaping it. Maybe this is how it always has been and always will be, maybe we’re all just mail mergers!
And in the real world…
Although seriously, my aim is not to belittle a very complex industry. So why is what we do so hard? Well here’s a few things we have to deal with day to day…
— merges are run many times
— output can change dependent on user context
— data supplied can be very complex
— data can have different states
— frequent change of template / data / scheme
— dependencies (interlocking mail merges)
— long term support
It looks like we have a very simple process that is made more complicated by how it is used. On top of this, there is additional stuff that is needed to turn the output into valid user interfaces
CSS
Javascript
SEO
Ultimately, however, it all comes down to one file: the view. Everything else is support. The view’s role is to display the information and adapt to the data that is supplied to render a specific instance.
On balance…
With this complexity, it’s no wonder that we need teams of specialists to build sites and that, I feel, is where issues lie. Throughout my career I have always been interested in the relationship between these different specialisms and in trying to work towards balance in the development process. I strongly believe that this is the key to making better sites. If one team has a stronger control of the process, then it affects the output, morale and effectiveness of the other teams.
Anyway, back to it. Ignoring all the other complexity that sits above our fundamental process, below, we can see that there is a definite separation between which specialism is responsible for which part of the process.
1. Creating a Main Document or and the Template. Frontend
2. Creating a Data Source. Back-office
3. Defining the Merge Fields in the main document. Frontend / Back-office
4. Merging the Data with the main document. View Engine (Automated)
5. Saving/Exporting. Web Server / Browser (Automated)
If we talk through each of the steps above. Steps 4 and 5 are automated processes performed by systems (web servers etc). There’s a clean split between the responsibilities for steps 1 and 2 to the two specialisms (back-office and frontend) andI don’t think that these steps cause many issues.
But step 3 is different as responsibility is shared. This is where markup integration takes place, it’s a critical stage in building websites with large teams of multi-specialists. The interface between frontend and back-office is hard to do well.
So maybe there is a point to this blog post after all…
The shared responsibility of Step 3 may explain that there is a conflict between these two specialisms. I think that it has created an unacknowledged tug of war in the web development world for the last 20 years, ever since back-office and frontend separated as different career paths and both wanted to control the View or more specifically the view data (ViewModel) and where this data is placed in the view.
The pains of markup integration
In traditional server-side web development, the frontend developer creates the HTML against a static design and then passes it to the back-office developer to integrate, they then add dynamic markers as placeholders and logic to control the state of the view. This process is inefficient, it’s largely a manual task and further changes must go through the whole process again. MVC was meant to help this by defining a ViewModel for the data that the View requires, simplifying the view, but the back-office defines this ViewModel, not the frontend, so we are back to integration, even if it is simpler.
I think one of the reasons that client-side frameworks (Angular, React, Vue etc) have been created has been, partially, as a fight back against the lack of control over this step. Unfortunately, as a result, much of the functionality of the back-office; routing, services, business logic etc, has been replicated / moved to the client. Whilst this can be great for larger applications (and makes micro services possible), this level of complexity is simply not required everywhere. It also leads to greater load in the client.
These client side systems also take responsibility for mapping a supplied model (via an API) into their own ViewModels: the front-end must have an understanding of the back office model for it to work. It also kind of makes a mockery of step 3, it just pushes this step onto the frontend. What happens when frontend specialisms split into javascript devs and html css specialists, we’ll get the same problem all over again!
A new way?
Looking at the clash of step 3, I think there may be another way: give the frontend complete responsibility over step 3, even down to defining what the ViewModel is, it can then be matched to the wireframes or designs. Then the back-office’s only has to map their models to these ViewModel definitions. As a bonus, the frontend has no need to understand the back office’s data structure.
I think this solution is clean and simple, it streamlines the process, removing duplication of systems and the mess of manual integration, allowing each area to work independantly but linked. It also solves the problem shown in step 3. A good name may be ViewModel centred web development.
Before I wrote this blog post I knew the way that we work at HiFi was different and interesting but I didn’t know why. I think I now understand what it is and why I think that it is important.
This is just the start, acknowledging that there is a problem is a great first step, but how can we develop solutions to overcome this issue and what will they look like. We will talk more about our work with the ViewModel in our next blog post.
