It is the dream of every company to hire the top talents in the market. The kind of opportunities that are presented to the talents in the IT industry in today’s world is huge. These talents consider a lot of factors before accepting the offers.

Though money is a bigger factor, it isn’t the only one anymore. Their career path, growth opportunity and the kind of doors this job would open up for them are other big factors.

It becomes all the more important for the organisations to attract the right talent that will stick around and grow with you rather than being a rolling door for people to step in and out of the company regularly.

The top talents in IT generally fall under one of these two categories.

One that is hell-bent on learning every new technology on the street and get their hands dirty by applying new design patterns onto their current solutions. Let’s call these people “Engineers”

And the other is so concentrated on learning the nuances of the domain they work on and they work to master their communication skills. Let’s call these people “Designers”

An engineer’s career path is very different from that of a designer. I consider an engineer to be a misfit in a bank. It's the designers, who find the banks a very attractive opportunity to work.

When the engineer wants to learn and do a quick PoC around a new technology which he/she feel the right fit for solving the problem at hand, they can’t download, install and experiment straight-away in a secured bank network. Firstly, it has to be checked whether the software is approved by the bank, then the engineer should get approval to install the new software in his local machine. The list goes on. By the time the engineer run pole to post for approval and get it, he/she might lose the interest to actually explore it.

Most of the time, the developers in the bank go the laid-out route. The big architects provide a template/pattern that every application in the bank should follow. It means two things. One, It leaves very little opportunity for a developer to go wrong. Two, It also leaves very little to the developer to explore and experiment. The Designers will find this to be perfect since they don’t have to break their head around technologies too much and can go about solving the actual problems in the bank that would provide them opportunities to learn the nitty-gritty of the business.

This is not at all a bad thing. Every enterprise has its culture. A bank should be secure. It shouldn’t allow random software on to its network. It should follow tight architectural design patterns mandated across its enterprise applications. In fact, most developers in the bank would aim to grow into this enterprise architect role over time since this requires much more than mastery over technologies.

Also, the other very important factor to be considered, How do they go about their daily schedules? In a typical bank, a developer will have to talk to various teams like Business solutions, QA and other ancillary teams. This kind of takes away 4 hours of the day on an average. For someone who is so interested in learning new technologies all along the way, this may become monotonous. The designers though will find it interesting enough because each of these conversations is a stepping stone for them to grow one level up in the hierarchy.

These developers/designers are much more inclined to get into solving the functional problems at hand than to concentrate too much on the technical side of things. Providing them low-code platforms to work with makes their job easier and lets them invest their time and effort onto things that really matter.

These developers don’t really have to think about handling threads or setting time-outs. Let the platform handle it for them. Let them design and build solutions on top of these platforms to help their business grow.

Many banks, by now have realised in their legacy-modernisation journey, how difficult it is to replace tens of thousands of lines of code. Sometimes, there are business nuances buried in the fine prints that are overlooked initially during modernisation. Low code platforms simplify these problems through their graphical representation of code. They employ a simple drag and drop technique to build solutions wherein the actual code in the backend is being written by the platforms and they are not the developer’s problem anymore. They align with the principles of the bank as they leave very little opportunity for a developer to go wrong.

In a way, these low-code platforms are an extension of what these primitive technologies do. They allow the user to write in coding languages (with syntaxes and libraries) and provide a base software on top of which the code can run on the machine.

These platforms they go one step further in the game and allows the developer to break the shackles of technology through user-friendly plugins and business-friendly GUI.

If you were wondering who would attract these talented engineers, all these low-code platforms run on the back of a very good engineering team.

Disclaimer: The views and opinions expressed in this article are personal

Why should banks adopt low-code platforms? was originally published in techBrews on Medium, where people are continuing the conversation by highlighting and responding to this story.

One of the major design drawbacks of the traditional ESBs is their close tight coupling with the provider systems and letting multiple consumer systems reuse the same adapters (services) just because they initially seemed to be consuming similar data (payload).

If we look closely, the tight coupling to the provider systems is something that was forced upon ESB by the provider systems. Most backend core systems used to communicate with SOAP over XML which meant any change in the services at their end has to have a similar change at the ESB system to ensure the communication don’t break. The paradigm shift towards granular APIs at the backend provider systems meant they no more force tight coupling onto any systems they communicate with.

Offering one API for multiple consumers is a tactical design approach that integration architects and the Enterprise application owners make together. There’s absolutely no problem with that. But knowing when to split these consumers into consuming different APIs and how to sneak in a few placeholders to help the same is up to the integration architects.

I worked for an Indian insurance giant helping their digital transformation journey. As part of it, they designed a new web-portal and mobile app exclusively for their agents. At the outset, the app and the portal seemed to be a mirror image of one another. This integration layer was the backend for their front end channels (agent web-portal and agent mobile app) inturn the integration layer orchestrated multiple core vertical systems like PAS (Policy Administration System), CRM (Customer Relationship Management), LMS (Lead Management System) etc. For phase 1, we went in with the tactical approach of both the mobile app and web-portal consuming the same integration layer APIs (same endpoints).

After phase 1, the business wanted certain mobile app specific enhancements as part of phase 2. This is where we took a decision of splitting the IL APIs such that each consumer (mobile app and web-portal) consumes a different API at the integration layer. And though at first, the application owners at the insurance company were circumspect of having two APIs doing a fairly similar job, As phase 2 progressed the Agile requirements brought in more and more complex enhancements at the mobile app within strict timelines. JFM (JanFebMar) is an extremely crucial time for the Indian insurance industry and the business wanted the rollout to happen immediately. Too many enhancements meant complete regression testing which also meant extra days. Since the web-portals are consuming the exact same APIs they did before and since those APIs (codebase) are not changed (meant only the mobile app needs to be tested) resulting in cutting back 50% of regression testing efforts and timelines.

Halfway through phase 2 implementation, the business team wanted a completely different set of features in web-portal for phase 3. /renewaldetails which was previously supporting both portal and app couldn’t have expedited these implementations. Whereas /agentwebportal/renewaldetails and /agentmobileapp/renewaldetails have effortlessly supported the timelines thereby making Business and IT teams happy.

Let’s take a step back to think what’s wrong with having the same API supporting both the channels.

• While one Sprint (agile Dev cycle typically 3 weeks) is running on an API for a channel, other channels have to wait to implement their changes if their cycle starts mid-sprint of the ongoing one.
• Not all channels expect the same payload. Forget about transformations, Sometimes there are derivation logics. If the same API has to be consumed, we force the consuming systems to derive/transform what they want by themselves which is horrible because that’s the very reason for the existence of an integration layer.

To put it simply, the biggest problems of traditional ESBs are more because they were rigid to changes and that’s mainly because they were designed to reuse their components across multiple consumers. In today’s world where enterprises are racing to capture markets with new ideas, the IT should enable them rather than delaying the go-to-market dates with complicated rigid systems.

The idea is only as good as the time of its execution.

Stop forcing reusable components on the Integration Layer. was originally published in techBrews on Medium, where people are continuing the conversation by highlighting and responding to this story.

Isolating applications from infrastructure in the age of containers

Work from home is tough, especially when your mom is interested in what you are working on. A few days back I had an interesting conversation with my mom, explaining to her what a docker container is. My mom is from a non-technical background. Cooking is her forte and hence I had to keep this story inside the kitchen. Hope this serves your appetite too.

Docker Platform — The Dosa Pan

Like how we need a dosa pan to make dosa, we have to install the docker platform to the underlying machine (physical or virtual) to be able to run docker containers

Docker Image — The Dosa Batter

Docker image is the underlying template on which the container is built. The difference between the docker image and the container is that of batter and dosa. The container is cooked out of the image and we can cook as many containers as we like from the same underlying image.

Docker Registry — The Bowl

Docker registry is where the docker images are stored.

Dockerfile — The Cookbook

Dockerfile is the cookbook that says how to make the batter. The dockerfile explains how a docker image has to be built. Usually, every dockerfile will have two important sections.

1. Base technology — This is the base image on top of which our application has to run. This is defined by the syntax “FROM” in dockerfile
2. Application code — This section consists of application-related information like an artifact, ports exposed etc.

Docker Config — The Dosa toppings

1. Config file — the document that contains the environment information.
2. Environment information — these are the details like the port numbers, database connections, interface application details etc.

Even though the underlying docker image is the same, based on the config that we apply on it, the characteristics of the docker container can be defined accordingly. In simple words, this is customising the dosa as we like.

Docker container — The Dosa

Docker container is the final product. Eating the dosa is similar to using the application. We can run as many docker containers as we like depending on the demand using the same docker image and same config file. This is called scaling the application which will be explained in the next post.

How does a developer use docker

For a developer, the major part of the work on docker is with writing the dockerfile. Developer programs the application and build artifacts and construct the dockerfile to build the docker image. This docker image then becomes his key deliverable along with the config file.

In other words, a developer spends most of his time in preparing the ingredients for making the batter. Once the ingredients are ready, he writes/updates the cookbook. Then the preparation of batter is now automated using CI/CD pipelines.

How docker isolate the application from infrastructure

The docker image package the infrastructure (Base image) with the application (code) and hence eliminating the dependencies that tie down the application deployment to the core physical machine.

While it also enables the seamless movement of the code to various environments (Development to Test to Production).

BC (Before Containerisation) era

Those were the times an infrastructure engineer would work for days to set up a new server and deploy the base software and dependent packages that are required to deploy an application artifact.

This roughly translates the process of making the batter from raw rice. This goes through a complex process of soaking and grinding and fermentation. This usually takes a day before the batter can be cooked.

AD (After Docker) era

With docker, the activity is done only once at the time of building the image. Moving the code from one environment to another is now as simple as pulling the docker image from the registry and running it. And this eliminated the dependencies that tied the application down to the underlying machine.

This roughly translates to getting the batter from a shop and making dosa in a matter of few minutes as opposed to days of preparing the batter in BC era. Think of the developer as the shop that prepares the batter.

Why the seamless movement of code is necessary

Once the developer programs the application it has to be tested before it is exposed to the real users. Hence the application has to be deployed at multiple environments to support the back and forth movement.

In the BC era, this happened to be the entire setup being done at every household before they test the batter and provide feedback. In AD, it’s as simple as moving a pack of batter from shop to household and thereby eliminating a lot of efforts

Takeaway

While it may be difficult for the legacy applications to move to docker, I strongly recommend the upcoming digital transformation projects to be run on docker and reduce the efforts that are spent on doing the same infrastructure setup over and over again, thereby eliminating any possible mistakes by isolating the applications from the infrastructure.

Bon appetit.

Docker — The Dosa Analogy was originally published in techBrews on Medium, where people are continuing the conversation by highlighting and responding to this story.

Most applications are not capable of changing with time because of one reason — The developers are not aware of the reasons that went into making certain design decisions of these applications and hence nobody dares to change them.

I started my career as a developer working on a modular monolith application for a Global Bank. The application was in place years before I joined the team and it is being enhanced years after I left the team. During my time, the one big difficulty I faced is to find why a web-service was designed the way it is. Even though I know it can be changed for better, nobody wanted me to touch it because of the infamous saying in IT “If something works don’t touch it.” The reason is so simple, Nobody knew why something was the way it is. They assumed there was a reason for the way it’s designed and a change might bring undesirable results. They were right. Not always the things you see as a developer is in sync with the system design. Those are decisions that are taken at that point in time for some very specific reasons. The entire thought process goes into one document which is the Holy Bible for all the developers, DevOps guys. It is the Technical Design Document.

But the problem at that point in time was the design document did not talk about design decisions and the reason behind it. It talked about how the code is organised, what each of the blocks functionally does etc, all of which I could see when I browse through the code.

Upon my transition from developing an application to designing an application, one of the first rules I made for myself is to log the reasons behind the design decisions in the TDD. Let me share a few pointers on how I write TDD and more importantly how I decide on what goes into TDD with an example.

The Integration layer that’s designed is supposed to subscribe to the messages on a topic and validate, enrich the data and send it to two core systems. (CRM and CBS) Now, this could have been done as a single API. But I have to split it up into two APIs. (A good developer will curse me on the first look of this) First API gets the messages from the topic, validates and enrich the data and send it to CRM and drop the messages into a queue. The second API will consume the messages from the queue and sends it to CBS. (It does nothing but this). Any developer can get all the information that I have described so far by just browsing through the code. All they need is the missing piece of the puzzle. Why two APIs? Now, this is what should go into the TDD.

The downstream CBS cannot handle huge loads. The Integration API receive a high volume of messages and if all them are processed in real-time the CBS would fail some of these messages. Their communication with CBS is over Http and hence the onus is on the Integration layer API to make sure it balances the load. And hence the messages were posted to a jms queue instead of calling the Http endpoint directly.

Now the interesting part is balancing the load. The second API had two important factors that justify the need for a separate API. Maximum sessions and the acknowledgement mode. Max sessions is configured at the jms queue listener. (It means at a given point in time, the second API will handle only ‘N’ number of threads/jobs where N is configurable). The max sessions ‘N’ was calculated at that point in time on the basis how many parallel requests CBS could process. The acknowledgement mode is set to the client (which meant if the listener process doesn’t confirm after processing the message i.e. if the processing fails, the message would still be in the queue to process it again until it’s acknowledged). Now just because CBS couldn’t take up the requests for processing, we can’t repeat the process of validating and enriching the data and we most definitely can’t send the same processed message to CRM twice either. Thus, sending the processed message to CBS has to be separated into an API.

Now the TDD also speak about what needs to be taken care during Ops. The Integration layer APIs are deployed on an easy to scale platform (read Docker and Kubernetes). On high loads, it is common to scale up the number of instances (read pods). The component on which the queue listener is running also has other APIs which needs scaling up. (Why we grouped these APIs on a single container is again something to explain in TDD) so while scaling it needs to be taken care that the sum of the max sessions of all the active pods sums up to ’N’ at any given time to avoid stress in CBS, which means a production support guy who scales up the pods, have to reconfigure max sessions.

As a developer/support guy, I will hate to read a TDD to explain to me how to configure properties in a configuration file (read config map and yaml files). These are things that I could either figure out myself or are they a Google search away. But what I really need to know is what value should it be configured to.

This TDD is now part of the code base and whenever a developer does a git-pull he gets to read the TDD to understand why something is designed the way it is. Now years after I moved out of the project, the bank migrated to a modern CBS that could support high volumes in real-time. With TDD being part of the code, the current development team concluded two separate APIs approach is not needed anymore. Their knowledge of the legacy design (from TDD) and the reason behind it empowered them to make the right decision.

TDD (Technical Design Document) as part of the code was originally published in techBrews on Medium, where people are continuing the conversation by highlighting and responding to this story.

The one where enterprises are Chandler.

“When I first meet somebody it’s mostly panic, anxiety and a great deal of sweating” said Chandler Bing.

Most men start off as Chandlers — awkward around women, desperate to interact with others, and interesting, so does the enterprise. The first tech stack of most enterprises (aka greenfield implementations) are bound by the limitations of time as there is a hurry to go-live as soon as possible. Such enterprises fear the process of getting a best of breed systems landscape, and in the end, settle for a monolith core that handles everything (read: Janice). While the purpose is met initially, as the enterprise grows, the monolith doesn’t cater to all its requirements. Getting out of this is painful, and if enterprises may simply be chicken enough to make the difficult decision of separating themselves from the monolith. In the TV series, Chandler has to fly to Yemen to break up with Janice.

The one where everyone wants to be Joey.

Every enterprise quickly yearns to have a best-of-breed set of applications, each catering to their specific purpose. They dream of being agile and interoperable, where adding and removing applications are as easy as can be. They want these applications to seamlessly co-exist without getting treading on each other’s toes; that helps the enterprise fulfil its true potential. It allows them to live with their quirks, yet be the favourite of in the crowd.

The one where most end up being Ross.

Sadly, not all wishes come true. While the ambition is high, the execution sometimes is so poor that these applications end up tied to one another in a complex, muddled spaghetti of interactions, such that the enterprise often finds itself in trouble when it needs to replace one application with another. Over time, they get married to their bad decisions and have to go through the painful process of divorce to finally move on. Worse, the enterprise chaos is so terrible that even after moving on, their thoughts are still filled with that of their previous applications as they haven’t truly put them out of the mix.

On paper, every application starts as an independent system. As they grow along, they start to interact with other applications, sharing data and wardrobes alike. However, these applications never intended to and were never designed to, behave this way, yet they somehow end ou getting coupled together, and moving in without even realising it. By the next major release, these applications would send out holiday cards together for all you know.

Imagine three applications, handling data related to loans, cards and customers, respectively. When the customer’s self-service mobile app requests for a 360-view with a mobile-number, many poorly designed enterprise architectures — with good intentions of sending out a quick response — move data that is unnatural to applications that shouldn’t have it. For instance, loan and card information in the customer systems. This makes all three applications grid-locked, and any attempt to replace any of these applications is exhausting in this spaghetti of relationships that are complex to manage.

The one how Chandler became Ross.

In the above example, customer contact information inside card and loan applications are corrupt data.

These data should ideally be kept outside of these systems. The contact information should only be maintained in the customer application. When the online channel request with mobile-number, the customer-number to be retrieved from customer application and the loan and card details to be pulled out and stitched to give a 360-view through an Integration layer.

The one where Chandler can become Joey.

Joey doesn’t share food. But he gets to eat everyone’s data.

The clean way of being a Tribbiani is to implement a strong integration layer and a reference data store (Joey’s social quotient and social network, respectively). The reference data store will have the customer-number, loan-number and card-number mapped to one another. The integration layer fetches the mobile-number in the request, will first have to look into the customer app to get the customer number, and then fetch the appropriate information from the loan and card systems. The integration layer finally stitches the data together and responds to the mobile app.

The one where anybody can be who they want to be.

With this approach, applications can have the free will to decide what they want to do without worrying about the data from external applications unless the business wants them to. It makes the enterprise Monica-like organized, it makes the IT landscape clean enough to attract Rachel-like attention, and companies can go about growing their business with the bare minimum fuss you normally associate with Phoebe Buffay. The integration layer and the reference data store helps enterprise be who they want to be.

— — — — — — —

The author is a nocturnal tech creature and a midnight snacker who works with multiple financial institutions to help them achieve their enterprise goals with integration and data management. For more information, visit this link to learn all about what a Tribbiani-class reference data store can do for your platform.

Reference data: How you doin’? was originally published in techBrews on Medium, where people are continuing the conversation by highlighting and responding to this story.

The most common argument I come across these days is the future of the ESB (Enterprise Service Bus). For many years, the ESB has been the key to building a loosely coupled integration platform capable of converting any protocol and format. However, many companies are starting to invest in a service-mesh and event-streaming platform, in the look-out for an alternative to their integration needs.

When I was first pulled into this conversation, I was very confused as the purpose of the ESB is very different from that of a service-mesh or an event-streaming platform. Later, I realized that companies are looking to solve the problem of integration within their applications, and focus on sharing the data versus than transforming data. Before we look into the existential crisis, let’s take a minute to look back at ESB came into existence in the first place.

Until the 90s, most companies had their applications interacting with each other, almost like people in a parliamentary session. A lot of point to point interaction with very discipline. At some point, it was decided to bring in a moderator who can efficiently listen to everybody’s opinion and stitch them together in a way that each of them will understand. The communication was so easy as the transformation was efficient, irrespective of the message format — be it copybook or EDI formats to SOAP/XML or JSON. The Canonical Data Models standardized the flow of data that enabled the organisations to efficiently partner with third-party applications at will.

The ESB has solved all these problems; but how did it manage to become a problem by itself? Well, the answer is the implementation pattern, leading to strange decisions. We live in a world where Pant and Pandya are sent ahead of Dhoni in a World Cup semi-final. Instead of letting the applications do the work they are supposed to do, the architects and designers started to exploit the ESB and write their business logic into it. Not every ESB implementation had its CDM data model and in a few cases, their orchestrations were ineffective. This often resulted in changes to the eco-system becoming more tedious. From being a catalyst, the ESB ended up as a bottle-neck in those few cases.

The ever-changing or rather ever-evolving trends in technology, from SOAP services with CDM, ESB has moved on to REST APIs (thanks to kubernetes) but the importance of the data model is lost in this madness. In the name of micro-services, point-to-point integration with simple transformations being pushed on as ESB forgetting the very reason for implementing an ESB. In other words, instead of a moderator who enables effective communication, we got ourselves into the “Newshour Debate”. Remember, most of these ESB products come with licence costs. This combined with the shift of trend toward service-mesh, event streaming, fast data has become lethal for ESB.

Switching back to our primary argument, is this the end of ESB? The answer is not simple. It purely depends on what the organisation wants to do (or how they see themselves being positioned in future). For tech-first companies, it is an effective strategy to invest in technology and to stay on top of the trends. But tech-enabled companies that look up to their technology wing (who have to play with a very limited budget) to provide a stable and cost-effective solution to their needs would have to consider which technology would be the right fit for their problems rather than simply staying with the trend just because everybody else is doing so.

The mind-boggling question is: does keeping up with trends influence the cost-effectiveness of the solution?. To answer this, we must first look at how an enterprise would function without ESB.

Firstly, When all the applications in the enterprise follow an API-first architecture, the need for a mediation layer to support multiple data formats becomes less significant. Secondly, The complex transformation and the stitching of data are expected to be taken care at the application end rather than an ESB.

So the problem that remains is that of sharing the data effectively, which service-mesh and event-streaming platforms take care of.

Tech companies, as they move out of ESB, they have a clear roadmap of what needs to be done. Rather than changing their architecture overnight, they transform their applications one by one before letting the ESB go. Most tech companies that you see making their waves with the shift in the trend has already gone through this process (or mid-way through it).

If we consider the case of tech-enabled companies, especially the ones with a small purse for investments in technology, they have to rely on COTS (Commercial off-the-shelf) products to make their solutions quick and cost-effective. The problem with COTS is that they are not flexible. Every change costs. That’s where the ESB comes plays a role. From complex transformation to orchestrating and stitching the data together, the ESB becomes the central nervous system. When these companies want to move out of the traditional ESB approach, the one question they should ask themselves “are they ready for it yet”?

In most cases, the answer is NO. Reasons? Cost of developing in-house solutions against that of COTS, and the ability/skill to do so. And hence the better solution for these companies is to stick with ESB and give themselves the time for them to evolve beyond traditional ESB rather than to jump into the trend.

The ESB is an architecture; it is up to the architects to design implementation patterns that allow the ESB to do what it is intended to do rather than making it a “Newshour Debate”. With the proper architecture and disciplined implementation, I say the traditional ESB is here to stay. At least for the next decade if not more.

Why is the ESB still relevant? was originally published in techBrews on Medium, where people are continuing the conversation by highlighting and responding to this story.

Nostalgia: Before JSON, there was aeSchema

In 2004, I worked on a product called Rendezvous (also known as RV) at a bank in Dubai. RV was a truly unique way of distributed messaging in those days. It was a logical bus, and all applications needed to do was to send or receive data on the bus over UDP broadcast. In fact, the two tools that TIBCO continues to ship to date, with some of its software, are tibrvlisten and tibrvsend.

Rendezvous at its time was, simply put, science fiction. It blended the easiest set up in the world with low latency and a beautiful piece of architectural prowess that married TCP and UDP. It was super simple to troubleshoot (listening on a wildcard subject all across the subnet was all you had to do), and you could see every message in an out; a debugger’s dream and a security engineer’s nightmare.

The output of a tibrvlisten typically is a message that’s being sent “over the bus”, was also known as an ae message (ae = ActiveEnterprise) and it looks very similar to this.

2019–15–11 10:54:59 (2019–15–11 10:54:59.520000000Z): subject=hello, message={DATA=”hey world” value=17.10}

Looks familiar? Here was something that pre-dates JSON: aeSchema. It was beautiful: key-value pairs with no fuss. It had basic data types although we largely dealt with strings, it had sequences and repeating structures, and it was easily parsable. Nineteen years later, we’re back to where we began. JSON remains one of the easiest way to represent data.

But how about data that is not in-flight; is JSON the right choice of tech to store all data at rest? When we look at managing data, we typically consider one or more of three use-cases.

Ability to search through a large data set

These are classic big-data use-cases that involve data exploration, full-text searches in both, structured data and unstructured data such as PDF/word attachments. The industry outcomes that organizations expect out of being able to search large data sets involve scenarios like negative-list checks, dedupes and customer profiling for risk (fraud) and opportunities (cross-sell/up-sell). Hadoop and Elastic are relatively popular technologies that address this problem really well.

Ability to report on the data set

As organizations tend to become more data-driven, the importance of spinning up visualizations and dashboards that aggregate data cannot be understated, particularly for consumption by thought leaders, executives and decisions makers. The more popular BI tools in the market — Tableau, Qlik, Spotfire, PowerBI — all rely on some form of relationships existing between data sets. Most of these tools were built to work with RDBMSes and now support document data stores, either natively or via well-established connectors.

Ability to serve the data as APIs

This is one aspect of the data that is closer to heart and often ignored. Let’s consider a simple example to first put this into context and understand why is it important.

A large bank deals with hundreds and thousands of transactions — POS swipes, internet banking transfers, UPI payments — every day. One of their data science projects is to build models that classify and cluster similar transactions for a given human together, to determine what’s common amongst their payments, and predict if the person is likely to spend on food, travel or tech gadgets. This requires a very strong analytical engine that can sift through thousands of records of data and make a prediction, not necessarily in realtime, but at regular intervals (such as every 24 hours).

The outcome of this prediction is a probability score that is served on a visualization dashboard. Traditionally, we would extract that data, create whitelists and campaigns around it and target customers with contextual offers that are relevant to them, to nudge them towards a store or increase their spend on the bank’s co-branded card.

Today, a lot of marketing tends to be done when the customer makes contact with the bank. A phone call into the IVR drops into the call center team, a user logs into the internet banking after a few weeks, or a customer walks into a shop and makes a purchase. These are the best moments to recommend a contextual offer for the customer to make a decision; at that point in time, they’re hooked.

Calculating that probability is important, but what’s equally important is to be able to serve that data as quickly as possible via an API and to be able to handle hundreds and thousands of hits per second.

This is where JSON rules.

All hail the Omnipresent Javascript

JS is a real langugage and that’s made it a key factor in the adoption of JSON. Coupled with TypeScript, it is ubiqutious; from Angular and React to NodeJS/Express and MongoDB; JSON/BSON documents are everywhere.

The beautify of it is in its versatility. Here are a few reasons I would pick JSON documents to store data.

1. Key-value pairs are way versatile than rigid columns and tuples.

For the same data set (aka collection), it is perfectly alright to have different records having different attributes. Not all employees will have an exitDate, not all accounts will have a closedDate, and not all payments will have a return element. I won’t end up having to design each and every attribute for each and every scenario up front, I don’t have to fire a shitload of ALTER TABLE statements every time I come up with a new attribute specific to a use-case, and I don’t have to stare at a hundred null values in a row.

2. It makes data seem more natural.

I have never enjoyed joins. For decades, an e-commerce order that has multiple line items would have to be thought of in a one-to-many relationship. It seems completely unnatural, why would I need two tables to manage and represent one dataset? It discourages “filler” attributes, and schemas that have columns such as “Address1”, “Address2”, etc. JSON documents simplify it; one record (aka document) = one e-Commerce order. Arrays to represent multi-value datasets. Nested structures to indicate grouping.

3. The overhead of APIfication is less.

While reporting is slow, I believe the performance overhead on reports is acceptable given that a few humans are requesting aggregated data over large sets on-demand. APIs, on the other hand, are built to scale, so fetching data from a database that stores it in BSON, parsing it in NodeJS / Express in JSON and then passing it back to the UI that renders it using JSON / HTML is the norm. The overheads involved in doing this serialization is almost nothing.

To summarise, JSON and the Document data model remains my most-preferred way of storing data at the moment. It certainly isn’t a one-size fits all, but it certainly fits most use-cases.

Disclaimer: The views in this post are personal, yet many of these principles were applied to design our open-source CRUDder that ultimately paved the way for us at CAPIOT Software to build two products we are extremely proud of, that are only getting started in market adoption. XCRO is the only enterprise-grade tool out there that manages complex escrow deals from constructing highways to producing movies and high throughput trust-retention constructs such as dealing with e-Commerce transactions. Omni Data Platform is an API-first enterprise data management platform deployed on a containerized architecture. Both tools are built on top of a world class database — MongoDB — with JSON documents at its heart. MongoDB, after providing ACID transactions in version 4+, is now starting to leverage Lucene to power its full text search, and there are several BI connectors out there that make it easy to report on the document database, enabling us to solve a wide variety of use-cases with a single design that can be configured for different flavours of data.

Data in its natural habitat: JSON documents was originally published in techBrews on Medium, where people are continuing the conversation by highlighting and responding to this story.

Many of us, in our work lives, are driven by victory.

Sales folks, pre-sales, and consulting teams keep count of their wins all the time. We call these targets. They’re linked to incentives — it is how most of the software industry’s compensation is structured.

When we win, we celebrate. We close a deal, sign the paperwork, have a beer and move on to work with that customer and make them truly successful with the technology and services they procure from us.

When we lose, we do a post-mortem. We try to figure out why we lost, and we try to look at it objectively. The idea is not to square blame on any individual, but to build a team that operates like clockwork.

But I’ve always wondered — in the midst of this obsession with numbers and meeting targets — how often do we take a step back and think about why we won? Why is it that we don’t do a post-mortem of a win?

In fact, why do we win? What makes us tick? What are we obsessed with? Why are we good at what we do?

I’ve looked around — and within — for the answer, and many come to mind. A great team. Integrity. Hard work. Dedication. Creativity. Technical superiority.

But honestly, none of these influences wins as much as one thing that is unique to your company.

Something that each of us didn’t have a choice in acquiring, but we did anyway.

Something that was passed on to us.

Something that we cannot unstick easily.

Something that is perhaps the biggest contributor to work culture.

Heritage is important.

To paint a picture as to how significant what we inherit is, here are three real-world sales-wins that we sliced and diced with the power of hindsight, to examine exactly what made us win.

2017. An insurance firm in Bangalore.

“Cheers”, says our new hire sales guy — let’s call him John Doe — as he sips on locally brewed hand-crafted beer in a microbrewery in central Bangalore that would make its ancestors in Michigan proud.

John Doe isn’t just a sales guy, but seasoned professional with over two decades of experience selling everything from boxes to cloud applications. It’s been a good evening. We’ve just been awarded a contract to digitally transform how our customer — an insurance firm based out of Bangalore — on-boards their customers.

The possibilities are endless, from automating underwriting, to customizable workflows, to making decisions in real-time. All of this wrapped with an intuitive user experience that does not burn a hole in their pockets to operate or run. The solution is a true best-of-breed, a combination of TIBCO’s classic integration platform, with Red Hat’s business rules and workflow engine powering the backends. Slapped on these services is a layer of Angular that makes a lightweight user experience possible. We’re very proud of the architecture and are excited to start the project.

2018. A bank in Mumbai.

It is the middle of the week on a wet evening in the offices of a large financial services institution. Mumbai. John Doe’s just done wrapping up the paperwork. We are going to help a large bank in the country automate thousands of their escrow deals.

The Real-estate Regulation and Development Act, or RERA as it has come to be known, is a set of laws that helps protect home-buyers. It was passed in the House of the People (Lok Sabha) in India on the 15th of March, 2016 and came into effect on the 1st of May the same year. RERA introduce many reforms, one in particular requiring property builders to open a separate account for each project they are developing. The property developer has to deposit 70% of the money raised for the project in this account, and these funds can only be used to develop the project.

With RERA in place, the number of deals/contracts that banks entered into, with the builders, and the structure of these contracts — very similar to an escrow arrangement — meant that their operations teams had to man up.

All of the aforementioned bank’s work is currently on excel and e-mail, and without a good system in place, their ability to grow the business and scale the operations, and run it all smoothly and risk-free, is hampered. They looked at our product and it solves most, if not all, of their problems. It’s a straight-fit and after a tough round of negotiations — they are a bank, after all — we’ve closed the deal.

2019. A financial services institution in Ho Chi Minh City, Vietnam.

A financial services company that offers employee benefits programs to its customers, targeting the rural workforce in the country, is looking at replacing their legacy technology. At the heart of this transformation program — quite literally — is the integration layer that stitches all the front-end applications (channels/customer-experience components) with the back-end systems (loan processing/payments/customer-servicing components). There’s an RFP out for which we’ve placed a bid. Our top solution architect has already visited HCMC, Vietnam, where he’s presented our architecture, approach and credentials.

Commercial negotiations are up next, a week later, in Vietnam again. Yours truly has just flown into the country, and Grabbed (the Uber of South East Asia) his way to a 13ft x 12ft conference room, surrounded by six representatives from the customer, mostly C-level execs and their direct reports. The possibility of a win (or loss), and a projector projecting my slides, hangs over my head.

In less than an hour, we have shaken hands. The negotiations were hard, but fair. There was no discussion on the technology components; it was purely commercials. I am, admittedly, not the best negotiator in the company, and the only reason I am in the city is that we forecasted — wrongly — that we would be discussing the design and architectural components.

We won all these deals. Except that, in hindsight, we had no business winning either of them.

We’re an integration company. We don’t build applications. We had no insurance expertise. Some companies live and die by building workflows to onboard insurance customers. We aren’t one of them.

We’re a data management company. Some companies live and die by building workflows to manage payments. We aren’t one of them.

We’re a company headquartered in Palo Alto with most of our business — both revenues and headcount — in India. We have no presence in Vietnam whatsoever. No one in our team speaks the local language. We can barely hand-motion our way from the airport to the hotel. We have zero understanding of the local customs. Surely, there are software companies in Vietnam that do exactly what we do?

So why did we win these deals? In all three situations, the prospect-turned-customers-turned-champions liked a few things about us.

We were honest. We didn't lie during the pre-sales cycle. We didn’t have to; our work and the technology speaks for itself. But that wasn’t why we won.

We worked our asses off. They could see the effort we put into our proposals, both technical and commercial, and they knew we had thought through every scenario, requirement and the approach in detail. But that wasn’t why we won.

Were we priced the lowest? No, we quite likely weren’t. We were, at least, 15% above most of the lowest bids.

These wins — and many other wins — boiled down to one important thing that we had, which subconsciously found its way into everything we did. Our heritage.

Our heritage is being real-time. Everything we do, from APIs to micro services, from data management to data integration, from escrow operations to building a partner ecosystem, is built on the bedrock of what we inherited from our past lives, which is to make data real-time.

Nearly every design decision we make, pre-sales and post-sales, revolves around making data real-time, whether it is about stitching systems together or distributing data through a kafka broker.

Nearly everything we code is wrapped in an easy-to-consume API, making it easy to integrate. The acronym “API”, to many, is a buzzword. To us, it is bread and butter. Building APIs is in our DNA, most of us are just coming to terms with this reality.

In all three of these deals, the customers went with us because we knew how to manage and integrate their data better than anyone else they had spoken to. It reflected in our architecture, our presentations, our proposal — heck, even in our contracts. They could see — better than we could — that we were the best at this.

And that’s why we won.

The heritage at CAPIOT and appveen comes from decades of experience working with APIs before the acronym got sexy. You can trace this DNA to two very different sources.

The TIBCO DNA

The first is a clear bloodline from the past-employer where many of us, including the core founding team, spent years at: TIBCO. Back in the late 90s when the dot com bubble was about to burst, one Silicon Valley company was focusing on solving a very important problem: data integration. TIBCO (short for The Information Bus Company) helped pioneer the enterprise service bus in the early days of data integration, and it was well ahead of its time. Its founder, Vivek Ranadivé, had a brilliant idea. From that IITian-turned-MITian brain, the notion of an information bus, where applications could just plug in and out of it and exchange information, was born.

I worked at TIBCO for eight wonderful years that ended abruptly simply because I had to give a shot at starting up, and all of my co-founders would agree that this was the only thing that could’ve gotten them out of TIBCO. We were quite likely to retire there.

At the helm of TIBCO, Vivek was no random schmuck, and his story is somewhat documented in Malcolm Gladwell’s David and Goliath. To put it into context, he managed to convince the heavy hitters on Wall Street to digitize their trading platforms using his technology, and he won against all odds. This cleary pissed a few of the biggies, who abbreviated TIBCO as “That Indian Bastard’s Company”. Vivek did the smart thing of surrounding himself with brilliant people and built a company worth being loyal to. He and his direct reports ultimately created what we called the TIBCO DNA.

The TIBCO DNA is a set of beliefs and practices that define how many of us do our day to day jobs in the world of distributed software computing. It has been passed on for over two decades, and remains at the core of many other companies that are being run by people who worked at TIBCO in their past lives. The TIBCO DNA isn’t easy to find — a google search brings up only 94 vague results at the time of writing this post — but it is quite prominent in many of the top tech firms today. From Red Hat to Mulesoft, from MongoDB to Solace and Elastic, you will find this DNA spread all around the people that make these companies tick.

The TIBCO DNA means different things to different people. Its origins are unknown, or lost as an undocumented memory. What inspired it, is something we may not find out. It has elements of entrepreneurship, pragmatism and integrity.

But above all, it focuses on keeping data real-time, and getting intelligence out of it. TIBCO’s obsession with speed and analytics is evident; they have a very well-documented partnership with the Mercedes Formula 1 team, and you only need to look at Lewis Hamilton’s or Valteri Bottas’ helmet for their logo at the chin.

We’ve inherited this DNA in our lives. Our first hire was a brilliant TIBCO architect. Few have influenced our people as much as he has. Over half of CAPIOT has worked on TIBCO products, and has diversified into parallel technologies — Mulesoft, Red Hat, Apache Camel, the works. This half of CAPIOT has an unfair advantage in comparison to our competitors; they have done nothing but integration and APIs day-in day-out. They’re the best at it.

The automation DNA

As great as TIBCO was, it remained proprietary technology for a very long time. In 2006, I met two of the brightest technologists I have ever worked with and simply put, I am super-proud just to have them as friends. They were good coders — maybe even great — but it went further than that.

They built a culture of automation. They built a culture of exploration. And they built a culture of consuming open-source technology simply because:

1. It was good
2. It was easy, and
3. It was out there

When we started up, these two guys were my early hires. It was a no-brainer; they were swiss-knives. Give them a tool and they would excel at it in no-time. But more than anything else, they focused on automation.

If it took them two minutes to file their expenses, they would spend days automating it. But they never had to do expenses again.

They went about automating everything, from slackbots to troll executives to CRUD operations. Ultimately, they helped build one of the most sophisticated data management platforms that is only getting started in terms of market adoption. Nobody does CRUD + APIs as well as we do. We’ve built over half a million in subscription revenue on the backbone of these frameworks, and we wouldn’t have gotten here if our team had gotten behind our obsession to automate the journey of managing data.

In the five years since we started up, we have looked at all our successes and failures, both critically and nostalgically. We are very proud of what we have achieved; the team we built, the customers we’ve acquired, and the work we’ve done for them. But it is only recently that we started recognizing the power of our heritage.

What got us here may not take us ahead. We most likely will have to re-invent ourselves as the world changes. But knowing what is at the core, being able to adapt to variations of that, and staying true to our real strengths while being honest about our weaknesses, is what will help us embrace the future for what it is.

The author is a nerd who dabbles in code, slides and spreadsheets for the daily bread. Almost the entirety of the rest his time is spent in watching, talking about and thinking of football and Formula 1 while attempting to survive as nomadically as life permits him.

I’m going back to the start. was originally published in techBrews on Medium, where people are continuing the conversation by highlighting and responding to this story.