Data Distribution via API — Can a Single Developer Do It?

Data distribution and your data availability strategy is central to any data department. It does not help that an organisation’s data is centralised and neatly captured, but users have no access to the data. There are many options, and depending on your organisation, you might need a combination of options to fulfil all user’s needs.

Some of the options include:

• Curated views
• Static file exports on a schedule
• Front-end data access
• Access via API

Providing data to your users and other teams in your organisation with an API is becoming quite a popular option, and it comes with advantages.

For the context of this post, we will be referring to an API when I mean to say REST API. A REST API is a web API that uses HTTP requests to interact with the server.

In this post, we will look at data distribution via API, its benefits and how a single person can create the API to facilitate this data provisioning with adequate documentation.

API as a valid data-providing strategy

I have worked on many Data teams where traditional data-providing mechanisms were used:

• Curated views — While you can give the users exactly what they need, and these views can be highly bespoke, you are giving a user direct access to your data mart. This might not seem like a big issue, but security and administration can become a headache. And this is by far the easiest of the options, it is also the least secure, and many infrastructure engineers will vehemently argue against it. There can also e some performance implications if you are not careful.
• Static file exports on a schedule — Something that you will see at most organisations, while the file is being extracted to fit the purpose, there is ample opportunity for something to go wrong. I personally encountered some of the most prominent issues when the file was not copied over due to network issues, firewall changes, drive availability, space constraints or anything else.
• Front-end data access — While great, you often need special tools such as MDS or alternative or front-end development skills.

I’m sure there are many other reasons I can add to the above, but what about APIs? Are they viable for data provisioning?

• APIs provide data per request, meaning users and teams can access the data when needed.
• You can provide a tailored endpoint, meaning that the data the user get is exactly what they need.
• If you use REST, data is usually in JSON format, making data ingestion and integration easier with modern systems.
• It is (arguably) easier to manage access and security for APIs, and security is according to your specification, meaning you can align with your organisational requirement.
• Easy documentation distribution — using some platforms, you can host documentation for the API with the API, a handy feature in Linx.
• Reusability — The API can be provided to whoever needs the data and they can use your pre-created endpoints.
• Access to live data at the time the API is called. This is a problem I have personally encountered where a client wanted access to live, point-in-time data, but the data was provided via a file on a schedule. You can understand that the data will not be “live” when they need to consume it.

Now that we have looked at why an API can help provide data to your users and clients let’s consider how feasible it is to develop one.

The API Challenge — development

It’s no secret that API development can be challenging, especially if it is not your speciality. To successfully develop an API that allows your users to retrieve data, you need to design, develop and host that API.

Each one of these activities has a collection of nuanced steps and requires a set of skills and tools to be used:

Design — To design the API, you can opt to use something like OpenAPI 3.0 specification. This specification will allow you to document the REST API’s look, define its interactions, and be used as documentation. A tool such as Postman or SwaggerUI is useful for this step.

Develop — Now that you have a design, it’s time to code. You will need to make a choice of what tools, technology and standards you will be using. This step will also involve the integration of different technologies and systems. Typically, an API that will be used to provide data to a user will involve connecting to your database, data lake or data warehouse. You will also need to take into consideration security and authentication.

Hosting — Now that you have workable code, you need to host it. Hosting gets tricky because of the number of options available. Are you going to self-host on-prem, or will you opt for a cloud solution? How will you handle security and monitoring? What is your budget? You must consider these questions when choosing where to host your API.

Traditionally each of these activities has a variety of tools. For example, designing the OpenAPI specification can be done in YAML or JSON. Coding can be done in python, javascript, C# or a low-code tool. Hosting can be done on either an on-premise server or with a cloud solution. These are all tools that, if you are not familiar with them, you will need to learn. It makes sense then why many organisations have multiple teams to care for different sections of the API development lifecycle.

But what if my data department is small, or even just me? Are APIs then out of my reach? This is where low-code tools come into play.

API Development in your reach

Low-code tools offer an alternative to traditional software development. They are built with a drag-and-drop interface, allowing developers to create applications quickly without coding knowledge.

One such option is Linx. With Linx, I could design, develop and deploy an API in a workable state in under 20 minutes. If you want to read more about how I did this, please view my article here.

I have worked in environments and organisations where Linx was used to provide data to clients and other teams. For example, let’s say that your data team specialises in data pipelines to centralise organisation data from various legacy systems and sources. You are getting more and more requests for an easier, more modern way of directly accessing data without using curated views. So you opt for REST APIs. You either need to upskill your team to create these APIs using a programming language and existing framework, or you can save time by choosing a low-code alternative such as Linx.

By choosing Linx, I have experienced the following benefits.

• Since the development interface is drag and drop, you get the common low-code benefit of speed of development.
• I do not have to worry about hosting and monitoring since this is all handled by Linx.
• With the added speed, you can deliver more APIs and focus on support more efficiently.
• The documentation is hosted on the Linx server, meaning that teams do not need to request documentation for the API. It already exists and is hosted for their perusal.
• Easy API debugging with a locally hosted API so that you can see what your API will do and what the outputs are before deploying it.
• Support is easier because of the larger chunks of development blocks. It’s easier for anyone, including business analysts, to understand what an application does. This comes with the added benefit of getting your junior developer up to speed quicker.
• One-click deployments

It might sound too good to be true but also beware of the downsides of using a low-code solution.

• You need to invest some time into learning the tool. This will be a net zero if you have to learn a new framework.
• Most low-code tools come with a price tag. You are inevitably paying for convenience, development speed, and easier deployment and hosting.
• Be aware of the limitations that the tool has. In terms of Linx, there are many integrations and you can create almost any back-end development. You can also create your own SQL scripts and C# functions if you require. However, Linx can only ever be used for back-end development.

Providing data to your clients and teams with an API does not have to be an achievable pipe dream. I genuinely believe it is within everyone’s reach, and the result is a better, more optimised solution for everyone involved.

Conclusion

Using REST APIs to provide data to your clients and teams is, in my opinion, a no-brainer. Just giving that instant access to data without the need for clunky files will make for much happier data consumers.

If you are interested in seeing how you can create a hosted API in less than 30 minutes, feel free to check out Linx here. This will give you a good idea of what it is like to work with Linx, and you will also get a good feel for what it is like to develop an API using Postman.