Build Software from Scratch: it is a matter of data management

Hftamayo
5 min readOct 3, 2022

--

I. A couple of facts

These days a developer has a lot of concerns: from software architecture, test driven development, choosing the best frontend framework, passing by taking in count metrics for the fastest, secure and reliable backend technology to questions so sub-realistic like: how much will the cloud provider’s monthly invoice be?, does the CI/CD tool have 100% integration with the chosen platform?, who messes it up with the latest commit? What about technical support? How many members do we need to run this project successfully? What about the budget?

Of course the list of the above items is very intimidating, especially for entry level positions, as a matter of fact, I do not consider myself neither a senior developer nor someone with a lot of experience in the information technology field, in that sense, the rest of us think: when I become a senior developer I will get -magically?- all the needed concepts to be a successful professional; well, I can’t say whether that is true or not but the fact is the labor market is so competitive that most of the job’s requirements ask for very heterogeneous technical skills.

But, putting aside all that, what I really miss in IT projects, and specifically if we are talking about software development, is the relevance of the data: at the end of the day the biggest concerns of clients are: is my data safe? And will my data be accessible in high demand periods?; I have learned this lesson: whether a project has data management as the most important asset, that project has a lot of numbers to be successful, as a team when we reach that level of client’s satisfaction all the effort was worth it and, of course, the investment as well .

II. What technology should I choose?

The data layer should be one of the most critical layers to keep secure and available at the same time; one occasion, I remember when I asked my manager: what is the purpose of using Redis in our system? “caching” was the answer; oh yeah?, is that all? I thought just by myself; then I asked -just by myself of course- why do some data requests take so much time to run? I thought — again- what is the point of using a software just for caching tasks and the system takes a lot of hardware resources and the time responses are not what the user expects?.

That was my first approach to Redis. Few days later I started to search for more information about it, maybe my curiosity was moved to find answers and after that think of possible solutions in our system. I was very surprised: Redis is more than data caching, in fact, it is an “in-memory data structure store used as a database, cache, message broker, and streaming engine” (https://redis.io/docs/about/), that means using Redis I can have access strings, hashes, sets, streams and other data structures.

Moreover, reading the official documentation and many other resources one word was always present: performance, but how does this product of software achieve it? Well, I found these lines: “Redis works with an in-memory dataset. Depending on your use case, Redis can persist your data either by periodically dumping the dataset to disk or by appending each command to a disk-based log. You can also disable persistence if you just need a feature-rich, networked, in-memory cache” (https://redis.io/docs/about/).

All right, slow down: if Redis can manage our data and the performance is its slogan, then why don’t we use it in the data layer for data management purposes? Well I have to say the answer seems to be pretty easy to answer, specially if we are involved in greenfield IT projects (https://en.wikipedia.org/wiki/Greenfield_project); no matter if our project is in the category of greenfield, brown field (https://en.wikipedia.org/wiki/Brownfield_(software_development)) or any other kind, it is mandatory to always look for solutions that help us to improve our data layer even if that means to restructure our software architecture and services.

III. Redis flavors

Basically Redis has two flavors: the first one is the open-source version, according to the official site, this version is written in ANSI C and includes the next features: transactions, Lua scripting, keys with a limited time-to-live and automatic failover for mention just a few, for more in-depth details please refer to https://redis.io/docs/about/ .

The other version is called Redis Enterprise with two installation candidates: Redis Enterprise Software, which offers 30 days of free trial, you may find more information here: https://redis.com/redis-enterprise-software/overview/; the other one is called Redis Enterprise Cloud, this one looks like a strong candidate with a lot of performance and scale opportunities, please refer to: https://redis.com/redis-enterprise-cloud/overview/ .

I am sure each version fits the needs of segment of clients, after take the time to browse each site of the different version, I have to say Redis Enterprise seems to be a strong candidate to take in count for the next years, a strong tendency of software development focus in cloud based services is a great opportunity for developers for considering the use of products for non traditional cloud providers such as Amazon and Google.

III. Best Competitors in this race

Before take a decision of buy or use any technology in a project, it is useful to check research studies, blogs, podcasts or any other media that help us to learn about the performance of similar products; after read several of these resources I would like to suggest the next ones: the first is called Gartner’s magic quadrant for Cloud Database Management Systems, this is a very detailed study of the best software for Cloud Database Management, even though it was published on December 2021 it has a very relevant aspects.

The second one is from official documentation of Redis, it is a comparison between the product and other databases, what I liked from this one is the fact to take in the comparison some of most popular Relational Database Management Systems, RDBMS, surely, most of the developers and database administrators will identify at least one of them and after read the research they will have a clear view about the differences between the relational and non relational database model.

Right, in the next article we will figure out how to put together all the pieces of this puzzle and let’s see how things go, see you then.

--

--

Hftamayo

Freelance WebDev, Infosec Newbie, Bug Bounty and Pentester wannabe