Taking a different direction

Recently I finally had the chance to meet up with some awesome people regarding my final work/thesis. I have received some great insights on the topic and I have heard their ‘concerns’ surrounding a professional application. And I received three total different visions about this matter. Wow.

First of all, thank you for your time for the interview!(I am not going to publicly disclose their names (yet))

I have to be fair, some concerns had been brought to up which totally slipped my mind… Such as GDPR! My initial idea was to clone AWS infrastructures to local machines, workstations, debug servers etc… But this is not possible due to GDPR, there goes my initial idea on how to do stuff. The plugin I started developing is going back to the backlog (I probably will not release a post about it’s progress because it leads to something unfinished and that is far from ideal.)

Why the original idea will not work

1. GDPR. You are not allowed to just copy and paste data around places, there are tight rules in place and especially in the professional world.
2. File sizes. Sometimes S3 buckets or databases could get very very big and it would be far from ideal to clone 100GB setups regularly. This is not practical for testing (because testing would take too long) and developer machines also do not have infinite space.
3. AWS or cloud platforms in general are changing at such a high speed that ‘mocking’ or faking their functionalities might get hard. There are probably tons of changes and bug fixes daily on the AWS platform itself and maybe a bug is a feature… A mocked environment might differ (highly) in functionality between the real deal and this is far from desirable.
4. Too focused on AWS. It would be more beneficial to investigate a method that works on different cloud platforms. I should perhaps be changing my ‘abstraction’ layer.
5. Amazon already offers a service that makes it possible to run the AWS infrastructure on premise, under the name AWS Outposts.

Some solutions

Maybe we could start running everything on a cloned instance or namespace of the real environment. This would allow us to test the exact behavior of the cloud platforms and we would not be dealing with mocking. But this seems rather… expensive. We would face the same issue if everything adds up to eg 100GB of space. It would take some time to spin up and it would delay our testing. Speaking of testing… Multiple clones would need to exist because one test could interact or modify an environment which could influence the outcome of another test that might be running, this sounds very bad and would cause a lot of headaches when trying to debug such a thing. Everything seems to get exponentially more complex both in configuration and time.

This is not the way to go forward, in my opinion.

Maybe I need to take a step back from the bigger picture. The story of mocking AWS locally makes sense for local development for when you’re working offline — think about long plane flights, working on the train or maybe some strict network rules are in place. And this would also offer an effective way to test things quickly. Yet there are some other issues that surface, maybe some libraries do not allow you to set custom AWS endpoints… A DNS override might work for this but it can interfere with other software too, which can be dangerous and yield some undesirable results.

When looking at it more generally, LocalStack somewhat offers this functionality already so how could we improve? We know that RDS in AWS is just SQL. S3 is a “file system” and there are other compatible solutions. Lambda’s are serverless functions, this can be implemented on another way. How do we deal with ‘agile development’? And another important factor has to be that the setup has to be easy. Easy in a way that it does not take a person more resources (as in wage/money/time) than it would save money and there are always time constraints in place for projects which have to be met.

This is the way

The most ideal situation would be to create an ‘alternative’ to LocalStack which is more modular. LocalStack does not support everything, which is far from ideal and requires you to use other solutions along side it if you need a certain functionality. Running multiple software ‘kits’ is far from being ideal in my opinion. The way to go forward would be using some kind of container orchestrator like Kubernetes and find or create compatible services for AWS or other cloud platforms. Kubernetes would be in charge of holding a cluster up and running and we would ‘simply’ define the services we need. Then we can spin this up either locally, on premise or in the cloud for running tests against. Testing can be done by simply using some data seeding like method.

This would make it easier to use multiple cloud platform services, and maybe even use multiple together. We would be more or less more up-to-date with the actual specifications like S3, Lambda, RDS, … and we do not have to deal with user access/IAM roles for just developing and testing. Another upside would be that it is again, possible to use offline.

In the coming days I will be researching my idea more deeply to make sure it is actually viable. This is a big change in directions and I hope that this does not set me back too far.