The Art of Security Obscurity

Thomas Edmonson
Hybrid Cloud How-tos
7 min readDec 1, 2022
Photo by Matthew Henry on Unsplash

If you’re like every other enterprise today, one of the headaches that never seems to go away is security. There’s always some new zero-day vulnerability, a competitor had a data breach, someone else had their key data encrypted and held for ransom. Development is halted for patching and remediation, costing the business time and money. It never ends, and these days it feels like it’s only getting worse. There are no end of articles about this topic, and different products and offerings around it, so we’re not going to cover any specific ‘how-to’s here. A quick Google search will do that for you. Instead we’re going to consider together the different things you need to think about when implementing your solutions and how to make it as painless as possible for your users and team members.

Standardize all of the things

We’ve already covered this subject earlier in this series, but it’s worth repeating: standards are everything. You can’t automate anything if you don’t know your end goal. You need to be able to define the golden state that your automation is trying to establish and maintain. You need to have the right people with the right skill sets to define how you want your enterprise to operate going forward. And you need to agree as an enterprise that these standards are to be followed by everyone, no exceptions. Without these agreements, you’ll just end up back where you started, with a fragmented environment and snowflake applications everywhere. Finally, you need to pare down your technology stack. You can’t automate and standardize any and every technology you want to use, so you need to decide what’s strategic and keep the focus.

Map the road ahead

Now that you have your standards, you need to figure out what you want your final goal to be. For example, at IBM, we’ve chosen to build security into the very fabric of our infrastructure, to make it as invisible and seamless to the end user as possible so that they never have to think about it. That’s what makes sense for us, but you need to think about what makes sense for your business and your organization, keeping in mind the amount of time and effort you’re willing to spare to secure the enterprise. Will you go with a home-grown solution? Rely on a vendor to handle it for you? Or a mix? You’ll need to determine which choice is right for you.

On that note, there’s a very important part of security automation that you’ll likely want to tackle before anything else, and that’s monitoring. If you don’t know where your issues are, you not only won’t catch critical exposures in your environment, but you also won’t know what to prioritize automating first. Bottom line, if you do nothing else, automate your environment monitoring. Even if you don’t have the time or budget to automate remediation, you can at least target areas for manual fixes. There are no end of fantastic articles and products around this space (including earlier in this series), from basic infrastructure monitoring to code scanning to network vulnerability scanning and more, so I’ll leave it there.

Before I move on, one last thing to keep in mind in this process is transition planning. You won’t get all of this done at once, nor should you. As you figure out what areas you need to target first and target your automation efforts accordingly, you will undoubtedly face challenges rolling out your new standards and processes to your environment. Especially if you didn’t have set standards before, you’ll be met with resistance from teams who have their own priorities and commitments to the business and don’t understand why you’re trying to change things on them. We strongly recommend that your executives are in lockstep with your strategy to make sure that the need and the message are communicated clearly to help everyone succeed.

The right tools for the right job

So you’ve started rolling out monitoring across your environment, and you’re getting a pretty good idea of what you need to tackle first. Now’s the time to think about the tools you want to use for each job. What tool you choose will depend on a variety of factors unique to you: What will let you move fastest? What will be the most flexible? What will be the most sustainable over time? And of course, what’s cheapest?

At IBM, we’re using a variety of tools depending on the situation, but our primary workhorse is Ansible. The modular format has given us a lot of flexibility in our extremely diverse environment, allowing us to more easily adapt automation for technologies that aren’t commonly supported in other tools on the market. We combine that with a variety of other tools in our ecosystem, building security-conscious CI/CD pipelines with tools like Tekton and GitHub, monitoring with Dynatrace and Instana, image scanning, antivirus, access management…the list of areas you need to cover goes on, and the choices you need to make over your tools won’t always be easy. Try to keep in mind the ecosystem as a whole as you choose, and find tools that will integrate well together or you’ll risk wasting extensive time on custom integrations. Companies like Red Hat that offer comprehensive toolsets may be a good place to start, but no one company will offer everything you need.

Prepare for success

Finally, you’ve defined your standards, you’ve decided on your toolsets, and from your monitoring, you know what areas to target first. It’s time to think about how this all fits together into an end-to-end experience. How do you get to a point where security is completely invisible to your developers and users? Where you almost never have to think about patch windows, downtime, or even data breaches again?

If you just have a small presence on a public cloud, this probably won’t be too bad. The cloud vendor has done a lot of the heavy lifting for you, and your teams can leverage its tools to create a uniform build process for your company, eliminating much of the manual and maintenance work that traditionally goes into security. However, if you’re like IBM or any other enterprise that operates in a hybrid cloud, you’ll be faced with opportunities and challenges. The opportunities are exciting, such as the ability to fully customize the infrastructure and code-deployment process for your company, providing a tailored experience that best fits the needs of your team rather than a one-size-fits-all approach from a vendor or public cloud. Your team also gets to challenge themselves, solve interesting problems, and learn a variety of new skills that contribute to the growth of both themselves and your organization. That’s an important mindset to keep, as the challenges you’ll face are many.

As mentioned earlier, you will likely face resistance from development teams that see your changes as unnecessary, especially before you have a full ecosystem and provisioning platform to support them. There are a huge (and growing) number of areas involved with security that you’ll need to constantly keep in mind and figure out how to manage behind the scenes as you tie each one together. The transition process will be longer than you (or your executives) would like. How do you even go about hiding all of this to create a seamless experience?

Experience is everything

How you build your user experience around security will depend on the results you want to see and the platforms you choose to use. For example, if you’re using virtual machines (VMs) or bare-metal servers, you’ll likely need to create golden images with security built into them, followed by automation to maintain their secure state. You’ll need to keep those images secure with an automated pipeline, implement automated testing, decide on cadences for automation to avoid disruption to the business, and wrap it all behind as simple a provisioning platform as possible. You can turn the complexity of everything you’re doing behind the scenes into simplicity with abstraction and obfuscation.

The platform isn’t the only determining factor here. What types of applications you most commonly run can play a role in what security automation you focus on. Are you creating a lot of web apps? You may want to invest in automatic SSL certificate provisioning. Lots of databases or regulated data? Make sure you have encryption built into the platform from data at-rest to in-transit.

There are too many considerations to list here, but I hope you’re getting a better idea of your goal. At the end of the day, the more security you’ve built into your environment and the easier it is to use, the faster your teams can develop and the more value you’ll see over the long term.

Finally, it’s a good idea to consider what you do want to show to the developers and their upline management. What sort of security metrics are important to your organization? How do you measure success? In all likelihood, you’ll want to have easy-to-use views and reports on things like patch levels and known vulnerabilities, and simple actions and automation that teams can use to control when issues are fixed to avoid business disruption (within reason). It’s important that you limit the options here to avoid slipping away from your standards again, but you may even want to use that data to drive further automation and even artificial intelligence (AI) to quickly resolve issues. The endless battle of security means the possibilities are near limitless.

Wrapping it up

I’ll summarize the points I covered here in a TL;DR:

1. Pick your strategic technologies and create standards around them

2. Start defining what success means for you and roll out monitoring to find your urgent issues

3. Pick the tools that are right for your organization and the problems you need to solve

4. Start your rollout and begin tying the experience together

5. Obscure all of your hard work behind a seamless experience to drive the business forward

I hope this has been a helpful read, especially for those struggling with how to move forward when the problem can seem so large. The lessons we’ve learned here at IBM are that you can’t do it all at once, so pick your priorities, take things one at a time, and eventually you’ll be able to bring it all together into something new and beautiful. Best of luck on your own journey!

Thomas Edmonson is a Hybrid Cloud Automation Architect at IBM based in Austin, TX. The above article is personal and does not necessarily represent IBM’s positions, strategies or opinions.

--

--

Thomas Edmonson
Hybrid Cloud How-tos

Career IT Infrastructure Nerd | Hybrid Cloud Automation Architect