Peeling the layers of proactive security engineering practices

[Attribution: Woodcut of Nuremberg by Hartmann Schedel, Public domain, via Wikimedia Commons]

“If you reveal your secrets to the wind, you should not blame the wind for revealing them to the trees” — Kahlil Gibran

If there is one theme in world history that keeps recurring — as narrated by the lives of Sumerians at Mesopotamian Uruk, the massive Aurelian walls of Rome, the impressive location of Hohensalzburg Fortress in Austria, or the imposing Mehrangarh fort in Jodhpur, India — then, it ought to be the one that relates to the need for protecting the people, settlements, wealth, from their potential aggressors.

The majority of such powerful administrations resorted to fortification as one of the means for protection. Some were simple and tall walls built around the settlement, while others were a complex maze of layered constructs with moats filled with water, pre-walls, gates, killing fields, keeps, stairs, doors, elephants, inner administrative blocks on formidable hills, you name it! While most of these efforts were effective in thwarting external threats, espionage efforts — as documented by Sun Tsu in 4th century BC, or later by Chanakya of Arthashasthra fame, or in modern world wars — necessitated inventing different strategies to identify internal actors like spies. History also taught us that the most impregnable fortresses were those led by the discipline of determined leaders & warriors, and not the forts themselves — and some did stand better than the others, for good reasons.

The relevance of technology was always at play over the millennia. What was initially thought to be unbreachable, was compromised eventually with the help of newer technologies. For example, medieval-style fortifications were largely made obsolete by the arrival of cannons in the 15th century. The arrival of gun powder and explosive shells in the 18th and 19th centuries led to the ineffectiveness of traditional constructions like bastion or polygonal forts. The invention of airplanes led to the evolution of newer and modern strategies involving diplomatic alliances, international treaties, in addition to traditional & technological deterrents.

It is evident that no single strategy worked/works — and the takeaway from history has been simple — the more value there is in what you are trying to guard, the more are the chances of aggressors wanting to claim it. Proactive preparation, multiple layers of deterrents as preventive measures, paired with constantly evolving security strategies have helped stop or delay such breaches.

Fast forward to 2021!

You may be a startup, thinking about building your solutions the secure way from the ground up, or an enterprise writing newer microservices or migrating your on-premises infrastructure into the cloud world and making it secure from the outside first, the analogy of multiple defendable obstacles from military history is highly applicable in our SaaS worlds. Information is wealth and a massive responsibility.

While the specific details needed to ensure how a product offering can continue to be secure can differ vastly from product to product, there is certainly a common set of diligent practices that can apply to most SaaS offerings. These practices encompass and expand the typical phases of SDLC — requirements-gathering, design, implementation, deployment, and monitoring. Today’s SaaS solutions are not monolithic but have tens, hundreds, and thousands of applications or microservices that are arranged as a collection of loosely coupled services that interact with each other, and with the outside world. These services typically end up running in multiple physical machines, in some cases, across geographic regions (even continents). The entire footprint of interactions & collaboration among these components and the protection of sensitive data thus becomes the concern of our security layers.

Security in deployment & monitoring layers

If you are using cloud platforms like AWS, Azure, or GCP to deploy your product(s), your challenge mostly becomes that of choice and arrangement of cloud platform services, compared to reinventing the deployment wheel. As of this writing, AWS is providing over 250 service offerings from various of its product lines and a similar volume of rich offerings from other service providers. With these many confusing options to mix and match, you have to go back to the basics to ensure that your security layers guard your product.

1. What strategies are you considering to minimize the surface area of impact from any potential threats? Have you considered separating your API/UI server layer from the rest of your backend infrastructure by keeping them in separate VPCs or DMZs? Are bastion hosts relevant for your setup to service access from untrusted networks or clients?
2. Most of the Cloud service providers supply hardened OS images(hardened images per CIS benchmark, for example), which get frequently updated and validated against known threats and relatively more secure out of the box. Is your IaC using these hardened images for deployment?
3. If your business requires you to create dedicated VPC/instances for your customer, have you considered adding customer’s WAF rules to make it more secure for any interacting customer application?
4. Have you segregated your dev, QA, and production instances into separate private clouds? Where is your common build and tools cloud located? Are you using secure VPC peering when these clouds have to collaborate?
5. Have you carefully crafted your Role-Based-Access-Control (RBAC) mechanism? Have you created separate policies and roles for different types of actors in your cloud ecosystem (developer, QA, Ops, Admin, etc., for example)? Do you start with a deny-all as the default policy and selectively start opening up access to resources based on what the role needs? Are these privileges added/removed based on scripts rather than them being manual? Are you using a layer of MFA where possible? Are accesses to your instances and other resources are auditable (e.g., plain ssh vs SSM)?
6. If your company uses Identity Providers like Azure Active Directory or GSuite, have you considered tieing the cloud RBAC role into these ID providers via SSO so that role management becomes less error-prone?
7. Do you insist that each of your production deployments is tested by your QA for VAPT (Vulnerability Analysis and Penetration Testing)?
8. To prevent variants of supply-chain attacks, do you rely on tools to identify security issues in the open-source/vendor libraries? How often do you audit them? How much control do you have to dictate where these libraries get pulled from during the build?
9. How robust is your monitoring and alerting infrastructure/framework? Can you detect the failure in that infrastructure itself? Does your monitoring service aggregate security issues related to deployment(static) and runtime? Have you created run-books for remediation when alerts are sent? Are these run-books periodically reviewed and updated?
10. From a monitoring and conformance point of view, try to bring findings from multiple systems like IDS/IPS (overall deployment health, threat reports, vulnerability summary, endpoint protection summary, file integrity monitoring, weblog analytics, etc.), WAF, Security Hubs, etc. into a single dashboard. Carefully choose policies to set the threshold for each type of alert and the remediation process.

Security in application layers

App security is not an afterthought. You have to carefully infuse security at your building block levels. It may be prudent to ask and address questions, like the following when services are designed/built/refactored:

1. Where are the credentials for your databases, API keys for external services stored? Can you confirm that none of them are stored as cleartext, in a config file, hardcoded strings in code, etc.? Do you make a point to periodically audit for the existence of default and/or very simple passwords for your services?
2. Have you considered introducing a credentials services (or secure secrets manager) component in your topology to encrypt your secrets before storing them on disks? Does this layer support the concept of leasing, revoking, and renewal of client tokens? Does it help dynamically generate secrets so that your deployment and auto-scaling are simplified? Does it help with the periodic rotation of keys?
3. If your architecture supports multitenancy, have you considered using a separate credentials server for each tenant (or a logical group of tenants)?
4. If you publish REST-like API services, how are your access tokens managed? Have you built the methods for the expiring, renewing, and revoking of such tokens?
5. Some of the most important and valuable things you have to guard are your customers’ PII (Personally Identifiable Information) & PHI (Protected Health Information) data. Are you encrypting data at transit and rest? Are you using anonymization when you are logging information to a potentially centralized location/server? Do you have ways to audit your logs, APIs for PII/PHI breaches?
6. Do you periodically review what gets sent out as part of your API responses for any PII or other vulnerabilities?
7. There may be a very good chance that your non-public-facing components talk to each other using Kafka-like streaming buses, RabbitMQ-like message queues, or web sockets. Going back to the aforementioned analogy of espionage and some of the most recent examples of supply-chain attacks — do you have a mechanism in your core infrastructure to confirm that the producers and consumers have established the trust (authentication and authorization) with your core system to access your messaging transport layer?
8. Do you have methods to uniquely identify each of your producers and consumers? Do they have a way to get some kind of unique token ID or App ID after registration with your core system, before message queueing can begin?
9. Does your messaging layer have an in-built security layer to allow only trusted participants to publish and consume messages? Are there ways to re-validate the established trust periodically?
10. Do you have the means to approve & attach roles (read-only, read-write, etc.) to your components during the initial registration? Does your messaging layer honor such roles?
11. In line with the point made in #7 above, how can you make sure that the clients of the credentials server are trusted? If you are supporting integration with external ecosystems, do you have a way to confirm the unique identity of your integration client?
12. Are you keeping OWASP top 10 vulnerabilities at the back of your design and implementation envelope at all times?
13. Then, those usual diligences: Have you deployed using HTTPS and the latest version of TLS? Redirected HTTP to HTTPS? Can you confirm that your application/microservice does not use the default port numbers? Stronger password policy, a shorter window of validity for JWT tokens? Does it make sense to tie as many external services to SSO as possible? etcetera.

Security via policies

When security policies are diligently drafted and adhered to by the security stakeholders of your company, the emerging practices will drive multiple functions to work in unison to keep the protection levels much higher and continuously improving.

1. First and foremost, make sure that there is a dedicated charter or a Special-Interest-Group to take the overall security objectives forward. Ushur runs an in-house SIG under the guidance of CISO, called SROC (Security & Reliability Operations Center). This group, which has representative members from various functions like design, development, QA, DevSecOps, Customer Success, Product Management, etc., meets at a very regular cadence to review policies, respond to SIEM events, review and recommend the latest tools and methodologies, perform threat-modeling, and run the mock audits, to name a few. A lot of Ushur’s proactive security engineering practices are influenced by this group.
2. Make sure that your team understands your cloud service provider’s shared security responsibility model and your share of responsibilities to make the deployment secure.
3. Take the latest CIS(Center for Internet Security) controls seriously and build methods to conform to and address the audits' gaps. The majority of the cloud service providers will have a subset of these CIS controls implemented via their SIEM framework (CIS AWS Foundations Benchmark Standards, for example). Address the issues ASAP and keep your SIEM dashboards as green as possible. Make aspects of these, KPIs for your deployment team.
4. Make sure you have policies in place to periodically review & confirm if the definitions of RBAC roles changed over time, how RBAC policies are applied during the onboarding & exit for employees, contractors, partners, customers, etc.
5. Enforce Vulnerability Analysis & network PEN Tests (VAPT) to be conducted at regular intervals by reputed external vendors to help identify vulnerabilities. Have the QA team develop a stronger internal VAPT automation suite and make it mandatory to pass before any deployment.
6. Evaluate at what point in time of your security journey it may make sense to introduce white-hat hackers (ethical hackers) from reputed platforms for a bug bounty program. VAPT tests generally are precursors to this step.
7. As a company, committing to stringent programs like SoC2 Type 2 certification, HITRUST certification & HIPAA compliance to better safeguard PII/PHI information will elevate your team’s security game by bringing in better overall security process improvements across the spectrum.

[Attribution: Hartmann Schedel, Public domain, via Wikimedia Commons]

Preparedness is one the biggest forms of strength, particularly, when it relates to security. Breaches can be extremely expensive and can shake up the existence of any business, if not proactive. Following well-proven preventive practices, coupled with continued education, can become our best friends in this journey.

Add more deterrent layers, and oh! make sure that the wind doesn't get a whiff of your secret!