So you want your developers to build secure applications during SDLC and not as an afterthought after being dinged by Security reviewers. You got this idea of highlighting security controls via reference architectures that can be easily consumed by developers early on in the SDLC life cycle.

But you are concerned! Would this reference architecture be another “lengthy”, “preachy” document with languages like “Thou shall do this, thou shall not do this…” and not really something that can be easily adopted by developers? If you develop a visual reference architecture infographic and give it to your developers, it will still be a “visual” form of a checklist. Not saying that such architectures don’t have a place, they are very helpful for a 10,000 foot view of the security landscape. But that does not really help your developers, does it?

In this blog, we will walk through some of the salient features of a meaningful security reference architecture and the process required to develop one. We will also look at the challenges that one might expect to face while launching a successful security reference architecture program.

Salient features of a successful reference architecture

These are the 7 jewels that must adorn your reference architecture program otherwise there is a high likelihood that your work might not be adopted by developers.

#1 Targeted to the developers — Always think from a developer’s perspective while framing the mission, vision and goals of the reference architecture and also while creating the technical artifacts.

#2 Relatable — Developers are not looking for another security guidance document. Use a demo application, sample code or other relatable means to demonstrate the reference architecture.

#3 Easily adoptable- Automation and scaffolding must be your favorite tools in order to make your reference architecture reachable. Also, reference architecture program must be added at key entry points of SDLC for easy consumption.

#4 Offering benefit to developers — Why would a developer follow your reference architecture? One example of a cherry on top of the cake for developer is — processes are set it place to fast-track reference architecture based apps through the Security review process.

#5 Offering holistic and mappable security- Security controls demonstrated in your reference architecture must be mapped to your company’s GRC standards or in some cases may go above and beyond to create new GRC standards.

#6 Iterative — Multiple deployment models demand multiple “iterative” reference architecture. It’s important to not waste effort on a reference architecture that is not useful to developers. Your process to develop reference architecture must have a minimalistic feasibility analysis phase for Go/No-Go decisions

#7 Measurable — Any measure to demonstrate the adoption of the reference architecture program would work here, but one metric that can convey a good story around showcasing security maturity is — how many apps (Or what percentage of your apps) have baked in security using security reference architecture?

So there you have it — the 7 salient features to make your reference architecture more adoptable to the developers. Next we will see the process of creating a sample reference architecture.

Process for creating reference architecture

Let’s take an example so that it’s easier for you to follow along. Most developers today are using a modular form of application development called microservices. These have also become very popular with the advent of managed Kubernetes clusters offered by public cloud platforms like AWS, GCP and Azure.

So consider a scenario where you are a Security Architect and want your developers to follow your prescribed reference architecture to develop secure microservices apps on a public cloud platform like AWS.

Phase #1 Start with a security blueprint

You can call this anything else but when I say “blueprint”, I am referring to the paper exercise for creating a visual reference architecture. The objective here is to “quickly” develop a good starting point highlighting key security topics but not deep diving into any of the technical areas. I quoted quickly as this phase can help you discuss the efficacy of the reference architecture with minimal effort

For our example, let us see the paper architecture of our ecommerce microservice application:

Overlay the paper architecture with your security controls. These controls can be open source, cloud provider specific or specific to your organization. This of course would require good knowledge of the current security tooling of your organization.

Quick tip → Before proceeding to next phase, talk to your Security Engineers, Security Architects and other Security Leaders, and discuss the efficacy of the security blueprint. Don’t waste your time and effort on something that is not going to be useful to your developers.

Phase #2 Build a sample application or use an open source one

Developers love using code samples and our objective with reference architecture is just the same to offer them meaningful security integration code samples. But for that we need an app.

For the microservices example, we can select from a number of sample apps that are available online — Sock Shop by Weaveworks, Online Boutique by GCP, etc. The example app architecture we used in the blueprint above is from the Sock Shop demo app by Weaveworks. Shop Sock is a cloud-native microservices demo application. This application is a web-based e-commerce app which allows users to browse different varieties of socks, add them to the cart and buy them.

Quick tip → While onboarding the sample app to your organization, be cautious of using all the developer best practices of your organization like Git setup, CICD tools, third-party package onboarding, golden AMIs, etc. As your sample code would be adopted by developers, you should not employ “shortcuts” for application onboarding

Quick tip → Where possible, reduce operational overhead on your team by using managed services for infrastructure

Phase #3 Integrate security controls with the sample application

This is the phase where you start to secure the sample application and offer guidance to developers while doing so. Outcome of this phase would be “How to” guides illustrated via the sample code of your demo application. Developers can directly copy your code and integrate with their own applications thus making it easier to follow along with these guides. At the same time, the guides must also be mapped to your organization’s GRC standards for backward traceability.

The security choices you make at this phase are very critical and your thought process (pros and cons comparison between tools) must be extensively documented. Some examples are — comparison between container CNIs like Cilium vs Calico, between service mesh like Istio or Linkerd, between managed and unmanaged security tools, etc.

Some example of security controls applicable to our example scenario and also highlighted in Figure 2:

1- Create and use secure CICD pipelines for your application and infrastructure code integrating with secure scanning tools like static analysis, credential scanning, third-party product scanning, secure Git repo, etc.

2- Inject trust certificates into application containers using organization PKI server

3- Integrate front-end with single sign-on server

4- Create least privilege IAM roles in AWS using IAM Access Analyzer

5- Encrypt service-to-service traffic with mTLS using ServiceMesh like Istio

6- Create service to service authorization using ServiceMesh authorization policies

7- Store, retrieve and rotate secrets using Vault

8- Encrypt cloud storage with AWS KMS

9- Monitor service using Zipkin

10- Centralize the storage for infrastructure (Cloudtrail, VPC flow logs, EKS control plane) and OS/app logs

11- Restrict network access with AWS security groups, NACLs, WAF, etc.

12- Secure connectivity to on-premise using AWS DirectConnect

13- Enable fine-grained access control in Kubernetes using Kubernetes access control

14- Detect and remediate cloud resources misconfigurations

Quick tip → Handover the maintenance of security controls documentation to Security engineering teams in your organization like Vault, IAM, PKI, etc. to reduce the operational overhead of such documentation on your team. Also, handover the mapping of compliance solutions to your organization’s GRC team

Phase #4 Make it easy for developers to use your reference architecture and track adoption

Use automation to convert your “How to” guides into one-click templates or create “Hello World” templates which are fully integrated with all the security controls proposed in your reference architecture. You can use any of the following to accomplish this starting with the least resource intensive:

• Use already existing scaffolding tooling in your company. Partner with the scaffolding team engineers and automate your security controls
• Offer your controls as infrastructure-as-code scripts (IaaC)
• Create your own scaffolding templates

Use metrics to track adoption. Some example metrics include — new teams adopting your scaffolded reference architecture templates, increase in documentation views for “How to” guides, increase in Git forks, etc.

Quick tip → Do not recreate the wheel with automation or scaffolding. Use your existing organizational tooling where applicable.

Challenges that you might expect to face

• Operational and maintenance challenges for sample app—You are a security architecture team, not an application deployment team. Your sample app needs to follow organizational CICD best practices and also needs to maintain patching and other ongoing operational requirements. This would mean extra overhead but it also gives your team a better understanding of developer daily routine and thus increases your team’s developer sympathy.
• Security controls becoming stale over time — Your team wants to develop new reference architecture but is stuck updating and maintaining the previous one. One possible solution is handing over maintenance to individual security engineering teams.
• Not enough developers using your reference architectures — It is important to understand developer requirements before jumping into a reference architecture. Talk to developers who have gone through the process and ask for their lessons learned with integrating security. Then make it easy to adopt your reference architecture by tooling or sample code. Refrain from only using theory to educate developers.
• Handle scope —If your team is not full-stack security then scope your reference architectures accordingly and make it clear the aspects of security that your reference architecture solves for, while caveating the missing areas.

So there you have it, some guidance to Security Architects looking to roll out their own reference architectures for developers. Please feel free to leave comments if you have questions or suggestions to improve this read.

References

Sock Shop by Weaveworks

Online Boutique by GCP

In the security world, you might have heard of the exploit used by hackers to reveal passwords from their hashed counterparts. We call this technique password cracking or in practicality ‘password guessing’. Even with the complexity of password controls put in by organizations today, this threat is very much real. This tutorial is intended for any individual with a mindset of security who wants to learn more about how hackers are able to crack Windows stored user passwords.

Introduction to hashing, rainbow tables

Hashing is a software process of generating fixed character length hash values for a text file. This is a one-way function meaning the original text file cannot be generated back from the hash value. This hash value is used to verify the integrity of original text when it is sent over a communication medium. For example, when A sends a text message to B, it first creates a SHA-2 (popular hashing algorithm) hash of the message and sends it along with the message. When B receives the message, it also creates a hash of the text message using same SHA-2 algorithm and compares it with the hash provided by A. If the hashes match, B can be rest assured that the original message has not been corrupted on the way.

Application engineers also use this technique for securing passwords of users logging into their systems. Instead of storing passwords in the back-end database in clear text, password hashes are used. This protects clear-text passwords from internal application developers and also from hackers in case they are able to breach the database. Hackers are cognizant of this process and have lot of tools in their arsenal to efficiently guess the passwords from the hashes. I use the word ‘guess’ because remember hashes are one-way functions, you cant decode them like you can do to an encrypted string. You would need to create a hash of a guessed password and compare to the extracted hash to determine if you have guessed correct.

Free online tables are available which store password hashes of common passwords which can make a hackers job lot easier if people are not serious about password complexities. These tables are called rainbow tables or hash tables. In case of complex passwords, there are free tools which use a brute-force approach of comparing hashes of multiple combinations of text. Regardless of the approach being used, it is appropriate to state that password hashes are NOT SAFE if in the hands of an ill-will hacker.

Windows hashing basics

You really need to know only the following three basic concepts before extracting Windows hashes:

LM hash

LAN Manager (LM) hash is an old and weak Windows technique for creating hashed passwords, which has been disabled by default in current Windows environments. But this can still be enabled manually on current systems — See Microsoft documentation on how to protect your systems from using it:

Network security Do not store LAN Manager hash value on next password change (Windows 10)

The reason why LM hash is easier to break is because passwords are not case sensitive, password length is maximum 14 characters and more importantly because it breaks the text in two halves of seven characters before hashing them separately and concatenating. So if your password is less than seven characters, it should be a breeze for a hacker to guess the password. [1]

NT hash or NTLM hash

New Technology (NT) LAN Manager hash is the new and more secure way of hashing passwords used by current Windows operating systems. It first encodes the password using UTF-16-LE and then hashes with MD-4 hashing algorithm.

If you need to know more about Windows hashes, the following article makes it easy to understand [2]

SAM database file

Security Account Manager (SAM) is the database file that stores the user’s password in the hashed format. You would need access to this file in order to retrieve hashes from your local or remote Windows machine [3]

Extracting local hashes from Windows Server 2016

In this section, I will show you how to extract hashed passwords from your Windows desktops using a very popular and powerful tool — mimikatz. The screenshots are from Windows Server 2016.

Step 1: Download mimikatz

Binaries are available at — https://github.com/gentilkiwi/mimikatz/releases

Step 2: Run (regedit)

Step 3: Navigate to HKEY_LOCAL_MACHINE and export SAM registry file and SYSTEM registry file to the same directory as the mimikatz installation. Save the files as “Registry hive files”

Your mimikatz directory should look as below:

Step 4: Run mimikatz.exe and type “lasdump::sam” command followed by the file paths of sam and system file:

lsadump::sam sam3.hiv system.hiv

If you get an error as below, you will need to elevate permissions of mimkatz

Step 5: Type “token::elevate” to elevate the permissions

Step 6: Type the lsadump command again and you should now see the hash values of local users

Confirm if you got the right hash

Use Windows commands to create local users and extract the generated NTLM hash using the above process. Once you have the hash, use the below online utility to generate hashes by yourself and confirm if it matches.

https://www.browserling.com/tools/ntlm-hash [4]

Windows commands for user and password modifications:

List of all users → net user

Add user → net user /add username -key=”password”

Update password of user → net user username newpassword

Other tools that can be used in place of mimikatz:

HashSuite, fqdump, pwdump2

Password cracking/guessing tools:

L0phtCrack, Cain and Abel, John the Ripper

A quick note on Salting

Salting is a quick way of increasing the security of your hashed passwords. Passwords first are concatenated with a randomly generated set of bits (salt) and then the hash is calculated. Even if users have same password, they will have different hashes since the salt is randomly generated for each user. Salting also protects against rainbow tables since the table now must contain “salt.password” hashes which is unlikely for a long and random salt value. [5]

Sources

1. https://en.wikipedia.org/wiki/LAN_Manager
2. https://medium.com/@petergombos/lm-ntlm-net-ntlmv2-oh-my-a9b235c58ed4

3. https://en.wikipedia.org/wiki/Security_Account_Manager

4. https://www.browserling.com/tools/ntlm-hash

5. https://en.wikipedia.org/wiki/Salt_(cryptography)

Hello multi-clouders! I recently started working on a Microsoft Azure engagement and wanted to get a quick introduction to Azure service landscape. So I started mapping Azure services to the two public cloud platforms I am most familiar with— AWS and Google Cloud.

I have initially started with the key cloud computing areas — Compute, Storage, Database, Networking, DevOps, Governance and Security. Hopefully in the future updates, I can incorporate other areas — IoT, Big Data, Machine Learning and Analytics. But for now, I want to provide you all the baby steps I have taken towards mapping cloud services across the three Public cloud platforms.

Providing screenshots of the excel since Medium does not have a good table structure, but if you like the content, feel free to use the Airtable link below to download the CSV.

Airtable with raw CSV data

Cloud migration can be a nerve-wracking experience for organizations looking to move their on-premise resources to the Cloud. In this article I will talk about some of the most important Networking things to keep in mind before you start developing your own strategy for cloud migration (specifically targeted towards AWS and GCP). This is a two part series with the first part focusing on Cloud Connectivity and the latter geared towards Networking in the Cloud.

Cloud Migration background

Organizations today have multiple use cases for choosing cloud over their own infrastructure. Some examples are:

1. Utilizing cloud compute and storage services for Big data processing jobs which can be pretty expensive to do on-premise.
2. Backup or archival of data to Cloud for cheap, durable and highly available data requirements.
3. Developing a Disaster Recovery solution on the cloud to act as active-active or active-passive replication system.
4. Migrating entire infrastructure to the cloud for the long-haul or employing a hybrid approach.

Whatever your cloud migration use case may be, one thing is clear that Networking is the most important capability to make the migration happen seamlessly and most importantly — securely.

Connectivity and Data Migration to the Cloud

One of the first most critical pieces to any cloud migration is the connectivity between on-premise data center and cloud environment. Efficient and reliable connectivity helps in solving data migration challenges which are critical in building any cloud migration project. Lets take the cloud migration use case, where an organization wants to leverage cloud’s compute capability to run big data solutions. In this scenario, possibly petabyte scales of data needs to be transferred from on-premise network to cloud services like AWS S3 or Google Cloud Storage bucket. A good connectivity strategy would help ensure that this data is transferred in a fast and reliable manner. Let us first review some of the options that cloud service providers (CSP) have today to fulfill such connectivity needs:

1. CSP-managed VPN : Both AWS and GCP allow the creation of a managed-VPN connection over the public Internet using IPSec protocol suite. A virtual private gateway is created at the side of the cloud provider and is then connected to an on-premise router using the authentication and encryption security configurations of the IPSec protocol suite. The cloud provider takes care of redundancy and high-availability at there side by automatically replicating VPN endpoints to two different data centers. The virtual private gateway also supports the use of dynamic routing using BGP routing protocol so that the router automatically learns new routes and does not need to be reconfigured in case of changes in the infrastructure networking theme. Amazon also offers VPN CloudHub service to create a hub-and-spoke model by connecting multiple on-premise data centers through a single cloud gateway.
2. Customer-managed VPN : Customers can also deploy there own VPN solutions on the virtual machines in the cloud for creating an IPSec VPN tunnel to their on-premise network. It goes without saying that customers would be responsible for creating redundancy in there design by deploying the VPN endpoints across multiple availability zones.
3. Private Network Connection : This is the fastest (low latency) and the most reliable option for connecting to the CSP network by using a dedicated fiber connection to the CSP’s endpoint. Network speeds can be reached from 50 Mbps up to 10 Gbps per connection. AWS service for the dedicated network connection is called — DirectConnect and in the GCP world, this is called Google Cloud Interconnect.
4. Data transfer via connection with Cloud Storage endpoints : For customers looking to transfer data to their cloud storage services like AWS S3 or GCS bucket, there are various tools available to make this a smooth process. Both S3 and GCS have a GUI or CLI commands to directly upload the files to the bucket in your desired cloud region using SSL to encrypt data in transit. Some of these tools for AWS are rsync, S3 CLI and Glacier CLI. In the GCP world, we have gsutil and Storage Transfer Service. While this may be a good solution for small distance transfers, this is not a good solution for transferring data across long distances. Amazon offers the S3 Transfer Acceleration which maximizes data transfer across long distances by taking an optimized network path. Amazon also provides a service called Storage Gateway which can help establish a permanent and seamless gateway between your on-premise applications to AWS S3, EBS and Glacier. This is most suitable for organizations working towards a hybrid cloud storage model.
5. Data transfer using physical transport (Offline) : For customers wanting a fast and secure way to transfer petabyte-scale data without creating a connection to the cloud provider, there are options to get your data physically transferred to the cloud provider’s edge location. Even with high-speed online network connections as described above, it can take days or even weeks to transfer humongous amounts of data to the cloud which is a pretty common scenario with Big data solutions in the market today. For example, a 1 Gbps network connection would take around 12 days to transfer 100 Tb of data. Using tamper-resistant and encrypted physical devices to securely transport the data, can significantly increase data migration time frame. AWS service for the physical data migration service is called — AWS Snowball and in the GCP world, this is called Google Transfer Appliance.

In the next part of this series, we will focus on some of the things to keep in mind after connectivity to the cloud is established and we are ready for the next level - Networking in the Cloud. This is the fun world where vital networking functions are created with a few clicks on the interface!

Researching on RESTful APIs, I found the use case of AWS services to build Serverless websites very appealing. This can help small to mid-size organizations to reduce their operational overhead while delivering highly scalable and reliable services to customers. In this blog post, I will document steps to develop such a serverless website in AWS.

Step 1: Creating a static web front-end

We will be using the unique public url for AWS S3 bucket as our web front-end. The S3 bucket would be simply used to host static content for the website like HTML files, image files, CSS files, etc.

1. Login to AWS and select your preferred AWS region
2. Create a bucket with default configurations using your preferred cloud region
3. Upload the contents of your static website into the S3 bucket — CSS, Javascript, HTML, and images. Your S3 bucket should look like this after the procedure:

4. Add a bucket policy to allow READs from the public. Below is the JSON that your bucket policy must have to allow Public READ access:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow", 
            "Principal": "*", 
            "Action": "s3:GetObject", 
            "Resource": "arn:aws:s3:::[YOUR_BUCKET_NAME]/*" 
        } 
    ] 
}

Note: Depending on your bucket setting, you may or may not be able to do the above operation. If an error like ‘Access Denied’ comes up while granting updating bucket policy, please first update the Public Access Settings to remove the block on updating the bucket policy.

5. To enable users in the region of your S3 bucket to be able to see the contents of bucket as if they are coming from a website, you need to enable website hosting on your S3 bucket. This can be done from Properties of the bucket:

6. Visit your S3 endpoint to see your implementation of static website at:

http://<bucket-name>.s3-website-<region-name>.amazonaws.com

Step 2: Creating a Server-less backend process

Amazon offers a fully-managed SQL database through RDS and a fully-managed no-SQL database using DynamoDB. You can use either one for your serverless solution depending on your use case. For application level logic like getting or putting data from these databases, Amazon offers a very neat and powerful solution using AWS Lambda. In this section, we would create a new NoSQL table in DynamoDB and do a test to insert data into it through AWS Lambda

1. Go to DynamoDB, select your Region, and create a table.
2. Create an IAM role for AWS Lambda allowing it to be granted below policies:

a. AWS managed policy = AWSLambdaBasicExecutionRole

b. Inline policy with the below JSON, allowing PUT permission to the DynamoDB:

{
 “Version”: “2012–10–17”,
 “Statement”: [
 {
 “Sid”: “VisualEditor0”,
 “Effect”: “Allow”,
 “Action”: “dynamodb:PutItem”,
 “Resource”: “<dynamoDB table ARN>”
 }
 ]
}

3. Create a Lambda function at your preferred region, attaching the above-created role and selecting your preferred programming language.

4. Code your Lambda function to receive a POST-type request, capture necessary details in variables, and put these variables in the DynamoDB/RDS using AWS-offered SDKs.

5. In order to test this implementation, create a Test Event matching the details of the POST request. Test the Lambda function against the test event and see if values are populated in the back-end database.

Step 3: Expose Lambda function using RESTful API

The back-end process we created in Step 2 is a stand-alone process and is not connected to the front-end in Step 1. Before the birth of API Gateway, AWS Lambda was just a cloud event-driven coding platform. API Gateway has revolutionized the way in which Lambda can be used for delivering application logic outside of cloud events. In this blogpost, we would be using API Gateway to expose the functionality of the Lambda function we created in Step 2 as a RESTful API.

1. Inside AWS console, click on API Gateway and create a new REST API
2. Create a new resource inside the above created REST API, and assign it to the resource path desired.
3. Create a new resource method inside the above resource to specify the type of REST API method — POST, GET,etc. Then select the Lambda function as the integration type which you have created as part of the previous step.
4. Deploy your API and copy the Invoke URL created to be used as part of your website configuration files

For the purpose of this demonstration, we have taken authorization for this API as None. However, in most of the cases, you will need to authorize your APIs using JWT authorization or API keys or other methods as considered necessary.