Leveraging your Role as Technical Product/Project Manager to Improve Application Security

Published in

Walmart Global Tech Blog

11 min readMar 12, 2021

Are you a TPM? You can play a critical role in helping to drive a security-first culture for your team. You don’t need to become a security expert, but you can help implement a few small regular behaviors and checks for your team to help prioritize security and avoid delays from security bugs found late in the game.

Why Technical Product/Project Managers (TPMs) Need to Care About Security

Secure Code is Good Code

Application Security must be an upfront, and front of mind concern for all developers, as well as all TPMs. First and foremost, secure code tends to be good code. But what do we mean by good code?

The code is easy for someone new to the project to read, the flow is clear
The code is also easy to maintain and extend — you can have high confidence changes won’t break existing functionality
The code can be tested and validated using a repeatable process
The code is kept simple and DRY (Don’t repeat yourself) to minimize dependencies and corner cases

When an application is vulnerable to pure security bugs (like when the bad guys can bypass a login check), it is also often vulnerable to logic and business rule violations and logical bugs as well. This means your app is not only insecure, but likely is not meeting functional/business requirements (🤯 Got your attention now I bet!).

At a high level, a secure app is one where you know that the intended processing will function as expected and unexpected flows/processing are prevented. To be confident in your apps functionality, you must know what all the functions and algorithms being used actually do and how they can be misused.

Are deadlines important to you?

From a more direct and practical standpoint, insecure code will be detected and will not be allowed to run in production. While your organization may not be blocking builds based on security today, this is becoming the norm from the Fortune 500 to small startups. If your goal is to deliver your app to your customers/users, it must pass all internal security checks before it is ever allowed to go live.

This means that when security is not considered up front, developers will be forced to figure out how to fix their code at the end of the development effort/sprint, before they can move forward to production; this could result in large amount of rework or changes after time and resources are already consumed.

Security simply cannot be bolted on at the end. You should not expect to maintain your critical path and meet your deadlines if security is not a proactive consideration. Code in production must be ready from Day 1 to meet performance criteria and customer expectations, and it must be secure.

Bottom line: Insecure code wastes time and resources, and will impact your ability to meet deadlines.

The Top Application Security Concepts TPMs Need to Know About

TPMs play a critical role in driving development efforts and setting priorities. Our goal today is to make you conversant in these topics so you can be sure to talk about them with your team members and spot opportunities to proactively address these types of security concerns in your projects/products.

Secret Management

What are Secrets?

An important application security control is secrets management. Secrets are things like passwords, API keys, or other pieces of data used for access control. Secrets management is a broad topic, but the key concepts are (1) properly restricting access to the minimum necessary systems/people, (2) at the minimum/appropriate point in time. Secrets management covers the full lifecycle of a secret, including expiring, rotation(replacing), use/change audit, and incident response.

In terms of application development, the main goal is to remove plain-text passwords for source code. Source code is often available on multiple servers as well as various different developer workstations. Plain-text passwords/secrets are also more likely to be saved in insecure storage or sent over insecure communications platforms by developers.

How to Store and Protect Secrets?

Ideally, secrets are maintained in a centralized, approved internally managed tool. There are many commercial and open source offerings, such a Hashicorp’s Vault that are designed to meet this need. If your organization/company does not already provide a standard offering for all developers, start talking with your Security and DevOps teams; these teams should also be leading the management and ownership of the tool. Using a centralized tool for all teams enables uniform policy enforcement, visibility, and the ability for Security teams to take action during an incident (i.e., security team learns of exposed secret and can then start to take action directly in the secrets management tool). From the TPM standpoint, you should generally not need to worry about overall configuration and implementation; however, you should be familiar with the tooling, how to get support, and required libraries, configs, etc. that your team will need to use.

As a best practice, secrets should be obtained from the centralized tool when the app starts up and loads in the production environment. While it is common to insert secrets during the build process (i.e., in the CI/CD pipeline), this is riskier and less resilient:

The secret will end up embedded as plaintext in the binary file, which could be recovered by a hacker who gain access to your artifact repository.
Apps that call and initialize the secret in production, as opposed to embedding during the build pipeline, can recover quicker during an incident since re-running the entire build pipeline may take a long time or introduce new build errors. When secrets are injected at run-time it usually only requires an app restart

Furthermore, if secrets are encrypted in source code (inline-encrypted secrets) when the decryption key is also in source code (even if obfuscated, in a binary/DLL, etc.) the secrets should be considered plain-text since it is trivial to de-obfuscate or decrypt the values. Designs which rely upon inline-encrypted secrets should be discouraged.

Key Questions to Ask Your Team about Secrets Management

Is this secret being used for multiple apps and services? Generally, every app and service should have its own set of secrets. When a secret is reused, this means that a compromise of some other app can lead to a compromise of your app. It’s not a secret if it’s shared!
Is a given secret used in both production and a non-production environment (e.g., test, staging, QA)? Generally, a secret should be different in production and non-production environments. Non-production environments are not as secure and if the secret is exposed in the non-production environment then the production application is at risk. This include “non-production” testing that actually calls some production service or database.
Do we have an incident response plan for our secrets? This should be a top priority for your team if one does not already exist. Your team must be able to quickly replace a compromised secret (this process is often known as “rolling a secret”) in case of a security incident. If a hacker knows a secret or it is accidentally exposed, you need to replace the old secret immediately. Your team needs to have a plan in place to do this quickly and accurately. On-call team members should know how to generate new secrets for all apps and services. Depending on the app, you may also want to build in an automated process which can be carried out by your org’s incident response team.

Dependency Management

What is a Dependency?

Modern applications are rarely ever built 100% from scratch. There is no need to re-invent the wheel when there is a good solution out there!

Most applications utilize existing chunks of code which are packaged into a reusable set of files instead of simply cutting and pasting. These things are what we generally call a “dependency”, because the application cannot function without this chunk of code locally. Other names that often mean the same thing are modules, packages, and libraries. Typically, the dependency is needed by your app, but is designed to be independent of your app and reusable in a general way by other applications.

While your app may be dependent on third part services, external APIs, databases, etc., the focus here is on the types of dependencies baked into the application and needed for the application’s internal functionality.

Understanding How a Dependency Relates to Your App

For example, your app may use a dependency to streamline the process of opening and understanding the underlying formatting of a certain type of data file. Rather than reinventing the wheel, your app can integrate this dependency. The dependency could be created, owned, and managed by someone who has no knowledge of your app.

Here is another way to conceptualize the relationship: A dependency is somewhat like a browser extension, it provides extra functionality that is not coded by the browser developers. The difference, however, is that the browser can function without the extension, but the app cannot function without the dependency/library.

Where Do Dependencies Come From?

A dependency may be provided by a third-party, another team, or it maybe be a reusable component created by your team itself. The package of the dependency may be manually inserted into your app or retrieved dynamically at build time (when all the code is taken and converted from human readable code to machine executable instructions) through a package or dependency management tool. Ideally, dependencies are not merged into your main app codebase (your repo) and are managed independently (their own repo, loaded at build-time).

If you ever hear your team talking about “maven central”, “npm”, “pypi”, or “nuget” these are examples of tools/services for working with dependencies.

Managing Dependencies

Dependencies are better than cutting and pasting someone else’s code into your own code because they are self contained. This means they can be updated independently and consistently across multiple apps. However, just because they can be updated, doesn’t always mean they are updated.

While instructing your app to use a new version is usually a simple 1–2 line configuration change in the source-code or config file, updates to dependencies may sometimes involve “breaking changes”. A breaking change is where a functionality is modified or changed in the dependency itself and requires a change in your own code in order to continue to utilize the dependency. This mean that to update/migrate to a new version of dependency is actually more than the 1–2 line change; you will need to run compatibility tests and possible modify an app so it runs with the new dependency.

As a result, unless developers need new functionality in a new version, they are typically reluctant to update to a newer version of a dependency. Using the bleeding-edge, most recent version is not always necessary or best from an operational perspective; however, security needs to play a role in this decision making process one way or another.

Understanding the Security Risks

The biggest security concern when it comes to dependencies is ensuring that your app does NOT use a dependency with known vulnerabilities. Software publishers and security researchers often make public reports detailing problems in dependencies. For most dependencies that are actively maintained, a new version is provided reasonably quick. Once a new version is available developers should upgrade/migrate to the version that fixes the security bug.

Upgrading may not always be easy. If your app uses an older version of a library, you may find the security fix is only added to the most recent version of the library (though for major projects, security updates may be “back ported” as minor updates to older versions). As a result, you may need to introduce breaking changes from the newer version of the library in order to get the protections of the security fix. The only other option for your team is to try to manually fix the security bug itself, which may not be easy and could be time consuming. This may also make it harder to move back to the “vanilla”/upstream version of the dependency since your app now depends on your customizations.

For this reason, it is important for your team to monitor and update your dependencies through a regular, defined process. It is much better to provide your team adequate time to update and deal with breaking changes then to pull an all nighter trying to make something work in order to avoid a security hole that could lead to your app being hacked.

What You Should Do

Many package/dependency management tools can ease the process of managing dependencies. For example, the npm tool has sub commands that can tell you if you are using a known vulnerable dependency and makes it easy to find a currently maintained and secured version. You can also run third party scanners, in a class of software known as “software composition analysis” (SCA) tools, against your code base and dependency config files. There are numerous tools like this on the market, such as the OWASP Dependency Check. Many of which can integrate with your build systems and version control systems. You should make use of automation whenever possible, but your team should be prepared to perform periodic manual checks by signing up for package mailing lists, chats, etc. and by manually checking project updates and source hosting sites.

Once your team knows they are using a vulnerable dependency you should be sure to allocate time and resources to upgrade/migrate. More and more organizations are implementing SCA tools as blockers in CI/CD pipelines. Your security team may also scan your app for vulnerabilities or may take action based on news and current attacks; this could even result in your app being taken offline when a high risk security bug is detected.

To help ensure that you can migrate/upgrade in a timely manner with few surprises, it is important for your team to write adequate automated-tests. With sufficient coverage of functions which use a dependency your team should be able to easily run a test with a new version of a dependency to determine whether there are any breaking changes: When developers are rushed and insufficient time is allotted for QA and development of tests, this becomes more difficult because now everything must be tested by hand which is slower and more error prone.

Key Questions to Ask Your Team about Dependency Management

Does our app / new feature have test coverage for all existing/new dependencies? Whenever possible, you should implement formal testing to validate functionality. With good testing in place you can very quickly evaluate updates to dependencies in a repeatable and reliable manner. If tests are not in place, they should be developed as soon as possible and while evaluating new updates to dependencies.
How are we tracking and managing our dependencies? Your team should have a standard operating procedure to regularly review and track dependencies. Many tools that are used to manage libraries, packages, and dependencies will provide reporting on each build or can trigger reports on demand. Your team should also check updates, blogs, and the source code repositories maintained by the dependencies authors to stay ahead of the game.
Do we have any dependencies where we keep skipping upgrades? If you stick with an outdated dependency, it should be for a very good reason. The longer you wait, the harder it may become to migrate to a new version as the number of breaking changes is likely to grow over time.

Wrap Up

While today we only discussed to application security topics, these are areas where TPMs can really lean in and help keep the team focused. While security is often seen as a non-functional requirement, it will be harder to get code out the door with the types of vulnerabilities that can easily be detected and identified. Therefore, you must make it a priority for your team to focus on these items, as well as remediating known bugs in the code your team codes on their own. The end result will be more on-time deliveries and and increase in “good code”. Once you normalize these types of checks, using the provided questions as a starting point, on a regular basis you will make everyone happier — you developers, your user/customers, and of course most importantly, your security team ;-).