Laying the Foundation for Cloud Migration at VA
The acquisition of — and transition to — modern IT infrastructure is an important way USDS can make a big impact for government agencies. This post is Part One in a two-part series discussing how Digital Service at Veterans Affairs (DSVA) helped the agency adopt cloud technology and the foundation the team built in order to make migration easier for future iterations.
In 2010, the U.S. Chief Information Office established the “Cloud First” strategy, which marked the first government-wide effort to establish the case for cloud adoption. The push to promote cloud adoption across the government kicked off some initial efforts, but agencies were still slow to adopt cloud computing, and, in many cases, is just now beginning to proliferate.
We began our procurement efforts with a discovery sprint looking at past efforts. We quickly learned that while VA had undertaken cloud-related efforts, nothing had made it off the ground in an official production capacity. We dug into what was available, what was working, and what wasn’t. While we found some utilization of cloud resources to support development efforts, nothing was in an official production capacity and it was unclear as to what next steps should be.
The Importance of Stakeholder Engagement
All too often, projects will get awarded and run full steam ahead…into a wall of struggle to get into production. These roadblocks often come in the form of outdated policies and procedures that prevent cloud utilization. The key is to get in front of the policies, people, and procedures that have the most impact and work with stakeholders to address concerns and provide solutions collectively.
The DSVA team found it was critical to engage the following stakeholders in our efforts:
- Inspector General (IG): The IG’s job is to be critical, but that is to ensure the safety, security and success of an agency. The IG shouldn’t be looked upon as someone who will shut down a project, but rather someone who can make it better.
- Network Ops: These folks are the gatekeepers to all things network connectivity-related.
- Accreditation and Authorization (A&A): This can take on different forms and duties within an agency, but typically it starts with an Information Security Officer (ISO) assigned to support a project and can make its way up to the Chief Information Security Officer (CISO) for Authority to Operate (ATO) approval.
We worked with each set of stakeholders to ensure that our cloud efforts would be successful for the long run. The Inspector General’s office was largely concerned with data access. We were fortunate enough to be present in cloud meetings with the IG’s Office and worked with them to address their concerns around data access. After gaining an understanding of the information and data they needed to see, and how we could provide that within the context of the cloud, we gained their trust and approval to proceed.
We found it best to work with Network Ops early on to ensure compliance (e.g. TIC) and simply to guarantee that data will be able to pass to and from the cloud. We ran into a situation that required a bit of work and creativity from Network Ops and our team because we were unaware of the capability of the hardware devices run by VA and their indirect compatibility with the Cloud Service Provider (CSP). This led to changes in our architecture and deployment procedures for monitoring and ensuring stability. By working with Network Ops in the design phase of architecture, we were able to limit unforeseen roadblocks and surprises that could have come up during implementation time.
For Accreditation and Authorization (A&A), early on we ran into situations where the ISO’s experience was based on obtaining approval for on-premise hosted solutions, which led to square-peg round-hole debates, causing confusion and frustration on both sides. As our team spent more time understanding the lay of the approval land we were able to prevent delays downstream.
Working with Federal Government’s Authority to Operate (ATO)
An ATO is a risk-based assessment and decision to approve a system’s usage. The overall usage and security of a system are evaluated by way of “controls.” These controls define the criteria that must be met in order for a system to be used in an approved, production capacity. Before undertaking the cloud migration our team had to obtain an ATO, and when obtaining the ATO for our cloud migration, the DSVA team defined the following controls:
- The expected FISMA level — The Federal Information Security Modernization Act (FISMA) is a risk management protocol to assess natural and man-made security threats.
- Cloud Service Provider (CSP) Approval — The agency should have accepted the CSP’s FedRAMP package, and approved it for use at certain FISMA-levels.
- FedRAMP-Approved Services — The CSP-specific services utilized within the architecture should also be individually FedRAMP-approved. Each CSP maintains a compliance list that can be quickly checked. Risk-exception may need to be obtained for usage.
Security controls break out into three categories:
- CSP-specific (usually addressed via FedRAMP)
- System/Application-specific (e.g. you are running a public-facing web application so how does your application do X?)
- Hybrid (e.g. CSP provides Identity and Access Management, but how does your system/application use it and/or you manage it?)
The best-case scenario is when Assessment & Authorization (A&A) staff have prepopulated CSP-specific controls within the agency’s Governance, Risk, and Compliance tool. If that hasn’t happened, staff will have to fill out hundreds of security controls for a given project. The DSVA team worked with the A&A team to pre-populate controls to save on time and iteration and omit security controls that were not applicable to cloud technologies.
Contracting Infrastructure-as-a-Service
When this initiative began, DSVA was lacking an enterprise-wide cloud contract that we could leverage. Given our technology stack needs, we set forth on creating our own. Working closely with VA’s Technical Acquisitions Center (TAC) to define our requirements and perform the appropriate market research, we initially executed an Interagency Agreement (IAA) between VA and the General Service Administration for Infrastructure-as-a-Service (IaaS). This has since been replaced by our own contract via NASA SEWP which is a contract vehicle that, in the context of cloud computing, allows us to essentially purchase usage credits of a cloud service.
Capital Expenditure (CAPEX) vs. Operating Expenditure (OPEX)
Aside from telecom services, the government has yet to adopt a monthly billing model (OPEX) that is attuned to pay-per-use services and instead requires all contracts to be funded for the entire fiscal year (CAPEX). The problem with procuring a contract for annual cloud services is that, for new and/or growing projects, it can often be difficult to accurately estimate usage costs for an entire year, especially if an agency has never run cloud projects and so cannot use past information in support of a baseline budget.
The DSVA team advocated that contracts awarded for the usage of cloud services should be thought of as a “line of credit” where each month usage costs are drawn down off the yearly amount. This way, even if funding is allocated for the full year, because the provider is being paid monthly, any money left over at the end of the contract year can be rolled into the next annual cycle.
We haven’t seen a reason why anyone using cloud services would pay a provider an entire year up front, unless there is an explicit guarantee in the contract that you’ll get the remaining money back. That said, each provider has a calculator that will provide an estimate based upon the intended configuration and/or usage. We’ve found this initial estimate to be a solid starting point, but best practice is to add 10–20 percent to account for any unplanned expenses.
Lessons Learned
Including agency stakeholders at the beginning of a cloud migration initiative can save a lot of time and headache in the end. Our CTO gave the DSVA directive to begin the cloud acquisition process, which opened doors to working with VA partners to see real progress in agency wide cloud offerings. By working with each of these stakeholders early on in the process, we were able to avoid many of the roadblocks that can often impede agency IT projects. We were also able to set up the proper controls in the ATO to ensure efficiency in future iterations as the project progressed.
With the concerns of the Inspector General, Network Ops, and Accreditation and Authorization taken care of as well as ensuring the infrastructure contracts would work for cloud hosting as infrastructure-as-a-service we were able to shift our focus to the next step: Initial Deployments. Stay tuned to learn how DSVA used Vets.gov and Caseflow to become one of the earliest adopters of cloud at VA.
The best of technology.
The best of government.
And we want you.
We’re looking for the most tenacious designers, software engineers, product managers, and more, who are committed to untangling, rewiring and redesigning critical government services. You’ll join a team of the most talented technologists from across the private sector and government.
If you have questions regarding employment with the U.S. Digital Service, please contact us at usds@omb.eop.gov and visit usds.gov/join.
