The movement of a data center is something that occurs for organizations time and time again. The steps are similar for all of us. The activities you would use to move the data center from one location to another can also be used to move your operation to support a Hybrid Cloud solution. This post aims to collect the steps in order to make it easier for others to move their data center. This post aims to provide a way for you to extract the tasks that apply to your situation and add them to your schedule. Please note, this post will be evolving as I have a lot of steps. In the interim, if you are embarking on a data center migration and need a quick rundown on all steps, feel free to comment on the post and I will try to help as much as time allows. Nobody should suffer any longer on a repeatable model.
As you embark on the buildout, you might be inclined to outsource the entire project. This is a mistake. Your current resources know your environment. If you bring in a new team, you need to spend time to train them on the infrastructure and nuances of your environment. You can bring in team members to augment the staff, but consider carefully the amount of tasks that are required to add an entire new team to your staff.
Step 1: Pick a location, provider and sign a contract.
Pick a provider that you are going to move to. Consider a provider that has multiple locations so you can leverage their backbone network to move your data between locations. This is also important as you need a street address for the circuits. You also need this street address for equipment shipping.
In selecting the location, you need to negotiate for a room where all of your equipment can arrive, be stored safely and configured there. You need to add in a requirement that your team will need an area to sit.
In the negotiation identify how you can have exclusive use of the elevator and path from the loading dock to your cage. If you are doing a large movement of equipment from one location to another you will want exclusive use of the elevator and the path from the source to the destination. You don’t want to compete with other move related activities with other companies on the same day.
The contract should include, the facility requirements for installing equipment in their location. Do they have any paperwork to fill out before you do anything in their space. Do they care how you put your racks together? The contract should also specify how long the facility can take for pulling a cross connect from their meet me room to your cage.
Document a task to pick the provider, get the street address, review with legal and sign the contract.
Step 2: Order Circuits
STOP WHATEVER PLANNING you are doing and order circuits immediately! They take forever to order. You are looking at 90–120 business days to get a circuit in place. You can’t copy a single piece of data without your circuits. Based on the provider you picked you might just need to order a circuit from your office to their location, but order something! Perhaps it is MPLS or dedicated 10G fiber, just order something! Make sure you get plenty of bandwidth. You will grow into it, so don’t go cheap! In the circuit order, specify that the vendor will provide the Letter of Authorization/Customer Facility Assignment. You can use your favorite search engine and search the term to see what this document is used for. The key word search is “LOA/CFA telco.”
On the topic of circuits, look at all of your other office locations and where those circuits go to. If those circuits go to the data center you are moving, then you need to order those circuits as well. You need to look at all of the VPN partners that you have connected to your offices. You need to start a task to move the VPN connections as well. Document a task to order all circuits and start moving the VPN tunnels immediately. Do this step while you setup the rest of the project. You are losing valuable time if you have not ordered them! You need to contact all of your partner companies that have added your internet ip address to their allowed list in order for them to add the new IP Address of your circuit. This is traditionally called “White List”.
One of the first circuits to order is your internet circuit into your new location. This should be a high speed connection. Ordering this at least allows your team to connect via VPN to your corporate office while they are working at the new data center. Alternatively to get started, consider bringing in a connection from Xfnitity or Fios to get basic internet.
When you order circuits, you will want to order redundant circuits from different vendors. This ordering may require physical construction at your offices as you will want different entry paths to the building. My recommendation is to order the circuits and set a different project for setting up circuit redundancy.
Document a task to order circuits and your VPN connections now!
The following items are required on your circuit order
1. What “mode” will be used for the circuit. Will it be single mode or multi-mode?
2. Obtain the LOA/CFA
3. Order the cross-connect from the vendor facility to your location at the data center
4. Upon receiving the circuits you need to test the bandwidth to make sure it meets the requirement you setup.
5. Confirm that the MPLS order will include receiving the full routes from your vendor. In the case of AT&T they have a “Restrictive Routing” setting which could impact your MPLS deployment.
6. Your circuit deployment needs to account for the duration of time that it requires for the vendor to come in and do the splicing. You will need to use ServiceNow (or your equivalent ticketing system) for the vendor to come in after hours or when you prescribe to do the cable splicing
Step 3: Find A Cabling Vendor
Do not proceed with this project any further without a cabling vendor that will be fully engaged for the design and buildout. Cabling has become an extremely complex engineering solution that requires a specialized skill. Your selected cabling vendor needs to know how to design the cabling plant for all of the equipment you will buy. Partner with them on the equipment purchasing so they know what will be needed. Do not proceed any further without a cabling vendor at the table with you. Do not order all the equipment and then call the cabling vendor. Task the cabling vendor with purchasing the cables and the racks. Issue them enough money to keep purchasing cables for the life of the project. Confirm with the cabling vendor if your compute, storage, network, firewall and security team can work in the cage when they are present. Some cabling vendors will want exclusive use of the location for their work. This will kick your entire team out of the data center and potentially causing slippage in tasks. Negotiate with this cabling vendor if the cabling vendor and your team can be in the location at the same time. The cabling vendor should be on a time and materials contract. It will take too long to know all of the tasks associated with the cabling vendor to setup a firm fixed price contract. Set them up as time and materials and get the engaged NOW!
The cabling vendor should take on the task to draw the rack elevations and identify where the equipment will go in the rack. They will need to have experience with the equipment to understand the hot and cold rows along with the ports required between the rows for cabling. Document a task for them to draw the rack elevations in a tool that can export the data to the format of your choice. For many this is AutoCad. Please note, they can’t draw the rack elevations until the bill of materials has been decided upon.
The cabling vendor will need to provide the system or spreadsheet that documents the cabling in the data center. Typically this is a spreadsheet with the hostname of the equipment, the interface type (data, management, integrated lights out management) and the destination port on the network or storage equipment. The cabling vendor will need to take requests from the server teams to assign new ports. In some cases your networking team may be the human resources choice to select the ports on the networking equipment and the cabling vendor chooses the ports in the rack. Consider purchasing software for switch port mapping such as www.switchportmapper.com or www.device42.com.
You need to verify with your security team if the cabling vendor will need to be a cleared resource. If the cabling vendor has to provide a cleared resource at your company, then you should plan to add extra time to get them the security clearance needed. Document a task to determine if the cabling vendor needs to be cleared and if they do, then document a task in the schedule that cabling does not start until all resources are cleared.
Document a task to get a cabling vendor on the executive team. Document a task for the cabling vendor to order a truckload of cabling, Category 6, Fiber and especially MPO cables. The cost of not using the cabling, outweighs the cost of “Sitting around” waiting for it to arrive.Document a task to identify how the cabling vendor can get approvals for cabling changes quickly and document who can authorize the change. This authorization should be more than one person.
Step: Cables Ordering and Installation
The time to order and install cables is far longer than you think, especially if you are ordering equipment that requires MPO cables, order a lot of them early! They take over 12 weeks to order and arrive. The scope of work for your cabling vendor should include: Selecting the length of cables, the organization of the cabling, labels on the cable, providing a final output of the entire cabling in the data center, coordinating testing and certification of the cabling with the cable manufacturer and managing the custom cable orders with the vendors.
In your buildout of the data center, you may want to prioritize the equipment to be deployed for your infrastructure components. You need your foundation components for network connectivity and LDAP authentication servers that need to be installed first. To do that install, you need to know what equipment is needed and their cables need to be prioritized. This cabling needs to determine if the cabling is going across the data center between rows. In order to prioritize this then you need to know what the cabling connections are and whether the cables you want are readily in stock. Again, if you have MPO cables, those are not in stock and you can wait 12 weeks.
Cabling vendors can’t install all cables at once. It takes MONTHS to build out the cabling in a data center. If you are not documenting a task for months of work with your cabling vendor, you are headed for a lot of churn and escalations. You may need to consider temporary cabling for your build out. This is highly discouraged as nothing is temporary in data center build outs.
Your cabling vendor should know that the MPO cables are complicated. They need to know which end will go into the patch panel and which end will go into the equipment. The mapping of the male to female ends needs to be done by the cabling vendor. The mapping requires the cabling vendor to understand if the connection is device to device or patch panel to the device. Document a task to order cables and find a company that will select the cables to order.
Step: Product Roadmaps
You need to tell your organization, what the “General Availability” (GA) of every software and hardware component is allowed in your organization. This data center migration project is a potential opportunity to get rid of old software and hardware, but to do so, you need to tell people the GA of the software available. If you don’t do this, they will just deploy the old code on old hardware and you will have a new problem to get rid of the legacy stuff when the build-out is complete. Take the opportunity to consider doing the change now. This can obviously add complexity to the program but avoids kicking the can down the road.
In developing the Product Roadmap, you need to find out who is going to approve the roadmap. Is it your architecture group? Engineering group? Don’t buy the hardware unless all of your governing bodies will allow the hardware to be installed. Document a task to find out who approves the roadmap for the software and hardware that can be used in the organization.
Unfortunately, not all products are going to fit in the Product Roadmap. Applications may need so much work, that the time required to get the application on the correct version could exceed your tolerance for this project. Therefore, create an exception process to the Technology Roadmap. When an application shows up and says “It will take us 5 years to get our application on the correct roadmap”, issue a technology exception to them so you can get them out of the way. This exception process will need a committee of people to approve the exception. Therefore find the group that will approve the exception. Draw a process flow chart of the process to obtain an exception. This flow chart will show the artifact moving to the committee of people that will approve the exception. Document a task to create a roadmap approval and exception process to the Technology Roadmap. This should include the committee that approves the roadmap and the exceptions to the roadmap.
Upon choosing the product roadmap, you need to assign a project manager that will get all of your new software through your change control process.
Step: Software Purchasing
In purchasing the software, you need to understand how your application will use the software. Will the application need use of the software in two locations? If yes, then you have an “active/active” software license model. If you are only using the software in one location at a time, then you may have an “active/passive” situation. You will need to decide if you will lease the software or need temporary licenses while you transition from one location to another. You will also need to know what software you will not be using at the end of the project.
Software purchasing must include EVERY piece of software running on the server. You need to factor in every backup agent, anti-virus agent, password management agent. Every single piece of software must be known for what you will buy or not use in the future.
Software for the development environment must be considered as well. You need to know your organizational setup for using temporary software or evaluation software. In many organizations, you need to have the security tasks executed on the software that you want to buy before you can buy it.
Document a task to address all of the software questions: Software model as active/active or active/passive? lease vs. buy? temporary licenses or removal of licenses?Executing purchase of the liceneses.
The next steps are related to the governance of the project
If you need to write a business case to get the money for your project you need to find out what the process is for the business case. Determine what document the executives need to sign to approve the business case. Perhaps there is a “resolution” document that allocates the funds. You need to find what that document is. Consider how your legal department may be involved in creating the business case. Do they have a document to create? For this step, Document a task and a duration to find out who is going to review the business case and the project milestone to get the financing approved
Step: Decision Process
You need to record all the decisions in the project. Find the system you will use. TFS has a Decision module that can work, or setup a spreadsheet. The spreadsheet should include: What you said, What you did and how was it accepted. Each of the decisions should be in a system that you can export into a table or other document for quick review. Document a task to setup the decision process and the system with the following fields that Microsoft TFS uses. Key field names are: Title of the decision, Assigned To, State (Open, closed,) Reason, Decision Date, Decision Type, Problem Description, Decision Makers, Decision Makers, Actual Decision, Justification. The traceability of the decisions is a project management principle. Keep this list updated.
Step: Project Management
Your project schedule needs to show, free slack, variances, critical path, what is going well and not going well and dependencies. Document a task to set up your project schedule reports to get the required data.
At one of the first executive steering committee meetings, you need to establish the criteria for project reporting for red, yellow or green status. For example, the schedule criteria could be in yellow status if 15–20% of the tasks are expected to be delayed. The budget could be in red if it is running over budget. This dashboard should be a table with columns to show the status of green, yellow or red against the schedule, scope, quality, budget, resources and benefits. Document a task for all parties to decide upon the dashboard reporting.
For the movement of equipment from one location to another, you need a detailed schedule listed out minute by minute. Use a schedule tool where exact tasks can show the time when they will take place during a move of equipment. The key categories in a detailed schedule will be: Task, the start date and time, end date and time, assigned resource and location where the task will take place. This task is especially required when you are performing a physical migration from one location to another.
Changes to the schedule need to go through a change control process or the steering committee.
Step: Project Charter
Your project charter needs to show the names of the people that can make a change to the project. It needs to document the process on how change requests are accepted or denied on the project, furthermore, you need to know what data is required to accept or deny the change. You need to show who the official voting members are. You need to show the intersecting organizations to the project. This should include all the divisions or departments that you are providing services for. Careful review the disaster recovery, continuity of operations group as an intersecting organization. You need to also include the name of the group that assesses the applications and their disaster recovery timeline. This group will be the one that is adding applications to the portfolio and you need to interface with them. The project charter needs to list the steering committee members and Document a task to set up the project charter. The duration of this task could be at least 2 weeks.
To assemble the budget for the project, you need to know all of the equipment, services and peripheral components you will buy. Peripheral equipment includes cabling, carts, and tools. After you know what you are going to buy, you need to prepare a report to show whether the purchase of the new equipment will be more or less than the existing data center. Your finance people may want to know how the new equipment will compare to the existing setup. If it is more equipment, you may need to defend why you need more equipment in this day and age of cloud adoption. Document a task to provide this data to your finance team.
You should use Microsoft Excel to setup this budget as a workbook with multiple worksheets. Setup a worksheet with the equipment that you are going to buy. Add in a column with the data that links the quotes you used for the budget for your equipment. This will be used when your finance people ask: “Where did you get this number”. Having a link in the spreadsheet to the quote will allow your finance team to see the traceability for the quotes.
You need to have a resource on this project with the sole focus on the budget. Document a task to hire the finance person and document a task to setup the spreadsheet for budget, actuals and forecast worksheets.
The spreadsheet for the program needs to be broken out by category. The spreadsheet also needs to include the services that you will have on this program. The services spreadsheet should show the following setup: Column A, name or role of the resource. Column B -> onwards, each month. You need to put the hours the resource will work, the hourly rate and get a calculation per month. Document a task to set up the budget spreadsheets.
For the human resources, you need to set up the internal charge codes for your employees to charge to for their time sheet. This code will allow you to see how much of the internal costs are working on the program. You can also build accruals for human resources. Document a task to work with your time card and financing people on the charge code for the project.
If you are hiring people specifically for the program, you may need to look at the offsets on the human resources. Offset can occur if you are moving a data center from one location to another. Human resources that were supporting your existing facility in one location, may need to shift to the new location and therefore you have the same amount of people as opposed to adding more people.
The accounting team needs to tell you all of the accounting codes that you will need. If you don’t setup the accounting codes early enough, your purchases may have a delay while codes are created and agreed upon. Document a task to get the accounting codes for the project.
The financing component also needs to include a list of the contracts that will be impacted by the purchase of your new data center. This dollar value needs to be included in the financing.
Step: Software Licensing
You need to know how much new software or what modification to the existing software licenses will be required. Changes to software license agreements can take months to modify. Your software licensing needs to include a review of EVERY environment that you will be impacting. In a data center migration project, you might be purchasing new software, in which case you need to look at the licensing in the development, QA and other non-production environments. You can’t just buy software for the production environment.
To effectively know the software you need to buy, it would be ideal if you knew your logical design in advance. Knowing the logical design of every application could help you know the software needed. In some cases, consolidation of the environment will take place and you will need fewer licenses, but in some cases, you will need more licenses, as you may take this opportunity to change your design and break things out. Document a task to purchase the software needed for the project.
Step: Bill of Materials
You are going to buy a lot of stuff for your data center migration. You need to know the process for purchasing in your organization. In some cases, organizations are still heavily dependent upon spreadsheets and quotes may need to be transcribed to a spreadsheet. Document a task to know the steps in your procurement process and confirm that a system is available for tracking the purchasing. This system should record the following data: When the request was submitted for purchasing, what was on the request, who approved the request, when the request was sent to the vendor, when the vendor asked and what they asked for questions, when the equipment arrived and when it was installed in the data center. The process should also document who will approve the final bill of materials. Document a task to identify what data the approval team needs and schedule the multiple meeting with the team for reviews of the bill of material data. The build-out of a data center requires more than just a spreadsheet. Consider adding your own procurement tool like Procurify: https://www.procurify.com/.
Your bill of materials needs to include the consulting resources that will be used on the program. When you are working with your procurement team, ask them what security paperwork is needed in order for a consulting resource to work in the organization. Your security organization is going to have paperwork to fill out and need signatures from a variety of groups. Furthermore, set up the consulting agreements to show that their period of performance will not start until the resource has full connectivity to all required applications and access. Document a task to fill out the security paperwork for human resources working on the project.
You need to submit the request to purchase the software for your project. You also need to tell your procurement people what software you already own. Document a task to submit the software that you will purchase.
Remember you are going to have multiple people that enter purchasing information into your system. You need to figure out who enters, who approves and who works to buy the materials. Document a task to setup who your purchasing people are and what steps they will need to use
Once you know the procurement process, you need your team members to obtain quotes from the vendors to buy stuff from. Then you need to transcribe all of the quotes into a single system or a spreadsheet where you can easily report on the data. If you don’t have a single system for purchasing, flag this as an immediate risk and add MONTHS to the level of effort to manage all of your purchases. Identify what system or location that the team members will save the quote. Document a task to your team members to obtain the quote for all of the equipment they want to buy and the other procurement related tasks.
In some organizations, the purchasing process includes the development of Request for Proposals. To speed up the purchasing, consider developing a process where you can issue a justification for non-competitive purchase. Your organization may have a form to fill out so you can speed up the procurement.
When you issue a request for purchasing, you should include a request to the vendor to provide the Visio Shape, or other AutoCad type software product that you will use to build your Data Center Floor plan. If you don’t do this, then you will need to find the shape for other milestones in the project. This shape should show the front and back of the equipment.
Each change to the Bill of Material amount needs to document why it changed. At some point you are going to have a “final number”, or at least you think you do. Then that final number will change. You will need to tell somebody what the specific change is to the budget.
You need to have a team of people that will collect, process and organize all of the bill of materials. The project manager can’t manage and submit all of the bill of materials data. Document a task to hire a team of people that will just focus on the organization and submission for the bill of materials. Ideally hire 2 people per area that will organize the ordering of the equipment and will deal with all of the questions about the equipment going forward. Inevitably technical people will ask if a piece of equipment got ordered and where it is. Your bill of materials team will handle all of the questions.
Once you know the vendors you will purchase from, meet with your legal team to tell them. Ask the legal team if they will be using this project as an opportunity to review or re-negotiate various end-user license agreements and terms/conditions on the contracts. If this is a new hardware or software product for the organization, factor in the time for the new end user license agreement along with the terms and conditions that will need negotiation. Document a task to know the duration of time that legal will need for end-user license agreement review along with terms and conditions review.
You need to document a task that your bill of materials team will verify that all equipment is installed in the rack and it is available for services and does not require any replacement parts. The bill of materials team should map each hostname in the rack elevation back to the equipment ordered on the bill of materials.
Step: Your compute, storage, security and firewall team needs to select their equipment with the known ports they want and speeds they require. Once that is complete the network team can design a network. Furthermore, once the equipment is selected, this gives the requirements for the power and rack elevations required. From here, you can develop the cabling required.
Step: Bill of Materials Checklist
On the equipment you order, the following needs to be confirmed:
1. The rails that you order for the equipment will work with the cabinets that you order. Make sure the rail kits work with the rack you ordered.
2. Auto-sensing power supplies may be required. Confirm with your data center facility manager if that is needed.
3. The type of power cord needs to be confirmed for what is needed with the power strip you ordered. Equipment ships with a default cord, you will need to make sure that this default cord works with the power strip you ordered. You need to make sure the length of the cord is appropriate for the data center. Document the type of power cords your data center will support. You will also want to review whether you want to color code A and B power. I don’t recommend it. Your equipment should have dual power supplies, so it does not matter on the color of the cables.
4. The power strip that you order needs to be selected in advance in order to tell the compute, storage, network, security and firewall team can order the right power cords.
5. SFP Modules need to be ordered with the equipment and they need to order the right type. Devices and circuits need single mode or multi-mode SFP modules. Make sure you get the right ones. The type of SFP module needs to be coordinated with your cabling vendor. Once the cabling vendor has designed the cabling infrastructure, you may need to obtain the type of patch panels that have been designed and that will determine the SFP modules to order.
6. Review the air-intake for the equipment. Equipment may need different air intake components.
7. Your team needs to know the fan kit to order for the equipment. The equipment generates heat in all directions and extra fan kits are needed to blow air different ways. Document a task to verify the fan kits.
8. The right amount of GBIC modules should be ordered for each piece of equipment. You should not let the team members borrow GBIC modules from each other. Team members should not make the assumption that an HP GBIC will work in a Cisco product. Document a task that every order has the parts from the specific manufacturer
9. The request to the vendor should include an ask that they provide the name of their individual that will need a badge to support the data center.
10. Visio drawing of the hardware to be delivered with detail identification of NICs, HBAs and other physical connections (not a general Visio on their web site those have been different)
11. Provide detail on the hardware environment: power amps, watts, HVAC, power connection types, weight, size, footprint, etc.
12. Provide proper power cords, rack mounting hardware to match each datacenter infrastructure
13. Identify whether the vendor submitting the purchase can combine products into one quote. For example, do you want to see quotes with DellEMC, NetApp and Hitachi on one quote?
Step: Build Development Environment
If you are using this project to make changes to applications, you need to build out your development environment to support the production environment. You will have a considerable duration to fix the development environment to match the production environment and the new software required. Remember, your network security model may not allow all developers access to all of the networks, therefore make sure you consider the connectivity for the developers in this design. Document a task to review your development environment and whether it is an EXACT match of production. If it is not an exact match, let your developers know. Document a task that all of the testing you do in the development environment will not matter since the production environment is different and you will discover new issues.
The networks in the development environment need to be reviewed. Examine the firewall rules that are needed in the development environment as compared to what you need in production. In many organizations, the firewall rules are not enforced in the development environment in order to make the development environment faster. This translates into a whole bunch of firewall rules that you did not know about when you deploy the application in production. Document a task to review the development environment networks as compared to the production environment. If the networks are not the same, plan for the additional firewall requests or plan to fix the networks. This fix should also include connectivity to your VPN partners and the Internet.
In my past experience, my database team built databases in the development environment without an issue. When they moved it to production, the firewall access lists were totally different. They had to submit a multitude of firewall requests for production to get the database to work. This added considerable duration to their deployment.
You need a list of every software, hardware and application component running in the location that you are going to move.
For this new data center build out, you will want to consider what applications you want to add to your environment. You may have applications that are in your primary data center that you meant to install in your secondary data center, but it never made it. Document a task to make a list of all the new applications you will install. You will need to factor this into all of your software and hardware and consulting service purchasing.
In your initial discovery of your environment or a customer environment, if they tell you to order the “Equivalent” of what is currently in place, then you should have them tell you how many servers they currently have. This includes physical and virtual servers. If you don’t have the existing inventory, then setup a long duration task to build this inventory at the physical and virtual level.
I once had a customer tell me to buy what is “Functionally equivalent” to production. That would have been easy if the customer had known the server count in their existing environment.
Setup a template ahead of time for all the data you want to collect about the application
- Make sure it includes everything about the application dependencies. What application depends on what is important.
- Group the applications that should move together into various wave’s
- You need to do a full logical discovery, even if the client tells you that you are not to engineer anything. Therefore you need to collect all components about your systems. The inventory of the applications is paramount for success.
- You need to know how many people use the application
- You need to know how much bandwidth is needed for each application
- Building an application template helps all individuals understand what will get installed. The application templates are extremely important to know the future installation schedule.
- The template should include a diagram of the application with the server names and port information.
- In building the schedule for the template, the following milestone dates should be gathered from the application teams:
- AppName — See the section about acronyms
Relocation Lead — See the section about needing a project manager for each application
Rough Draft — When the document will be ready
Peer Review — Setup a team to review the documents
Peer Review Comments — Setup the date for the return of comments to the application team
Optional Peer Review — Consider adding another review
Rough Draft Customer — Submit the draft to the project stakeholders
Customer Comments — Obtain the stakeholder comments
Final Draft — Submit the document
- To collect the inventory, you will need to utilize the technical inventory components from the vendor. For example: EMC Grabs
- The documents provided by the cutomers needs to be reviewed with the relocation team.
- The relocation plan needs to include how the data will migrate from the source to the destination.
Step: Floor Plan
You need to develop the floor plan for your new data center. To perform this, you need to know what equipment you are going to buy. You need to be working with a firm that knows the equipment and the hot and cold aisles that will be built for the data center. The floor plan will need to identify the key rows that are for distributed communication and main data facility (I.E. where cabling will terminate)
The floor plan design requires a firm that specializes in data center design. They are going to have to make a decision on the placement of equipment by the environment or the system type. They may ask you whether you want to have a data center setup that the rows are setup by windows, linux, storage or maybe it is setup by production, development or QA environment.
The floor plan will use the bill of materials, therefore the bill of materials needs to identify the products that come with multiple free standing racks. In several purchases, there is one line item for a rack, but it ships with multiple racks. This often occurs with storage purchases from DellEMC or Hitachi. You need to know all of the racks and equipment that maps to the bill of materials.
Step: Existing consulting arrangements
Review all of the existing consulting agreements in place. Determine if they are important to the build-out of the environment. If they are, make sure that the contract will not expire during the program. If the expiration date is approaching, extend the arrangement immediately or recompete it before the program begins. Do not start this program with any contracts that could expire while the program is going on. Document a task to review the current contracts, make a list of their expiration date and what you need to do with them.
Step: Equipment configuration
Determine where you are going to ship the equipment you ordered for configuration. If you are going to ship it to your current office and configure it and then ship it to your data center, then you need to make sure that your current location can receive the equipment, can store the packaging materials and then can repackage and ship the equipment. You need to make sure you have a room big enough to store and configure the equipment. You need to make sure your shipping and receiving area can process all of the equipment. Document a task to set up the shipping and receiving for equipment into your office and out of your office.
Step: ServiceNow Integration
If you are using ServiceNow, these servers need to be built first. These servers perform the discovery in your environment. Add these servers as the first builds in your new data center so they can discover things that are added. This includes building them in your test environment if needed.
Step: Testing Environment
If you are changing the applications you may need to build a new test environment. If you choose to do this, then you need to hire a team of people with a single job to manage this test environment. Their tasks will include: How to build the environment to match production, including how data will get into the test environment. How to install the application in the test environment. How to capture the installation specifics in order to move the application from the test environment to production. How to decommission the servers from the test environment when they are done. What the naming convention is for the servers in the test environment. How to report the test server builds to ServiceNow. This test environment will need to consider connectivity to the Internet and DMZ related activities. Furthermore, if you have a Mainframe, determine if your test environment can reach the mainframe environment. In summary, if you don’t have this test environment, you are going to create an entirely new setup and you will need all of the processes that go along with it.
If you have an existing development environment, you need to determine if the existing development environment will merge together with the new environment you build. Your project needs to document a task to decide if you can use your existing testing environment or if you need to set up the milestone for building the new test environment.
A task is required to obtain the test plans for the applications. If the applications do not have a test plan, document a task to build the test plan and success criteria from the test plan.
Step: Shipping Equipment
For equipment that will ship to your new location, you need to know who will receive the goods at the new location. That person needs to know the process of receiving the goods. If you are a ServiceNow customer you are likely using the asset management module and the system will need to be updated. You need to train this person on the steps to receive the equipment, especially if you use ServiceNow. Coordination with your asset management team is needed. You should publish the agreed upon contact and shipping address for all parties to have easy access to. Consider adding the site into ServiceNow upon signing the contract. Document a task to train the people at the new location on how to receive equipment that is arriving for the project.
You should coordinate with the team that signed the contract and the data center facility to determine if a certificate of insurance or certificate of liability is required.
Step: Communication Plan
You need to develop a word document with all of the presentations and schedule of the presentations that will be required. Work with the team members on the project charter and asks them what presentations will be required daily, weekly, monthly, quarterly or yearly on the program. The communication plan should be a table with the following columns: Type of communication, purpose, sender, target audience, Key Messages, Delivery method, target date/frequency and the actual date it was delivered.
The communication plan should also indicate what awards you want the program to apply for. Use your communication plan and document the list of awards that you will apply for. Your company may have an annual awards ceremony from the CEO that you may want to apply for.
You need to set this table up when you begin the program. Then you need to set dates on the calendar for the following milestones: When you will create the slide deck, when you will send the slide deck to the “most vocal” team members for them to review it in advance and provide comments and when you will send it to the entire group, which should be at least 1 business day in advance.
Step: Agile Deployment
During the program, you should setup the deployment of the equipment to take place in waves. To do this, you need to know what cabling will support the initial equipment to deploy. The tasks needed to support this initial equipment deployment includes:
1. Identify the equipment, the ports and power that will be needed
2. Determine if the cabling to support that equipment is in the rack or will go between racks
3. Provide an initial spreadsheet with the list of devices and the cables
4. Confirm that the cables you need have been ordered.
Step: Providing Access To The Data Center With A Badge
Develop the process to get your team approval for a badge at the data center. Once the contract is signed, get the team members processed for badge access. Document a task to modify your existing process to badge team members at the new data center or document a task to create the process to badge team members at the data center.
Adding the data center to your environment means getting new badges for your team members. Identify who will be the person that approves the access. Consider setting up the approval process in ServiceNow. Identify the process for visitors to go to the data center and obtain escort access. The visitor process needs to identify multiple people that can submit requests to the data center 24 x 7. The documentation of the process needs to include whether the facility has a system for entering the request for visitor access. Document a task to build the process for approving badge access to the facility and escort access.
The badging process should include meeting with the vendors and determining what vendor should have a badge. You will want your vendors to have a badge so they can come in and service the equipment. If the vendors don’t have a badge then you need to send a request to the team that approves access. In the bill of materials order, ask the vendor to list the team members that will need to be badged to support the equipment.
Step: Tools To Order
Your new data center needs tools and carts. Create a list of tools that will be needed at the data center. Key items will include:
A high-end Genie Lift to lift servers into position. Make sure you have the proper plates that fit on the Genie Lift, as they are sold separately
Walkie-talkies: Get the model that will use a repeater. You need these when you are doing a migration of a lot of equipment in and out of the data center from the loading dock to your space. Make sure you test the walkie-talkies inside the facility you are using. Sometimes the walls are so thick you can’t get a signal.
The list of tools goes on.
Document a task to buy the tools for the data center.
Step: Applies to all activities
You need to document all assumptions for the project. This is related to the regular project items and finances.
Step: Change Control Process
Determine the team members that issue the most amount of questions at your change control board meeting. Schedule meetings with them in advance prior to a change control presentation to get their approval.
The change control process has to include a review on the bill of materials. Given the amount of equipment that will be purchased and the changes, these changes need to be approved as they impact other areas like cabling, cooling, rack space, etc. Consider using ServiceNow to approve the changes to the Bill of Materials.
If your new data center is an airplane ride away from your office, you need to staff the facility with a team of people that deal with receiving equipment, shipping equipment, installing equipment and application support. This new site needs a manager where the people at that site report to the manager.
Step:Adding wireless to your data center
During the buildout of the data center, you will have a parade of vendors and employees working there. Consider adding robust wireless to your cage information. Do not depend on the data center vendor wireless. Document a task to install wireless into the data center location.
Step:Hand Off The Data Center To Operations
You need to determine who operates the data center once it is built out. Usually, your engineering group works on the buildout and then your operations team manages it. You need to document a task to get the acceptance criteria from operations. What paperwork do they need to call the data center “live” and in production? What standard operating procedures need to be updated before it is in production.
The project needs to select dates on a calendar when you will conduct a forum to give an update or other widespread information about the program. This forum could include a demonstration on software that is new in the environment. The forum tasks will include:
1. Reserving the space months in advance for the presentation. The reservation should also include reserving the room for 3 time slots prior to the actual meeting in order to practice the presentation
2. You need to verify the network connectivity in the room for the presentation
3. Do not schedule the forum for a Monday. You need practice time the day before
4. Schedule the forum for the afternoon so you can setup and practice in the morning
5. Make sure your presenters do not have vacation time scheduled for the practice sessions before the forum
6. Determine how people can view the forum remotely. Will you use WebEX or some other video distribution platform?
7. If your internal communication/video team has a requirement to load your slide deck in a system, determine the date when they need the slide deck in order for them to load it
8. Consider ordering food/refreshments for the forum
9. Create the calendar invitation for the forum and send this 4 weeks in advanced and periodic reminders
You need to set up a task to find all of the documents that you will need to update during the project. If you are building a new Disaster Recovery site, then you need to update all of your disaster recovery documentation. If you are moving to a new location then you need to update all of your security items including the document on how to get access to the data center. For example, if you have a badging process you will need to gain access to the data center. The teams that require the documentation should tell you what format is needed and what sections need updating. The teams should tell you what they want to see for the application diagram in the documentation. Document a task to hire a team of people that will handle updating the documentation.
Step: Initial Infrastructure
Organizations have different names for each application. The foundational components could be referred to as Tier0. This might include basic infrastructure. This could include directory services, internet access, the virtualization software (VMware of course!), firewall and storage components. Once these components are installed you need to be able to tell people what services you can support for what you have installed initially. Ideally, the equipment will map to the applications that can now be supported. For example, if you have your Internet installed, compute and storage. You could tell people that you can support applications that need to access the internet. If you installed, your Internet, firewall devices compute, storage and connectivity to the Internet, could you tell people you can support VPN services? In a perfect world, you can map all of the hardware needed to support each application. In this mapping, you will know what hardware is needed to support each application.
The build out of a data center requires a number of network resources. Make sure your existing network team can spend time to build out this environment. Don’t think you can hire a new team to configure the equipment and do it for you. The existing people that know the environment need to configure the networking equipment.
The people chosen for this project need to have a positive attitude. Moving data centers is a challenging activity and therefore you need a team that works well together and is positive. Lose the negative team members.
Database resources need to be present. You will have a lot of databases to install and you need to coordinate with your existing database resources to perform all of the installations.
The resources need to know in advance what the working hours are during the holidays or other times when the company is officially closed. If you are a government agency, identify the schedule that your team can work when the office is closed for snow or other reasons. Consider adding all the resources to a mission essential personnel list so they can work whenever they want even if the office is closed. Your human resource schedule should establish that everyone is working when the stock market is open.
The resources MUST have the disaster recovery team as a key part of the program. The disaster recovery team is going to be impacted by the new buildout, therefore they need to be providing tasks and asking questions early. If they are brought in at the end and you hand them a data center, they will have a lot of clarification questions as they were not involved early enough.
Step: Configuring Equipment
When you kick off the program, you need to obtain the procedure in order for team members to obtain a static ip address for the computer they use to configure the equipment. Team members will often need a static IP address to configure the equipment. This may require multiple security forms to be completed. You need to shift the request for the IP Address request to the team member. You don’t want to be the middle man requesting the IP address for your team members.
Step: Receiving Equipment
When the equipment arrives you need a way to notify each team that the equipment has arrived. During this program, you will have a lot of equipment arriving and your technical teams need to know that it has arrived. You don’t want equipment sitting in the receiving area without team members being aware that it has arrived. Document a task for your receiving area to notify your team members that the equipment has arrived.
You need to document a task to determine how applications are deployed to production. You may have a team that takes care of interfacing between the business and your technology group. Confirm with the interfacing team that the existing process will be the same to deploy an application to the new data center. Confirm that the existing team finds the resources to do the deployments. This task is necessary in order to ensure that the interfacing team does not use this project as the time to change their role and shift the deployment coordination and resources coordination to your data center project. This confirmation should also confirm that the existing team will be the liaison to perform the testing with the business.
Step: Performance Testing
Document a task to identify the group and the tools that will be used for performance testing. During this program, performance issues will show up. You need to find a group that will handle the issue. If you don’t do this, you will be the person herding the teams together to resolve the issue. The group that does the performance testing should know the tools that they will use. Document a task that the performance group knows their tools and if they don’t then have them submit a budget request to buy the tools. If they are going to buy the tools, make sure they see the acquisition items in the step titled: Product Roadmaps.
Performance testing needs to be at the front of the project. You can’t wait until the end of the project to start the performance testing.
Performance testing tools could include Wireshark and Dynatrace. Document a task to get these tools installed in advance. You don’t want to have a performance problem show up and then start figuring out how to install the tools.
The resources required to support the performance testing will need to be available 24 X 7. Once the performance issue appears, team members will need to work tirelessly to resolve it.
Step: Production Installation
Transitioning the application from the existing environment to the new environment which may include new hardware and software versions could introduce performance issues. Applications are optimized for the hardware and software that they are using. Now if you move them, you may need to adjust the settings. For example, in the case of an Oracle Database, you may have optimized the application for using Oracle on version 11.4, but now your application is using Oracle 12G. The tuning you may have done for your application on Oracle 11.4, could be different for Oracle 12G. Document a task that your application teams need to add the performance testing in the schedule and account for the duration to resolve the issues. If you don’t do this, the application team will call for an escalation because they need to meet the release date. This escalation means you will need to cover the issue 24 X 7 until resolved.
The installation of software needs to occur 24 X 7 X 365. This means you need to setup a team that can “Follow the sun” and perform installs all night long. Organizations often have a team of release engineers that only work during core business hours. If this is a case, factor this into the software deployment duration or document a task to build a team that works 24 X 7 X 365 in order to deploy the software.
The deployment of the applications should have a single product manager that has setup the schedule for the deployment and cut over to the new application. Do not cut your human resources plan and think you can assign one project manager to multiple applications. You need a 1:1 match. You need one project manager/product manager to work the installation of the application from beginning to end.
You need to create a task to build the following presentations:
- For the team that will arrive and install the equipment, they will need a presentation and a meeting that covers the logistics of the location. Key components in the slide deck will be: Who is going to the facility, what are the phone numbers of those people that are going, what they are going to do there. Document a task to build a slide deck with all of the logistics for the people that are going
Step: Operations Handoff
Document a task to meet with your operations team to determine what documents and meetings they need to take over the operations of the new data center. Your task may need to list all of the documents that the operations team wants to take over running the data center and all of the new pieces of equipment.
You should meet with your Operations team, which includes compute, storage, network, firewall and security to confirm what documents are needed for them to operate the data center. Document the list of documents they need. After you obtain sign off, add more duration to the project if they change the list by adding more documents.
Document a task that every new piece of equipment and software will require updated operations guides and training to the employees. When this task completes, the operations team or other designated individual should know how to run and operate the equipment including upgrading it. If you fail to do this, then your team that ordered the equipment may be the ones that end up operating it. If you have the traditional Engineering hand off to Operations model, then your Engineering team just became the Operations team.
Step: Network Buildout
Based on the amount of network equipment, they will need at least 5 full days to build out the network. In order for them to build the network, all of their pre-requisites need to be met. Document a task to determine all of the pre-requisites for them to build the network. Document a task for them to provide a written checklist of what task they need to perform in order to deploy each piece of equipment.
The checklist for the network buildout should include:
- Install the device to perform TACACS authentication
- Add all of the devices to the monitoring system
- Review the routes on all devices and make sure the routes are carried from the old system to the new system. VPN devices will often have a route on them for an internal application, you need to make sure that route is carried over!
Step: Firewall Build Out
The firewall team needs to provide a list of tasks they need to do in order to deploy the firewall devices. The tasks may include the following:
- Allow the firewall devices to send their logs to Splunk
Step: Firewall Rules
Applications and infrastructure equipment need firewall rules. In a data center build out, you will need to submit requests for the firewall rules needed in order for the applications and infrastructure to talk to each other. If your development environment is not an exact replica of your production environment then your teams will discover the required firewall rules when they deploy their application in production. This means your firewall team will need to scramble to implement the firewall rule. This puts undue pressure on them to complete this task. Document a task that your application and infrastructure teams need to know the source and destination for the firewall request. Document a task for the date when they will submit the firewall request. Document a risk that you will need more firewall changes once the application is in production. Document in the program charter that a missed firewall rule will extend the duration of the program.
Firewall requests are often implemented by different groups in your organization. Document a task to know what the process is for submitting the firewall rules. If your organization does not have a process for performing firewall implementations, then document this as a new task and add duration to your program to help them setup their process. The process needs to determine what group installs the firewall requests on the internal network between the web, app and database layer as compared to the external network to the Internet.
In setting up your environment, you can either spend time to delay the actual project and build and exact replica of production with the right firewall rules that match production or you can setup a system to log the data in the development environment and extract the data from the log in order to create the production request.
Having completed documenting the process for the firewall requests, document a task to create a firewall marketing document and send to the application teams. The data in the marketing document should include:
- How to use ServiceNow for submitting the requests
- The duration of time that it takes for the firewall team to review the request
- The dates when the rule gets implemented after the request is approved
- A review of the security practices associated with firewall requests
- A review of the escalation process that the team will need to use in order to get the firewall change implemented immediately.
Step: Getting Rid Of Equipment/Finally Getting Out Of Your Old Data Center
If you are purchasing new equipment, then you will need to get rid of the old equipment. This is commonly referred to as decommissioning. You need to consider the following tasks when you get rid of the equipment:
- Count all of the disk drives that you will need to get rid of. Document a task to count all of the disk drives.
- Remove all of the disk drives and shred them. If you don’t have a contract with a mobile shredding company, then you need to create a task to create a contract for a mobile shredding service. The count of the disk drives will be important for the shredding contract
- You need a spreadsheet with all of the physical equipment in the data center that you are getting rid of. The spreadsheet should have the following columns: Serial Number, Purchase Cost, Residual Value, HostName, Application that uses the device, What you want to do with this device: (Dump it? Use it somewhere else?), warranty support, end of life from the vendor. Once the spreadsheet is built, you need to get it approved by all of the equipment owners. Their agreement is to confirm the equipment that will come back is useful and they agree that it can be used.
- You need to have a process to transfer the ownership of the equipment from one group to another. You could have a piece of paper that you put on each device and the current owner signs the piece of paper and the new owner then signs that they received the new device. You need to train your team members that will remove equipment on the paperwork required to fully dispose of a piece of equipment. This procedure must also include how to get rid of a hard disk drive (HDD).
- The perfect schedule to decommission equipment would include removing all servers from the environment first. Once the servers are done, then you can remove the storage. Follow on after that will be the firewall devices. Then you can remove the network devices and circuits.
- Early in your project you need to terminate the circuits that are used from the site that you will jettison. Consider disabling the circuit interfaces on the device for a few weeks before you fully remove the circuit and device.
- In the data center you will move out of, check the lighting and cooling for comfort. Since you are moving out, you may not have visited the data center in a while. Therefore check that you have the maximum amount of light or consider purchasing some drop lights for a temporary measure.
- The decommissioning process will involve tools and flashlights. Consider purchasing coal miner head lamps so people can see what is going on in the rack as they are removing the devices. The removal of equipment also puts a lot of people in your data center. Make sure you have enough chairs tables and network locations for people to sit. During the process they will be updating a lot of data.
- The removal of equipment from the data center requires that you know every device in the data center. For the virtual servers, you should know every host that runs on the virtual server. This host should be mapped back to the application. Doing this mapping allows you to know when the application is fully shut off.
- For the removal of equipment, engage your network load balancing team to run a report to determine what servers are still using the load balancer. This gives a good indication of what devices are still at the data center
- You need dedicated team members that can unrack the devices and track them from coming out of the rack through the surplus or reclaim process. Hire 2 people that perform the following work: Unrack the equipment along with the rails. Tape the rails and label them so people know what device they go with and place it in a dedicated area so people know if the device will be reclaimed or if it will be sold off. Use a hand scanner to scan every item in the area to build a spreadsheet list for the moving company to come and pick up the equipment. Once the pickup is done, they will update your master list to show that the equipment has moved locations. Only these 2 people should be moving and unracking things in the data center. This keeps the piles organized for moving the equipment from the “Sell off” or reclaim designation.
- Make sure all equipment has a bar code and the bar code matches to the serial number. If you do not have this, you need to go back and inventory your data center and set this up.
- Provide a hand scanner that can scan the bar codes of every device. If you are in a rush, evaluate the bar code scanner that can be used on a mobile phone.
- For your larger vendor appliances that come in a dedicated cabinet, like DellEMC or NetApp, order the vendor-supplied packing materials weeks in advance and pay the vendor to pack the device for you. Remember, ordering packing material does not mean the vendor will pack it for you.
- Using ServiceNow or your equivalent system, you need to set up the steps that a device goes through in ServiceNow so people can see the process for the removal from the environment.
- Team members need training on how to remove equipment from the environment. This should include the ServiceNow or equivalent system, how to clear the configuration from the device and what paperwork is needed to remove the device
- Understanding the dependency for each device allows you to create the schedule to determine what to unplug first. For example, your firewall team depends on the network, therefore you can’t unplug the network until the firewall team is done.
- You need to hire a moving company to move your stuff from one location to another. Start this procurement early on. You may want to consider putting them on a schedule to pick up devices over several months. This allows you to take the devices that you have unracked off of the floor and make more room. This also allows you to “practice” the removal of equipment with fewer devices. Then if you need to do a large move, you have the kinks worked out ahead of time. Remember to follow the process for getting the moving company access to the data center and the loading dock.
- At the beginning of the project, send your teams into the existing data center to identify what they are not using. Start the process to remove this stuff now. You should not assume that all equipment is in use in the data center.
- You need a full inventory of your existing data center and the mapping to each application. If you don’t have the mapping then you don’t know when it is ok to shut something down or not. If you know the mapping, you can build a list to show the installation of the service in another location and know that the service is running somewhere else
- You need to know how to remove the racks and cabling inside of the racks. If you do not want the racks, talk with your vendor to determine if they will want the racks. Also, plan to remove all of the cabling from your data center and hire a team that deals with that.
- Identify the process to alert your contracting folks when you are done with the data center. Do they want an email or is there a system to update to terminate the contract with the data center vendor.
Step: Moving Equipment From One Data Center To Another
If you are physically moving equipment from one data center to another, you need to know how each application will work if the equipment is lost in transit. Document a task that every application needs to identify how their application will work if their physical piece of equipment is lost or damaged in transit. If you have one server, that runs multiple applications, you need a plan for each application to recover if you lose the server. Consider this especially if you have a Mainframe.
Checklist of items to deal with on the move from one facility to another.
- Start early in the project to conduct a detailed inventory and build the spreadsheet of equipment. The teams should identify any equipment they can get rid of now and not wait until late in the process. You are not using every piece of equipment in the current data center. You can get rid of stuff now.
- Document the task to identify who powers off the equipment.
- What is the required floor protection at the source and destination facility?
- What is the packing materials that the vendor needs to bring to ship the equipment? Do you need an elevator plate and masonite to cover the floor from the loading dock to your location?
- If you are using the same team of people to pack up the existing data center and then they need to be at the destination facility, charter a jet and limo service for them. They are working so hard, take all of the logistical worries out of the schedule. This also allows you to make sure that their tasks are finished and do not get cut short in a race to the airport.
- Your move in team needs to arrive before the equipment
- You need to let people know that the arrival of the equipment may occur before the team is on site to set up the equipment. If this is the case the equipment will sit in a truck until the team arrives. Don’t try to adjust the schedule on the fly for early arrival of equipment
- Distribute the detailed schedule of the exact steps and when they will occur for each minute of the move weekend
- Your cabling vendor needs to be on site when equipment is installed and connected to the cabling. This will allow them to make changes at the time of installation and avoid re-work later.
- Setup for the building electrician to be on site to plug in the power cords and monitor the load as the power come up
- If you are moving into a location that is in a cold climate, review the outdoor tents and heaters that will be required to make sure the doors open or snow can be removed.
- Setup the catering for the move events. Breakfast, lunch, dinner and late meals will be required.
Step: Moving Into The New Data Center
When equipment arrives at your new location as a new delivery or an entire truckload, you need to reserve the elevator and the required security team members to assist with the movement of the equipment from the truck to your location. Document a task to reserve the elevator and loading dock for deliveries.
You need to know the existing inventory at the data center and what the equipment is used for. You need a tool that can map the servers to the application at the virtual and physical level. You should consider a mapping of the servers to the switches, but for some this is a waste of time. The inventory provides the ability to know what server you can turn off when the new one is available at your new facility.
In order to perform the inventory, the existing application teams need to provide the data. Document a task to develop a list of questions that are needed from the applications and the people that can answer the questions. The list should include a review with the storage, compute, networking, firewall and security teams. The list should include all of the people that you want to meet with. The customer will want to know the schedule of when they need to provide a piece of data for the inventory. This meeting will need to be sent at least three weeks in advance so they can collect the data and then the meeting reviews the data. The meeting should document the list of issues or things that are not found during the meeting. The consolidated list of things that are unknown needs to be in a central location for all team members to see what is missing on a per-application basis. <NOTE TO JJD INSERT PREVIOUS FIELDS USED IN DISCOVERY PROCESS>
Your inventory process should show the old hostname and what will become the new hostname. This mapping helps to understand that all of the applications are covered for installation.
Step: Application Testing
Once the new application is deployed you need to execute the test cases to prove that it works as designed. The tests could include the ability for the application to support an individual components failure or the ability to support workload from a different geographical location. This testing requires a coordinator that will bring all the parties to the call and organize the test. The test should document in ServiceNow the steps that can be executed to perform the test. You should hire enough people that can do the coordination for each application and enough coordinators were you can execute the tests for each application in parallel. If you just have one coordinator then they might not have the time to do a test for all applications at the same time. Document a task to hire enough team members that can run the testing for each application.
Step: Human Resources
The migration to a new data center should have the following resources:
- A project manager for each application that needs to be deployed
- A project scheduler that has the dedicated role to update the project schedules and build reports
- A finance team member that focuses on the budget, forecast and actual results
- A steering committee for governing the project consisting of executives
- A small group of technical people that make the technical decisions for the program
- An architect that can establish the technical setup for the applications
- A risk manager that can provide an independent assessment of the risks
- A team of people that will handle the assembly for the bill of materials, paperwork and follow up for the arrival of equipment
- A cabling vendor that sits on the steering committee
- A security team member that can provide the rules for security implementation.
- A team member that can draw the logical design of the updated application. This is an architecture level person
- All resources should have the correct clearance and equipment from the customer before the “clock starts ticking” for your deliverables. Do not start a customer engagement until you have all of the equipment and remote access for your team members. Furthermore, do not start the “clock ticking” until you have completed all of the required customer security training. You don’t want to pay your people to work on training items un-related to the customer engagement.
- The existing team that knows the applications and infrastructure needs to be allocated to the program. Do not go out and hire a team that you think will come in and take over from your existing team to perform the installation. The existing team needs to do the installation and data migration.
Step: Program Management
The start of the program should create all of the standard project management plan documents. This will include:
A Program Management Plan with appropriate sub plans (Requirements Management, Change Management, Communications Management, QA Management, etc.). Requirements management is one of the most important. You need a plan on collecting the requirements and provide the traceability as to the persons that provided the requirement. The program management plan needs a task to document the receipt of customer documents. The system for receiving the document should include: When the documents are received, the data and the contents. Consider using SharePoint or another system that can provide audit level data for who sent the document and the date sent. This system needs to also track the deliverables that are required for the program. Deliverables could include the sign off on the milestones for the program.
The program needs to set up the acceptance criteria for each application that is installed. This could include a test from the business customer and then they send an email that they accept their new environment. The acceptance criteria needs to be established with the early program management activities. Document a task to devise the acceptance criteria in the program.
Step: Tools For the Relocation
You need a tool that can track the relocation of the applications. This tool can map the physical and virtual server to an application. If you can’t afford the tool, then you will need to use a spreadsheet and that is not possible. You need a tool that can track all of the hardware in one system as it maps to the application.
You need a tool that can draw your rack elevations, i.e www.netzoom.com
When using spreadsheets, set a rule that you can’t use merge cells. Merge cells will not allow for quick filtering of data. Document a task to identify the tools that will be used to manage the data
You need a tool to manage the IP Address assignment.
You need a tool to share all of the documents. Microsoft One Drive or SharePoint are good solutions. Consider other tools like smartsheet.com, Monday.com, Trello.com, etc.
You need a tool for change management and issue tracking. Consider using Microsoft Team Foundation Server and/or Service Now.
Document a task to setup the definitions for each acronym in the program. You should list each application, what it does and any other names that are used to describe the application. In many cases, team members call an application different names, but they are the same application.
Step: Proposal Submission
In order to submit your proposal to the customer, the following tasks will be required in order to collaborate with all of the parties to submit your response.
A. Have teams take two passes at their section of the document
B. For the third pass, schedule each team in a conference room to go through their section in real time
C. Put out the schedule in advance so people know when they will be coming in for their changes
D. Hire Tech writers
E. Never agree to an RPO or RTO without a review of the architecture to see if the customer meets the RPO and RTO now
F. Never agree to install a client application
G. Never agree to execute their change control process or any customer defined change management for a process
H. Never agree to shipping anything for them. You are not a shipping company
I. If the customer presents you with a schedule and says the dates on the schedule must be met, you need to analyze that schedule carefully to see if the dates they identify can be successful. If your customer has not moved/built a new data center, then they may not know the duration of the tasks.
J. Review the location of the customer applications. If they are not on a shared Storage Area Network, consider the current installation and the time it will take to convert them to an updated architecture.
K. Provide flexibility in your response. At bid time, you know the least amount about this project and it has a great deal of risk. When you finally win the award you will begin to know more about it and you will need flexibility to change your contract to accommodate thee new things that you find.
L. Setup the payment schedule that you can get paid for milestones in the contract based on servers installed. Make sure your payment schedule is not tied to the deployment of every server prior to getting paid. You want to get paid based on incremental servers that are deployed.
Step: Drawing the new layout
Prior to the deployment of the application, the updated logical design needs to be produced. This design needs to show the new server names, ports and the flow of data from source to destination. The diagram should show what security layer the application is located in. Don’t let anyone tell you that you can’t draw the existing and new application. The drawing can be obtained.
Step: Server Naming
Identify the server naming convention that will be used. If your organization does not have an existing server naming convention, document a lengthy task to build a server naming convention. This will need to include all environments that will have servers
Step: Naming Documents
During this project you will get a number of documents. Document a task to create a document naming convention that identifies the final published version of a document. Due to the diagrams and multiple revisions of data, you will want to know what the final document is for review.
Step: Government data centers
If you are part of a government agency and using a service from another agency, contact me for an additional step. Here is one agency that could be an option for you: https://www.ocio.usda.gov/about-ocio/data-center-operations-dco/data-center-hosting-services
Each application will need a migration plan. This needs to show how the application will transition from the current source location to a new destination. You need to document the test plan for the application as well.
The migration plan for the data needs to be identified early in the program. If an application team establishes the migration plan late in the schedule and then has a lot of data to copy, the amount of time that they want to copy the data could exceed the duration of the program. Infrastructure teams may have a lot of backup data to move from the old source to the new destination, if that is the case then you need to start the data copy early in the program. Document a task to understand how each application will migrate their data.
Step: Services Offered by the data center
When you sign the deal for your new data center, operate on the mentality that the data center has nothing and you bring everything. Assume they don’t have chairs, tables, secure closets or anything you might find at your location. If you want something at the data center, it needs to be covered in the contract.
Step: Working with multiple vendors
If the buildout of your data center includes working with multiple vendors, you need to standardize the documents with a focus on the cabling request with the different vendors. You will lose valuable time and need to focus on the movement of data from one spreadsheet to another for the format from each vendor. Coordinate the document fields with each vendor at the start of the program.
Step: Data Migration
Identify the data that needs to migrate from one storage platform to another and the size. Start to copy this data as early as possible. If the data will migrate from one vendor to another record the risks that will go along with the new platform. For example, will you have any new security issues with using the new platform? Will the new platform include all of your CIFS and NFS or other storage components?
Step: Applications To Install
Determining what applications to install is a task to record in your schedule. The options for determining the list of applications could include: Using the existing list of applications in your secondary data center is one idea, or you could examine your entire portfolio and determine what application you want in the secondary data center. Once you have the list decided upon, that list needs to be signed off on with the executives. Setup the following tasks: Document the list of applications and signature parties on the applications that will be included in your deployment.
Step: Isolation Tests
During the buildout of your data center you need to perform isolation tests. This test will cut the connectivity from all of your other locations to your new data center location. This will validate that the new devices installed are not dependent upon your existing data centers. If you do not do this, the first time you isolate your data center from the other facilities, you may find that devices are using logon servers from another location. The isolation test must also be performed every time a new application is installed. If you install the application and do not isolate the data center, then you will not know if the new application can work independently of the other locations. You do not want to wait until the end of the program to determine that applications may not work independently. You should perform this isolation test with applications that you have moved to the cloud as well. If you do not test the operation of the cloud-based applications without connectivity to the existing locations, you will never know if they are dependent upon other devices in another facility. Isolating the data center will immediately help you determine where devices are pulling data from.
When you perform the isolation test, the items you need to coordinate with for graceful shutdown includes: Storage replication, database replication, password replication. You also need to consider the applications that might be impacted. For example, if your users access the internet from one location, you may need to move that connectivity to your other location. (You do have redundant Internet connections in different data centers right?). You may need to adjust your telephone lines to accommodate an isolation test.
This will be an entire section on test plans and the detail required in the test plans for testing ldap authentication and every other authentication method used in your network that needs to be at your new data center.