Agile vs. Waterfall

An introduction to Agile BI. First, we will review the key differences between Agile and Waterfall methodologies.

Why Agile? The short answer is money. You can start to see return on investment sooner. Businesses won’t dole out millions of dollars and wait several years to get a product (if they are lucky). The business environment may have taken a dramatic shift in that time. Getting working software to the business consumers in a timely manner is now required. Think of it as the Minimum Viable Product.

Let’s define the key concepts of each:

Waterfall

1. Construction projects run this way
2. Easy to measure against
3. Know estimated cost up front
4. Safe standard to follow

Agile

1. Individuals and interactions over processes and tools
2. Working software over comprehensive documentation
3. Customer collaboration over contract negotiation
4. Responding to change over following a plan
5. 80–20 Specifications. Start coding when 80% of the requirements are understood. Colocation will fill in the rest with only 20% effort.
6. Integrated QA

Now we can compare them at a high level.

In addition to process changes, technical advances in development environments and tools (e.g. Continuous Integration and Test-Driven Development) aid in producing working software quicker than before.

Modifying agile methods for a BI project:

How is a BI project different than a typical agile OLTP project?

First, we will take a look at the architecture of each.

As you can see, there are more pieces. Multiple sources, repositories, downstream databases, and applications that increase the complexity. This can be broken down into the Breadth and Depth of Complexity. Breadth includes extracting, cleansing, integrating, and transforming. While depth includes conflicting data definitions, business rules, and high data volumes.

Agile Adaptations:

Creating another layer of requests under user stories

These are developer stories. If the user story is being able to view sales by product type and region over time. You may need to break this down behind the scenes and have tasks for product, customer, geography and sales data loads.

Adding additional team roles. *A person may fill more than one role.*

Project Architect — business model. Balanced tech solution

Data Architect — logical/physical model

Business/System Analyst — Processing patterns, transform rules, Source to Target Mapping (STTM)

System Tester — Validation

Organizing the team into a pipeline of delivery activities:

The architect has to be just ahead of the ETL developer. And, the ETL developer has to be just ahead of the report developer. Etc.

Conducting two lead-off iterations—This will give the Project Architect and the Data Architect time to establish the environment.

Splitting user demos into two stages—The first will be static test data that can be used to testing. The second is production data for full scale testing.

Maintaining many small managed test data sets—Relating to the previous adjustment, the agile team needs quick turnaround of test cycles. Often consisting of multiple times per day. A small set of data is to facilitate this.

Visually the Roles for the Sprints look like this:

As you can see, there are a few steps to come up to full speed; but, you quickly reach full velocity and producing functional BI applications.

Converting a traditional development team to using agile methods:

Here are will look at the steps you go through to transition a team to agile methods. If your organization is new to Agile, think of this as a Proof of Concept project. Start small and allow for some challenges and use it as a learning experience. You should have at least one experienced Agile/Scrum person to educate and guide the project team.

Stage 1 — Time Box and Story Points

1. See how much working code you can get done in a sprint. Everyone will get better at estimating.
2. Introduce estimation, product backlog, burn down and velocity
3. Most likely fall into Waterscrum or Scrummerfall. Waterscrum is dividing each sprint into a mini waterfall methods where each phase is only a few days each. ScrummerFall is making each phase of a waterfall project equal one sprint.
4. Can take 2 sprints

This is the initial adjustment phase. You introduce Scrum concepts and work through resistance to change.

Stage 2 — Pipelined delivery

1. May be hard to keep everyone productive at first
2. Colocation can now be seen as a benefit
3. Introduce Pipeline development when productivity drops. By Sprint 2 (See chart in Part 1), everyone is a full capacity and are not waiting for handoffs. Each member is doing design, development, and testing.
4. When quality of hand offs (Architecture to Developer, Developer to Test) reaches a high level, move to next stage.
5. Time: 2 –4 Sprints

Stage 3 — Developer Stories and current estimates

1. Transition to developer stories (E.g. Load Customer Dimension)
2. Manage Sponsor’s expectations. Change in estimation process. If you are moving to story points, or you can stay with hours.
3. Develop new current estimate with buildup of experience. After a few months, you will be much better at estimating what can get done (Velocity).
4. Time: 1 –2 Sprint

Stage 4 — Manage Development Data and TDD

1. Eliminate data churn. Do this by using a small test data set describe in part 2.
2. Test Driven Development (TDD). Use automated tools.
3. Time: 2 –4 Sprints

Stage 5 — Automatic and Continuous Integration

1. Fail Fast. Use Continuous Integration tools to automate builds and testing.
2. Can be used as a short cut to Agile if implemented early (stage 1)
3. Time: 2 –4 Sprints

The average time to become comfortable with agile methods is 8–16 sprints. Try and make your first candidate project something that is in the 9 to 12 months range with a senior development team and a good agile practitioner.

How is Agile worth the effort of converting to? Here is a chart to review the benefits.

Conclusion

We discussed some modifications to Agile to fit a data integration project and the steps to help a traditional development team transition to an agile data integration development methodology. After becoming comfortable with agile development, you will be able to reduce the time e to market of data integration projects significantly.

Originally published at www.capaxglobal.com.