A brief history of the software development process according to my experience

Over almost 20 years, I have been involved in a multitude of software development endeavors. Out of sheer luck or just time passing, together they seem to show an “evolution“ worth sharing. Even if not systematic and according to a scientific process, experience does provide insight and I hope lessons to be learned.

source: pixabay

5 scenarios of software development processes are presented and at the end a brief list of takeaways. The framework includes:

  • Context, key circumstances surrounding the scenario. Context tends to give meaning to options made process-wise.
  • Process, a brief description of the process in a broad sense, including customer interaction, estimations, etc.
  • Advantages/shortcomings, a set of impacts by following the process. Not clearly classified, because that is open for debate. Like ACID, some constraints are a blessing for some people and a curse for others.

Note: Expect some very compressed statements below, rich with content but hard to follow. Please excuse my brevity.

“Ad hoc process”

Context

End of XX century. Small dot-com company “changing the world” at a fast pace. Including all the mumbo jumbo that brick and mortar business was dead. Small project teams by functional areas doing consulting services, in an effort to turn those projects into repeatable products.

Process

Customer-centric. Iterative in a customer feedback loop. No formal test phases besides customer acceptance. No particular technical best practices followed. Tech-savvy team members coding as a craft. Mostly fixed price projects. Pricing is done more according to the level of competition and customer “reading” than actually effort estimation.

Advantages / shortcomings

Apparent agility seemed innovative and seductive. Worked fine for well-behaved customers, who knew what they wanted. Failed miserably otherwise. Scope creep. Long hours. No definition of done. Low margin. Good for making money with maintenance contracts. Code artisans led to strong code ownership even if short lived.

“Waterfall”

Context

Big telecom operator. Moving from a highly decentralized growth organization to a more stable, centralized by specialties organization. IT consolidation, including IT demand management and cost control. High degree of business process digitalization and integration.

Process

Development teams by application area. New product features would require cross-application developments (CRM, Billing, ERP, Web, BI, Middleware…). 4/5 yearly releases. Cascading design phases. Some degree of decoupling via SOA stubs and mockups. Shorter life cycles delivered sooner for small independent features. Multiple waterfall cycles running together for a centralized acceptance phase. Each application area with developers and testers. Operations and reference data teams isolated. Design and documentation by functional area. Little mobility between application areas. Manual testing.

Advantages / shortcomings

The risk of feature contamination turns into “all or nothing” releases. Resource exhaustion or package upgrade from a single application area freezes features. Client areas only see the end-to-end solution during acceptance phase. Multiple complex acceptance environments to support production like scenarios. Prior architecture and design phases sign off, minimize the risk of last minute design issues. Hard to adapt to scope changes. Delays and delivery risk push client areas towards thinking “small” or in small batches with a growing impact.

“waterfall ++, continuous improvement”

Context

Consulting company with an “Application Development and Maintenance” contract with a telecom operator, similar to the context described above. Focus on IT cost rationalization. The contract is effort driven, with locked rates and high penalties for metrics infringement.

Process

Most of the work delivered by the offshore team. Comprehensive design done locally. Waterfall overloaded with 3 estimation/proposal and project post-mortem phases. CMMI Level 3 (local) and Level 5(offshore). Strong focus on project management. Estimation supported by historical data, expert review and function points. Tech debt and tech adoption regulated by a joint architecture team. Minimum of 21 documents per project. Process for scope request changes. 3 project life cycles to ease the processing overload on smaller requests. The process is under control by use of metrics, issue management, and risk management; tracked at board meetings. Multiple project sign-off gates and release CAB meetings.

Advantages / shortcomings

Predictable and reliable delivery. Delivery perceived by customer areas as slow and expensive. Tech debt growth was driven by cost pressure. ‘Industrial’ code produced that gets the job done. Disengaged resources, reduced staff risk. Continuous improvement. Most documentation supports the project management, little technical value. Adequate to projects delivered under strong and adverse circumstances. No incentives to make it leaner or use more productive resources.

“Continuous Integration”

Context

Product startup trying to grow with a limited investment. Small engineering team. Key product with two distribution models: SaaS and on-premises appliances or virtual machines. Moving from a single product to multiple product lines.

Process

Daily standups and builds. Test automation. Development and test environment deploy automation. API driven development. Loosely coupled product modules packaged individually. Team per feature, 1 to 3 members. Minimal feature specification to establish ground rules and push product analysis prior to development. The team performs an initial feature architecture and/or design subject to peer review. High priority issues get delivered same day. Small releases with bug fixes and enhancements every month (to ease the burden of on-premises upgrades). Groups of new features get delivered together to create a marketing buzz twice a year, time box release. Little code review or unit tests.

Advantages / shortcomings

Flexible and granular production deploy. Little red tape. Code ownership. Focus on design control over implementation control. Reduced production issues backlog. Future proof. New environment setup with lead time and human intervention. Demanding onboarding.

“Continuous Deployment”

Context

Product startup born in the cloud computing era. Focused on the US market. Starting a new SaaS business model. Well financed and growing at a fast pace. Product drives engineering. Employees are seen as investors (stock options) with transparent access to information.

Process

Weekly planning and daily stand-ups. Multi-disciplinary ephemeral teams by product feature. User story oriented, loose sprint size. Ballpark estimation. Ephemeral cloud development and test environments. Quarter planning. Mandatory code review and unit tests. Architecture decisions pushed top-down, tactical design decisions per feature. Automation over documentation. Integration test automation. Multiple daily deploys to production for front-end, less frequent deploys for the back-end. Fast supported onboarding. Beta features for targeted customers and feature toggles as possible.

Advantages / shortcomings

End user sees product improvements every week. Ephemeral code ownership. Resource mobility. Product-driven tech debt. Undocumented status quo. Fail fast, fix fast, turns into “legacy freeze” risk aversion as the customer base grows. Talent shortage for automation roles. Infrastructure cost derives from activity — customer and development.

Takeaways:

  • Each organization is unique and operates in a specific context, processes need to be tailored accordingly.
  • The need for customer responsiveness and satisfaction is pushing organizational change and consequently process change.
  • Each process has advantages and shortcomings. Picking just by the advantages will bring a lot of organizational stress.
  • In spite of what others might “sell” you, there is no “silver bullet” software development process. There will always be trade-offs. It is more about “who you are” than “what you do”.
  • Everything starts in the organizational culture. Is failing acceptable? How flat is the organization? Is the client at the center? What is the distance between the client and any employee developing product? How are decisions made (centralized vs decentralized)? etc.
Like what you read? Give Malcata a round of applause.

From a quick cheer to a standing ovation, clap to show how much you enjoyed this story.