How to Buy Enterprise Software

A modern guide for enterprise software evaluation.

On the 7th of November 2016 the Waratah room in Sydney’s Parliament House was filled with the sober atmosphere of bureaucracy covering its tracks. Five executives sat under examination from the Public Accounts Committee. They were being interrogated on the decision to dump the student management system they had been attempting to implement since 2012.

Earlier that year the State Department of Education and its vocational training provider NSW TAFE had quietly reported that the estimated total cost of its failed learning management and business reform (LMBR) program was $752 Million, and that it would ‘cut its losses’ and dump the student management system it had been labouring on for four years.

This was a huge admission of failure.

Replacing the student administration systems was a heart transplant for NSW TAFE and would effect users from the head office right down to school-based staff and students.

They had gathered requirements through a strict government managed procurement process and selected a primary vendor for implementation. In addition, to de-risk the project, NSW TAFE engaged two further global consulting organisations, one as business partner and transformation advisor and another as independent quality assurance.

Yet by 2015 the NSW audit office had rejected NSW TAFE’s financial statements after it was unable to corroborate revenue claims. The new student management system had lost track of enrolments and revenue. Outside of direct project costs, it has since been revealed it took 185 accountants and $10 million dollars to get NSW TAFE’s books back in order following the disaster.

The challenge that we had, and we could have managed better, was that each time we agreed what we were going to build — a new essential requirement came along!

So now, sitting in front of the Public Accounts Committee, Mr. Peter Riordan — the Deputy Secretary Corporate Services for the Department of Education, has the unenviable job of explaining why a failed IT project will come close to costing $1 Billion dollars and how they intend to fix the mess. Untangling the cause of the issue he suggests “The challenge that we had, and we could have managed better, was that each time we agreed what we were going to build — a new essential requirement came along!”

Ah ha! Scope creep.

Except, blaming scope creep doesn’t make any attempt to understand the true underlying cause. Was it that NSW TAFE didn’t fully understand their business and requirements before selecting a vendor? Was it that it took them 2+ years to implement and in that time the business changed? Was it simply poor project management?

The answer is in the approach NSW TAFE has taken to remediate the project.

You see, on the very same day they announced they were ditching the failed student administration system, they also announced they would be going to market for a new ‘agile’ solution for student administration. Detailing requirements for the new platform they asked for “an agile and scalable solution that will facilitate a positive and effective user experience for students and TAFE NSW staff” and adding “It is envisaged that the solution will be cloud based”.

Replacing the underlying system indicates they felt a fundamental issue was poor platform selection. A failure of procurement. Should they have selected a more ‘agile’ solution to meet their changing business needs initially?

If procurement and evaluation of solutions was ineffective the first time and resulted in the overall failure of the project, then the comments of Mr Glen P Babington, COO of NSW TAFE, during the sitting of the Public Accounts Committee should have set alarm bells off. Referring to the actions taken to find a new student management platform he stated “We will go through the normal procurement guidelines to get the system that best suits us”.

Now, if insanity is doing the same thing over and over again while expecting different results, it doesn’t take Albert Einstein to see the fault in their plan.

No doubt there are many failed IT projects where the finger is pointed at implementation, or change management. I postulate that the issues begin well before then. Most organisations are ill equipped to evaluate and select enterprise IT solutions. They fall back on ‘normal procurement guidelines’, adopt feature based ‘apples to apples’ comparisons, or use tendering processes to mistakenly claim due diligence.

Effective business software evaluation is a skill few organisations are adept at. It’s a costly and high risk process.

It’s time we re-thought the enterprise software buying process.

The Way We’ve Always Done Things

The Software Acquisition Funnel

The traditional buyer journey is modelled as a software acquisition funnel that progresses from awareness, through requirements analysis, evaluation and eventually results in vendor selection. Its the flip side of the traditional sales funnel.

Conceptually this model makes sense. As buyers learn more about their requirements they gain clarity on the best solution for them. The simple logic is appealing and explains why most organisations still procure software using this framework.

This is not an effective way to buy enterprise software.

B2B sales studies and over a decade of commentary highlight this traditional logic is flawed. The flaw is the assumption that buyers progress linearly from top to bottom.

Buyers are much better informed than ever before. Contrary to popular belief more information doesn’t de-risk our decision and move us closer to an outcome. Often it forces us to rethink previous conclusions.

This is compounded by the fact B2B purchase journey’s involve multiple people and teams. In “The Challenger Sale” Matthew Dixon & Brent Adamson highlight the average number of enterprise decision makers is 6.8, and the more stakeholders in a decision the less likely a decision will be made.

It is more accurate to think of the modern software buying process as a cycle. Unable to make a decision organisations loop from evaluation back to the top of the funnel. They drop out of this cycle and make a selection only when a compelling event occurs.

This thrashing or treading water, costs money, wastes time and worse, limits growth and organisational progress.

Waterfall evaluations with hefty requirements gathering upfront are misconceived. Any delays in procurement and implementation accomodate changes in the business conditions. The funnel procurement mindset fosters low quality decisions.

An organisations inability to buy often hurts them just as much as a failed implementation.

Enterprise software has been revolutionised by cloud making it faster, cheaper and more flexible. Software implementation has been revolutionised by the Agile method increasing speed, versatility and IT/business collaboration. This shift to cloud and Agile creates an opportunity to move faster, fail faster, and ultimately de-risks enterprise IT projects.

Organisations can further increase the impact of technology on the business by de-risking their procurement model too.

We need a better way to evaluate software to keep pace with technology and the business.

Searching for the Magic Formula

The goal of enterprise software evaluation is to make the best possible decision, given the available information, in an amount of time that doesn’t further impede the organisation.

We can begin by considering the prevailing evaluation extremes:

• Pick a market leader and start with a small pilot (just do it),
• Full requirements gathering/rfi/rfp/tech assessment/implementation plan etc (the long, arduous, fail prone traditional funnel), or
• Find an expert to tell us what to do (outsource everything).

All three are viable paths. All three have various pro’s and con’s.

The ‘just do it’ option trades off short term wins for long term risk. It de-scopes long term capability. The ‘outsource everything’ option shifts the burden of effort. It increases the cost of change and disengages the business. Un-compromised, these extremes are illogical and introduce too much risk.

By default organisations tend toward the traditional funnel option or a blended model. The traditional funnel view unproductively focuses too much effort on data collection and comparison of options.

Powell’s 40/70 rule.

Former US Secretary of State Colin Powell is credited with the 40/70 rule.

When faced with a decision you should never have less than 40% of the information or you’ll be guessing, but you should also never have more than 70% or the opportunity will pass you by. We’ve been trying too hard to have all the information. Thrashing and cycling are the common symptoms and mediocre decision making is the outcome.

We need an another alternative.

To improve our ability to buy we should focus on what the best possible decision for us looks like, and how much time we have to make it. What we want is an optimal process that limits thrashing, overcomes the challenge of decentralised decision making, and a credible evaluation model without detailed requirements dependence.

A Five Step Evaluation Plan

Your business technology is constantly evolving. The days of buying a system, implementing and letting it run unchanged for the next few years are long gone. Best practices would state you have a pipeline of IT projects. A business analysis team can initially assess each of these for priority as capacity to begin a new project arises.

Once you identify the need, here are five steps to guide you through the effective selection of an enterprise software solution. Think of these as the 5P’s of software procurement.

• Plan — Use scenario planning to build a narrative of how the new systems will benefit the business
• Pitch — Engage potential vendors and shortlist the most viable based on an initial pitch for business
• Prove — Evaluate vendor capability using technical assessments, references, demo’s and other methods
• Propose — Knowing what we now know, build a model for each vendor that leverages their strengths
• Progress — Select a preferred vendor and begin planning for implementation

A key differentiator for this method is that a number of the steps produce artefacts to gain buy-in and agreement from the decision making group. This momentum and progress stops the need to cycle back and revisit previous steps, reducing thrashing.

The 5P practice is designed to keep minds open by setting a direction and deferring solution specifics. This gives vendors enough room to make recommendations, call attention to areas of friction and ultimately better emphasise disparities.

Plan: How to Build a Flexible Strategy

To build an adaptable long-term vision we use scenario planning techniques.

Scenario planning is used to navigate uncertainties and the thrashing or decision paralysis that can result. Scenario planning will help us expand our thinking, produce a strategic plan that maps a path from the current state to the future state, and builds support and enthusiasm within the business.

Conventional scenario planning establishes divergent scenarios and then looks to build strategies in response. Given the defined scope of our task is to select a new software platform, we reverse conventional scenario planning. Because we build scenarios from a set of fundamental options, this method may be better described as Narrative Planning.

You will be creating four divergent narratives. I call them:

• Treading Water,
• Bulls-Eye,
• Nightmares, &
• Daybreak.

A regular blunder of the traditional procurement funnel approach is to spend significant time and resources only to eventually decide to do nothing. To avoid this we will first build out the ‘Treading Water’ narrative.

The Treading Water Narrative

If we did nothing, what would happen?

“If we did nothing, what would happen?” is the question this narrative answers. We need to assess if we made no change to this IT system what effect that would have on our business. The outcome is a plausible storyline about the business, the market, competitors and the influence of new technologies. It paints the most likely outcome should we do nothing.

This is the most important of the 4 narratives as the others will use it as a baseline.

Developing the narrative will take a mix of research, insight, creativity and analysis. There is plenty of literature on the execution of scenario planning (I recommend Peter Schwartz’s book ‘The art of the long view’).

Begin by sharing the details of the project with the team, give individuals time to complete their own research, then come together in a workshop/brain-storming session to map it out. If possible include outsiders to add broader points of view.

Specifically you should look to identify trends that are already unfolding, and map out inevitabilities. Inevitabilities are the predetermined outcomes of changes already underway. For example, it may be relevant to your business that baby boomers are approaching retirement age, or that new information security laws where recently passed in your market and this will force all your competitors to migrate from legacy technology at once.

Your final Treading Water narrative must be a shareable picture of the most probable eventualities. It’s a vivid image of the future.

For example, a waste management organisation looking to replace job scheduling and tracking software identified the below key plot points for their Treading Water narrative:

1. The current state of play: The current system doesn’t optimise truck routes which results in manual rescheduling monthly — costing significant time — and route inefficiencies directly impacting truck operational costs. A 5% increase in efficiency would result in close to ~$1M pa in savings. The current system offers no API, is custom coded and was built prior to modern mobile GPS availability.
2. Our competitors will update their job scheduling platforms. This will allow them to offer real time tracking to their customers and immediate custom scheduling as opposed to route based jobs. This improved customer service will give competitors an advantage.
3. Increased focus on environmental impacts will escalate waste governance. New compliance measures will likely mean trucks will need further detail on the waste in their vehicles before they can dispose it. This will force amendments to the current truck onboard systems or additional manual work for drivers. Competitors with newer more agile systems will adopt the changes faster and with less friction.
4. IOT sensors on bins to monitor weight will allow for increased optimisation of schedule to ensure trucks with adequate capacity arrive to specific jobs.
5. Driverless vehicles will significantly reduce costs and upend the market, however the complexity of end-of-trip waste collection will retain the need for human occupants for some time yet (E.g. unlocking gates and accessing bins).
6. The possibility of self-powered vehicles that run on their own waste will further automate the waste management lifecycle. Competitors ability to adopt new driverless and self-powered fleets readily into their business will enjoy significant cost savings and fund further adoption.

A narrative should layout events in a chronological order. We should seek to make estimates of timelines, and measures such as cost, time savings or customer satisfaction. It helps to approximate the magnitude of impact.

The narrative isn’t a prediction of what will happen, its our best guess of what will happen. It forces us to think of second and third order consequences. It’s used to justify and make better decisions.

However you create it, the Treading Water narrative will be a map of your business, in your market. It will chronicle the compounding advantages or disadvantages your competitors, employees, and customers will experience over time in relation to your non-investment in this area.

On completion of the Treading Water narrative it will be shared with project stakeholders and a ‘Go/No-Go’ decision should be made. If there is no major impact on your business — this isn’t the project you should prioritise.

If there is, you should now have commitment to action. The Treading Water narrative is documentation of your compelling events to avoid cycling and thrashing when further decisions get tough.

The Bulls-eye Narrative

Using what we have built for the Treading Water narrative, lets rewrite it so that we are an active player.

The Bulls-eye narrative scripts out a path to success. It spells out the investments we will make in technology, how they give us advantages, how we are prepared for the new trends and disruption, and how our employees, customers, and business will benefit.

The Bulls-eye narrative is written out like a book. It describes the decisions we will make and the positive outcomes they drive. The positive outcomes compound and open up new possibilities.

Picture yourself in the role of various users to identify benefits from their perspective.

Estimate the value of the outcomes. If anyone else in the industry has done this before, use their results to guide your estimates. If not, use your best judgment and estimate expected results, not best or worst case. This is a narrative not a business case so precision wont be scrutinised.

For example, we might say — The new contact centre system must be intuitive and simplify data capture for contact centre staff. By reducing current manual entry and copy/pasting between systems we will save 1hr per day per employee.

The additional 1 hr will be applied to an up-sell strategy. Contact centre staff will execute guided up-sell scripts during calls. The volume of inbound calls to the contact centre will allow us to rapidly A/B test up-sell scripts and continuously improve up-sell success ratios. We would expect a 5% increase in revenues, and we would see a reduction in contact centre staff turnover due to the up-sell gamification and therefore increased staff engagement.

On completion of the Bulls-eye narrative you will have a picture of what success looks like for the business and for key personas. It defines a path to an optimal solution and, at a macro level, outlines the key success factors. The Bulls-eye narrative should be circulated widely to ensure everyone is aware of the long term view.

There are huge behavioural benefits to completing a Bulls-eye narrative.

Sharing the Bulls-eye narrative scales the benefits of mental process simulation to teams and organisations.

Without it project teams proceed with a view of the outcome based only on awareness of an immediate pain or need. Many studies show mental process simulation improves performance and increases the likelihood of success. Narrative Planning formalises the benefits of mental simulation across an organisation.

In the study Harnessing the imagination: Mental simulation, self-regulation, and coping. Shelley E. Taylor, Lien B. Pham, Inna D. Rivkin, and David A. Armor show that visualising the steps, actions and effects required to achieve an outcome both significantly improves the chance of achieving it and makes our perception of the effort required in achieving it easier.

Their study specifically differentiates between visualising the outcome versus visualising the process towards the outcome (the act of process simulation).

They also show that process simulation greatly improves the planning fallacy dysfunction. The planning fallacy (Buehler, Griffin, & Ross, 1994; Kahneman & Tversky, 1979) refers to the fact that people invariably underestimate the resources, such as time and money, that will be required to finish a project and overestimate how easily it can be done.

Taking students who had a significant upcoming project to complete, they trained some of them in process simulation techniques. The results are astounding. Where only 14% of students in the control group began the project at the time estimated, 24% of the process simulation group did. And, where only 14% of the control group finished their projects on the time estimated, 44% of the process simulation group did.

Traditionally detailed planning, timetabling and gantt mapping have been the corporate cure to the high risk of the planning fallacy. Yet it is proven in every domain from construction to professional services to be ineffective. Doubtless because detailed planning itself is time consuming and underestimated.

Creating and sharing the Bulls-eye narrative scales the benefits of mental process simulation to teams and organisations. It’s a relatively low calorie, high benefit exercise that will align the organisations view forward and help to anticipate and manage consequences.

The Nightmares Narrative

While the Bulls-eye narrative defines our desired path, the Nightmares narrative forces us to think through all the weaknesses, hazards and alternate possibilities. The value of this exercise is to open our eyes to potential roadblocks.

The Nightmares narrative exists as a list of ‘What if’s’. For each we define the impact that has on our Bulls-eye, is there anything we can do respond and course correct, or does it define an entirely new course for us all together?

When diving into each What-if nightmare search for the drivers that lead to that path. These are the signals that flag a nightmare is happening. Being aware of the indicators, watching for them, allows us to respond quickly.

Nightmares can emerge from many sources. A change in market conditions might constitute a nightmare. For example, a hospital might anticipate a rise in day surgery that would change the demand on patient admissions. Another nightmare might simply be the low adoption of their patient portal.

Nightmares should be prioritised by probability. Those that arise earlier in the narrative are generally more concrete but the probability may decline with time. Those further out might be uncomfortably inaccurate. If the consequence is high even the wild possibilities should be considered.

Risk arises from both probability and consequence.

The Daybreak Narrative

You now have commitment to do something, an ideal path forward and a list of uncertainties that may impact our success. The next step is to assess our options for moving ahead. A key realisation here is that the path forward is not binary.

We have more options available to us than to proceed or not to proceed.

There is a middle-ground where we can minimise the downside risks of the uncertainties and maintain the upside benefits of the project. Recognising the full range of options at our disposal, and structuring our project to exploit them greatly increases our chances of success.

For example, options might be:

• Deferral — By waiting, would we reduce the risk of unknowns and increase the viability of the project?
• Expansion — Are we thinking too small? Would investing bigger halt a potential nightmare scenario?
• Contraction — Are we thinking too big? Could we realise 80% of the value with 20% of the investment?
• Suspension — Is the potential availability of resources to complete the work a concern? Could we start, then pause the project if needed?
• Segmentation — Are there parts of the project we could defer (could we limit the initial scope) until such time as consequences are certain.

Irreversibility is a core fear of strategic IT decisions. It’s why every vendor will trumpet ‘open and agile’ as a benefit of their solution. If organisations can postpone or avoid irreversible decisions without impacting the benefits of full commitment, they should.

The daybreak narrative is about identifying and limiting irreversible decisions.

There are a couple of things to consider when identifying these junctures. An option must not give up the advantages of full commitment. Ideally it would place us in a preferential position relative to where we are today without locking us in to any one course of action. They should also be a fraction of the cost of full commitment in order to provide value in limiting loss.

The management technique Real Option Analysis offers a formal but complex way to mathematically value these possibilities. The International Journal of Industrial and Systems Engineering article, A framework for applying real options analysis to information technology investments highlights how this might be used for IT projects.

I worked with an organisation in 2017 who was bullish on Chatbots. They were implementing a customer service platform and seeking to introduce new, cheaper customer interaction channels beyond phone support. Their research had identified leveraging Chatbots.

The risk that the Chatbot customer experience might be poor due to the relative newness of the technology, or that by go-live the chatbot market may significantly evolve due to the speed of change in the space was established.

A standard, manual web chat channel is without the risks inherent in a chatbot. Running standard web chat would allow them to build knowledge and log chat interactions, thus increasing the viability of a Chatbot implementation and improving clarity on its value. This was a strong example of where segmenting and deferring limits sunk cost risk. The potential downside was deferred cost reduction, and a possibility of falling behind the market should chatbot uptake be swift — which was minimal.

In this instance implementing standard web chat and deferring chatbots put them in a preferential position and poised them to more rapidly adopt chatbots in the future.

Considering our Nightmares and Bulls-eye narratives and pinpointing the key junctures will result in us defining a minimal rational model. This is our Daybreak narrative. Anything before the first juncture are the fundamental capabilities of our Bulls-eye. It’s where we begin.

An Accurate Plan vs A Precise Solution

We now have commitment to action, a shared vision for our future, an evaluation of risks with links to indicators, and a minimal rational action model. This constitutes an accurate plan for achieving success.

The traditional funnel model advocates deep requirements gathering with the ambition of precisely defining a solution.

Both models offer alternate approaches to evaluate software. Both are predictions of what is needed. But any prediction or forecast needs to be questioned as the future is inherently unpredictable.

Accuracy vs Precision

When dealing with uncertainty it’s clear that getting hung up on precise details and metrics is futile. The most important factor is asking and answering accurate questions. Accuracy refers to how close we are to the actual outcome. Precision refers to how detailed or exact our prediction is.

The traditional funnel model ignores uncertainty because its complex. In an uncertain world, overcoming project risk by collecting more precise, detailed requirements is a waste of energy. The Plan step of the 5P software procurement model benefits from focusing energy on the accuracy of its design. It also brings the organisation along on the journey.

The bottom line is, you can precisely define a requirement that has very little relevance to your overall success. Instead, spend your energy accurately defining success.

Pitch: How to Go-to-market for a Solution

As we enter the market looking for suitable solutions our goal should be to execute a process that results in the best selection, while minimising the effort involved in evaluation.

To run an efficient evaluation we need to decide who will decide. The probability of thrashing and low impact decisions increases if we have not defined who makes the final decision and when they make it.

Ultimately it doesn’t matter who has responsibility for the final decision but they must firstly be senior enough to sign off on the project. They should also at least be aware of the project, be onboard with the plan phase (the go/no-go decision plus the narratives) and be prepared to own the decision within the timeframe.

While it’s best if one person owns the final decision, most enterprise organisations will assemble a project team to assess and recommend a solution to the decision maker. In bureaucratic organisations this is another potential bottleneck. Make sure all ‘recommenders’ understand their responsibility and the scope of the decision or recommendation they are required to make.

Keep the total people responsible for decisions or recommendations to a minimum. In most business IT projects this can be kept to 4 stakeholders. The final decision maker and a representative for each of

• The users — they are responsible for viewing the solution through the eyes of those users/employees who will be using the solution most. They typically look at how intuitive and easy it will be for these people to do their job, how much change it will require, how productive or how much it will improve their performance. If key users include external customers or stakeholders they should own this lens too.
• Management — responsible for assessing the solution from the perspective of the organisation. They want to understand reporting, visibility, and generally what data they can capture and track. How can they monitor and drive performance. Costs and efficiencies are key.
• IT — responsible for understanding security, integration, data management, customisation, maintenance, ongoing flexibility, etc. Accountability for the non-functional assessment and suitability of the technology falls to this person.

Enterprise software in bygone years did not truely consider the users. We recognise green screens are out and mobile apps are in, but do we equally weight the needs of the users?

Original enterprise software existed to track business data and rigorously execute business processes. The consumer tech revolution has changed modern enterprise technology. Usability and simplicity can make or break enterprise software projects today. User expectations are high. Giving equal assessment value to the users cohort recognises this shift in business technology.

Some software evaluations wont need stakeholders from each of these groups. For example, if its a purely IT platform decision — like a database or integration tooling that does not directly impact management or users — then we may get by with just an IT decision maker. The less decision makers the better.

So we have a project team with decision makers defined. Now its time to approach the market.

Here is what you don’t do. Do not write an RFP.

Or an RFI, or an RFQ. Or an RF-anything.

Do not write an RFP.

A Request For Proposal is a massive waste of effort. RFP’s chew up huge amounts of time to both write and respond to. They rarely provide insights that couldn’t be gathered from face to face conversations. The usual structure conforms all vendors to a similar mould and increases the difficulty of differentiating between the top solutions. It’s a request for punishment.

Organisations often use them to vet vendors, in an effort to prove due diligence, or in the belief that written responses reduce risk. But RFP’s are an ineffective tool on all counts. There are lower calorie ways to vet vendors, and due diligence is better governed by scrutinising decisions. Vendors don’t have the detail to give specific responses at the RFP stage and will often issue caveats or respond generically which limits the veracity of written responses.

RFP’s are an illusion of progress and control, but the biggest issue with them is they achieve the wrong outcome. At the early stage of approaching the market for solutions organisations should be broadening their options, not reducing them. Using the design thinking paradigm, we want divergent thinking before convergence on a solution.

The Design Thinking Paradigm.

What the RFP does wrong is pre-define a solution. Think about it, you reach out to the leading technology innovators and rather than ask for their advice, you prescribe adherence to your solution.

When first approaching the market for enterprise software just communicate your situation, challenges, and objectives. You want to hear vendors different perspectives, ideas and designs. This helps broaden your thinking and at worst validates any preconceived concepts.

Keep things simple, write a brief. A brief is a summary of the project’s purpose.

Your project team can take the narratives from the Plan phase and write up the vendor brief, with viewpoints from each stakeholder cohort. As it’s a high level document it should take a fraction of the time it takes to write an RFP.

It’s crucial that the brief focuses on the problem, not the solution. Define only business objectives, not criteria or requirements no matter how critical they seem.

Next, investigate potential vendors. Choose a broad but manageable set of options to maximise contrasting ideas. Send them the brief and ask them to pitch their solution.

Give each vendor equal opportunity to present or demo but leave the format of the pitch up to the vendor to decide. Be open with the vendors that they are in a competitive situation, and allow them to interact with you and learn about your business. You may ask that vendors indicate price during their pitch. As its early days, a good approach is to ask for a range including low end, high end and best guess price estimates.

For each vendor you invite expect to meet them a couple of times prior to their presentation. At least once to walk through the brief and likely once more for the vendor to complete high level discovery. A one hour discovery (Q&A) session run by each vendor is a worthwhile commitment of your time. It allows you to feel out the vendor and ensures each vendor can ask for the detail they require to present their point of view. Attempting to shrink your commitment further will only dilute vendors’ comprehension and reduce the impact of their pitch.

The goal of the Pitch phase is to learn. We are still educating ourselves about potential solutions, vendors and approaches. With relatively little effort we get a wide understanding of available options. The next step is to shortlist a subset of vendors.

Prove: How to Validate Solutions

Begin by creating a shortlist of the top vendors. We need to invest more time with each remaining option. Make a judgement call and choose the top three you can agree are most likely to deliver your Daybreak and then Bulls-eye narratives.

For organisations that need to justify their actions, use the 4 stakeholder axis to assess each vendor pitch. The 3 cohort stakeholders should prioritise choices based on their needs. The primary decision maker should be looking at the big picture and the ability to achieve the organisation’s Bulls-eye narrative.

Keep this decision group to a minimum, hold a collaborative session soon after the pitches are complete, down-select the top vendors and keep up momentum.

We now have three viable options.

Next we probe and test these solutions to build confidence in their capability and identify weakness or friction. The goal is to gain a deep understanding of each solution, not to compare or contrast them. Don’t treat this as a competitive bake-off. We are starting to build working relationships and openly sharing information ensures an optimal outcome.

We want the shortlisted vendors to show us a vision of our future and a path to get there. During this process we assess the risk profile of each solution.

The vendors will need a deeper understanding of your business to accurately propose a solution. A good way to start out is with wider discovery sessions for each vendor. Assemble a team from each cohort and allow each vendor to meet and interview them to uncover their needs and pains.

Vendors may request ride-alongs, sample data, demos of existing systems, or even workshops to fully grasp your business needs. Entertain all requests to the extent possible.

You may need to identify a project lead or vendor liaison at this stage to manage vendor interactions. For this reason it’s a good time to attain any internal project approval or funding.

There are a number of tools you can use to assess capabilities and establish confidence in the solutions. The exact process is up to you. Different projects may require different tools.

Below are the most commonly used activities.

• Presentation — the simple presentation offers the vendor an opportunity to explain and educate us on their solution. Its primarily a one way dialog with the goal of increasing our understanding of the solution, or an aspect of the solution.
• Workshop/Q&A — a session that requires collaboration and two way sharing of information. Usually used to resolve specific uncertainties. The outcome is typically unknown by either party before the session and the goal is to come to a common agreement on how something will be achieved.
• Demo — a staple of the software evaluation/procurement process. Demo’s offer the vendor a chance to display their solution and customers get to experience it. Many organisations run a checklist demo, with requirements that must be shown and ticked off. This can often hide other complexities and makes it hard to differentiate the experience. I prefer persona based demos, identify key personas and ask for a demo through each persona lens. Don’t prescribe the demo script, allow vendors some freedom to exhibit their solution.
• POC — a Proof Of Concept is typically used to resolve a technical risk or uncertainty. It builds out a piece of functionality to prove its possible. A POC may also be used to understand feasibility. How easily or conveniently can something be done? If a demo shows WHAT the experience will be, a POC can show HOW it is achieved. Usually a POC is run by IT teams to validate uncertainties. POC’s are high-calorie activities and shouldn’t be used where another form of proof will suffice. It’s notable that POC’s are throw-away work and will not directly progress into production.
• User Testing / Hands-on Assessment — used to ascertain usability and ease of use. Often this can help determine required user training, adoption friction and change management plans. Be aware, depending on the solution, it may take equivalent work to build a prototype for usability testing as it would a full system. Hands-on assessments are another high calorie activity so consider the value of executing them carefully.
• Pilot — used to mitigate change management risk. It’s a roll-out to a contained subset of the business first where the benefits are monitored. The Pilot is used to learn from before endorsing massive change across the broader business. During a Pilot, failure or rollback of the project is still a viable option. Be aware Pilots go-live and organisations typically only Pilot a single solution, so it’s not a tool to resolve between different vendor solutions.
• MVP — used to mitigate scope risk. A Minimal Viable Product validates the project concept. An MVP maximises our learning with the minimal amount of effort, its most often used in customer facing applications. In enterprise software it also promotes an agile approach where we start small and expand, de-risking the project by shrinking the feedback loop. Again, MVP’s go-live, so it’s not a tool to resolve between different vendor solutions.
• ROI Model — the Return On Investment model highlights investment risk. Determining ROI for technology projects is difficult as many benefits are intangible. How do you quantify user productivity or measure engagement? For this reason organisations often skip ROI modelling. Vendors may assist in building ROI models with guidance and metrics from previous implementations. Start with an effort to understand TCO (Total Cost of Ownership) as direct license costs aren’t enough to plan budgets.
• Analyst Reports — third party validation can help support your decision. The analyst viewpoint may help you direct further questioning. Be sure to understand the scope of any report you rely on, there will be more value in the detail than in any overall ranking.
• References — allow you to validate what a vendor is saying. Ideally you’d speak to at least 3 existing customers with solutions similar to your proposal. Vendors will only offer happy references so the volume of references is important, also try to speak to a local reference. This is a great opportunity to learn from those who’ve gone before, probe for bumps and learnings.

The Prove phase is about building a level of trust in our understanding of the solutions. If you can not map out how the solution will be used to achieve your Bulls-eye narrative with confidence, you have not completed the Prove phase.

Up to here we have been evaluating the software solution in the absence of an implementation partner. Obviously the right implementation partner and approach impacts project success as much as the software selection.

As we shortlist vendors, the Prove phase may be the right time to begin engaging implementation partners for each vendor.

Assessing implementation in parallel with solution capability from this point forward has some benefits. Engagement here will reduce repetition of discovery and assist in building a full picture of the proposed solution. I don’t cover implementation partner evaluation further but it is worth our consideration.

Propose: How to Make a Decision

In the Propose Phase we select a preferred vendor.

It’s easy to fall into the trap of comparing the options against each other in a vacuum. The idea of this being a competitive process with a single winner leads many organisations to lose sight of the outcome.

In Amazon’s 2016 shareholder letter, Jeff Bezos discusses the need to resist proxies. He cautions that in large organisations ‘The process becomes the proxy for the result you want. You stop looking at outcomes and just make sure you’re doing the process right.’

For organisations that haven’t put thought into their software procurement process and are probably following the traditional model by default, the final decision is often made based on scorecards and direct comparison. Decisions are justified because the process showed vendor A was optimally priced and met 85% of the criteria.

Don’t compare A with B. Don’t let determining superior capability be a proxy for determining the best path forward.

It is likely possible to achieve your outcomes with all shortlisted solutions, the decision is in the nuance. What we need to do is map out all options against our strategy. How well does each option support our Bulls-eye narrative?

The best question to be asking at this point is “What would our organisation need to change in order for this option to be the best path forward?”

What would our organisation need to change in order for this option to be the best path forward?

Model out each option and note all positive and negative variations from the narrative. Then make a plan identifying what characteristics would need to be true for this option to be the best. Consider the effort needed to make this option successful.

Assess and document these characteristics. They will make up the justification for your decision.

Reviewing the risk, change effort, probability, cost, and impact of each characteristic is a great framework for understanding the strengths, weaknesses, opportunities, and threats of each option.

Include cultural fit as one of the characteristics you consider. Organisations often rely on gut feel based on interactions with the team from the vendor. This is hard to quantify and justify. It is worthwhile assessing team synergy and trust built during the sales process. Evaluating it as an ingredient of success allows us to review it objectively.

Whichever option is selected, getting buy-in from across the organisation is essential for success. Any decisions made in isolation of the people they will effect will result in the organisation needing to hard sell the change. When people feel sold to, they are more likely to resist.

Buy-in, support and alignment are facilitated by participation.

Involve people in the decision. Share your narratives to enable an understanding of why you must act and illustrate the opportunity for the organisation. Then share the plans you’ve developed for each shortlisted solution. Include the characteristics that differentiate them and the proposed changes the business must make to ensure success. Invite discussion and let people refine the plans.

The secondary benefit of wide involvement at this stage is to ward off bias.

Per Soelberg discovered that people arrive at decisions intuitively before they do logically. In his Decision Confirmation Theory (1967) he reasons that once a favourite is identified, people seek to confirm its superiority by unconsciously discriminating against alternatives. It’s possible for project teams to identify implicit favourites early in the process and then retrospectively rationalise the decision that has already been made intuitively.

Invite criticism. If people don’t have opinions or objections then they don’t care. Indifference is worse than disagreement when managing change.

Facilitating a series of workshops with relevant teams is a great way to collect feedback. If teams are large and distributed use collaboration technology to collate critique.

The final step is for the four decision stakeholders to formulate their opinion based on the plans and feedback. The three cohort representatives can make their case to the final decision maker. Ultimately the final decision maker must choose the path forward and communicate the decision to the organisation.

Progress: Where to From Here?

All this and we’ve only selected the underlying platform. The hard work is ahead. Implementation methodologies, change management, data cleansing, integration, testing. A lot could still go wrong, but best practices for these tasks are widely analysed.

Selecting your implementation partner should be the priority at this stage.

As implementation plans evolve, detailed requirements will need to be gathered. Gathering requirements for implementation is valuable. Gathering them for evaluation is misguided. The difference is a reduced risk of change — as the lag between collection and build is short. And we are now in a state of contextual harmony. We know the prospective technology, can directly include the implementation partner, and are able to discern the necessary level of detail.

Progress is infinite. Any initiative shouldn’t end with implementation.

Too many organisations complete implementation and hand technology responsibility over to an operations team who’s remit is maintenance. Without continuous improvement software decays.

Business evolves. Competitive organisations understand technology is the leading enabler of growth. Consider your I.T. solutions a living part of your organisation, not just a project to be completed. Assessment, improvement and investment should continue indefinitely.

Check in on nightmare indicators and plan to review and tailor your narratives as time progresses. Continue to use the narratives as a tool to guide the solutions direction.

Technology Decisions are Business Decisions

The 5P software procurement practice recognises software is fundamental to modern business. It changes selection from a functional assessment to a strategic analysis.

By acknowledging the importance of technology decisions on businesses we raise the task into new awareness.

Business software evaluation is a skill that can be improved.

Technology is not a black art. Enterprise business is dependent on technology. We’ve been buying software all wrong, and it’s time we progress our thinking.

********************

All opinions and views expressed here are my own and do not reflect the opinions of Salesforce or any organisations I’m affiliated with. This article is informational and any information relied on is at your own risk.