An Attempt to Rate the technical Maturity of Startups and ISVs
Software Development is a complex beast. I have nearly 30 years of experience with coding, architecture and governance, but it’s easy to get lost in the sheer complexity of all the aspects of what makes great software.
ROI on Sponsoring
Three years ago, when Microsoft asked me to help them assess startups that entered the Microsoft BizSpark program, I had to come up with some structure to be able to evaluate the maturity of each startup in their app development. No small task, especially if you want to make it relatively objective.
My first idea was to follow the popular road of mapping the challenge onto a canvas. After all, Business Model Canvas and Lean Canvas have had their share of success.
After a few iterations, a moderately usable Tech Canvas with a basic classification was born, and quite a few startups were assessed using it.
Over the last three years, as I was working with startups, scale-ups and ISVs, I have been using the Tech Canvas on many occasions. As I found out, it helped in creating a skeleton for reporting on what I found at startups and scale-ups, making sure I covered every subject.
Although it has done a good job in quick and basic assessments, the fact that the Tech Canvas was far from perfect has always been clear to me.
Meanwhile, during the same period, designing and working with ontologies became one of my specialisms. Describing a knowledge domain in a structural way, with taxonomies (hierarchical sets) and custom relation types, turned out to be very helpful. As I discovered, ontologies can add lots of value. I have implemented ontologies for several clients, and each time, the app they were building became much smarter, for instance in search & match scenarios.
Case: IT Domain Ontology
Let me showcase one of the projects, in which I had to automate assessment of groups of LinkedIn Profiles in terms of seniority on given (IT) topics. It turned out that, whereas it was fairly easy for someone like me (with a certain level of domain expertise) to do quick assessments of someone’s LinkedIn profile, automating that process turned out to be much, much harder. Why? For a number of reasons:
- Terminology to describe one’s skills varies, and LinkedIn is pretty forgiving in describing them (it offers autocomplete but doesn’t enforce a closed selection)
- Job titles in IT are anything but standardised. For instance, there are quite a few job titles that describe people who write code for a living: Software Developer, Developer, Software Engineer, Senior Software Engineer, Programmer, Coder, Architect, Software Architect, etc.
Developing an IT Domain Ontology turned out to be the only structural solution to this problem. An ontology can be seen as machine-readable knowledge around a topic.
What I ended up developing had a combination of a Lucene full-text search with fuzzy matching, and a (limited) IT Ontology with roles, functions, technologies, platforms, coding standards, stacks, etc. that was quite reliable in analysing LinkedIn profiles.
Incorporating Ontology
With the knowledge of ontologies, my original ‘Canvas’ approach has evolved into a more structured one, with an underlying knowledge graph that captures some of the essential parts of what a software development ecosystem looks like.
The image below shows a simplified view of this Knowledge Graph:
Although this view is highly simplified, it conveys the message that there are many, many aspects that all come together in the process of building a software product, each contributing to the quality, compliance, sustainability, robustness, scalability, user-friendliness, cost-efficiency and other aspects of the software.
From Canvas to Survey
Many of the underlying variables that together make up the ecosystem in which a given startup, scaleup or ISV builds software can be measured in terms of maturity. In fact, as a CTO, I continuously measure the maturity of startups and scaleups I work with, in order to then help these companies in ‘scaling up’ their team, process, technological know-how, governance and Native Cloud fit.
In my recent efforts to translate the canvas to a repeatable, somewhat objective online experience that would help myself and others to assess companies in terms of tech maturity, I have decided to craft a survey, with multiple-choice questions in these 6 categories:
- Teams & Process
- Architecture & Data
- Security, Privacy & Compliance
- Non-functional Requirements
- Cost Management
- Governance
How it works
The survey is taken online using any modern web browser. The survey content is stored in JSON format and contains a number of questions in the 6 groups, each annotated with metadata that is used to score and weigh the answers according to the stage a company is in:
- Ideation
- PoC/Prototype
- MVP
- Scaleup
- Established ISV
Many of the questions asked will be scored differently depending on the stage you’re in. For instance, not having a dedicated Agile Product Owner is considered to be perfectly okay if you’re not even building your Minimal Viable Product (MVP), so the scoring system will account for that.
Tagging
Many answers are tagged internally with three possible types of tags:
- Bonus points (highlighted good stuff)
- Risks (on which we can provide guidance)
- Conflicts (detected conflicts with other answers — see ‘Reliability Scores’)
Reliability Scores
The system will build a reliability score based on:
- the number of unanswered questions
- the internal conflict detection mechanism
Open Source
The Maturity Survey is in continuous development. Currently, Neerventure is working with Microsoft on a first production version.
Important to note is that the survey is vendor-neutral and as such usable to assess any startup, scaleup and ISV, regardless of which stack, technologies or Cloud Platform they use.
All code is open source. Both survey (JSON) and simple test app (Web) are available on GitHub.
