Survey plan: Developer portal first impression — they either hate it, love it or neglect it.

NOTE: the document is constantly updated plan. Small pivot occurred 10.8.2018. I will include managers to the survey as well to get comparison in results. Furthermore expectations of these two segments are expected to differ, yet both are also expected to be served with same portal .

First 15 seconds determine how developers feel about your precious developer portal (1). They either hate it, love it or neglect it. Builders of developer portals, business people drawing nice customer journeys tend to have an assumption that people are objective rational thinkers. They start from here, select that and then click here and so on until they make a purchase. In fact, our decision making, judgment, attitude, and behavior is influenced by things we are not even aware of such as feelings(10), emotions (17) and mood (11). This survey contains screenshots from various APIs driven Developer portals. Aim is to collect information about developer preferences, what works and what does not work. Results can be used in designing new developer portals, drive more traffic to APIs accessed via more attractive and developer focused API portals. Results also help in understanding what kind of content works in B2D marketing.

During that first 15 seconds developer does not have time to read your API documentation, mission or nice getting started packages. The first seconds goes with intuition and gut feeling. Developer either continues with the portal or leaves and possibly never returns. Mike Stowe (18) advices not to market at all and gives 6 other advices for developer relations:

Do tell us what your product is and how it can help us, but be concise and honest, be courteous, be respectful, be yourself, be a part of the community, and invest in the community, in developers, and in your product.

Good first impression is known to build social relationships between people. Good first impression is important for portals as well. Kahneman identifies halo effect as one of the heuristics affecting our decision making. In halo effect we have “tendency to like or dislike everything about a person — including things you have not observed” (5). The warm emotion we feel toward a developer portal predisposes us to like everything about that portal. Good first impressions tend to positively color later negative impressions and conversely, negative first impressions can negatively color later positive impressions.

How do we make decisions?

Intuition is involved when developer lands on a new developer portal. Intuition is immediate pattern recognition, not magic (13,14). Humans are hardwired for patterns and we have strong tendency to construct patterns since it has been vital in our survival and learning(7,8). Good practices and widely adopted naturally born de facto standards are patterns too. Patterns make learning faster, reduce errors and create feeling of familiarity. That is why standards are suggested to be used on API development as much as possible. What is widely accepted practice, is most likely what developer expects to see.

The dominant theory to have emerged divides decision making into two broad types of processes: intuitive (fast, reflexive, and requiring minimal cognitive resources), and analytical (slow, deliberate, and demanding more conscious effort), and the characteristics of the two systems are now well described (5,12, 13,14).

Kahneman uses different terms in his book Thinking Fast and Slow (5). “System 1” is fast, instinctive and emotional while “System 2” is slower, more deliberative, and more logical.

System 1

• Subconscious values, drives, beliefs that influence our “gut reactions.”
• Jumps to conclusions regarding causality.
• Operates effortlessly.
• Can be wrong but is more often right.
• Influenced by heuristics.

System 2

• Articulates judgments, makes choices, endorses or rationalizes ideas and feelings
• Makes up stories to either confirm or deny those created by system 1 conclusions.
• Requires conscious effort to engage.
• Can be wrong or right depending on how hard it works.
• Examines those heuristics (of system 1) when so inclined.

During the first seconds of evaluating developer portal we are dealing with “System 1” defined by several scholars.

Despite the importance of first impression, portal and developer experience must be a fully functional chain. If one part fails, the whole experience fails. As suggested by Kahneman (5), we each have an “experiencing” self and a “remembering” self. The latter usually takes precedence over the former. That is, I can experience 13 days of vacation bliss but if on the 14th day things go bad I tend to remember the vacation as negative. My memory overrides my experience. Thus developer portal must enable fluent customer paths to success.

While system 1 is crucial in first encounter, the system 2 takes over when developer decides to stay in portal and continue deeper. However, system 1 is present in later phases while developer explores the portal. The decision making can be divided to such extremes, but human decision making involves both processes, and different situations require different approaches. If Kahneman is right, we prefer to stay with intuition (system 1) and thus rely on fast decision making process.

Decisions are made during the customer path multiple times, not just after first 5–15 seconds. You need to convince developer all the time along the path. The shorter and simpler the path to API usage is, the more likely diverting decision is not selected. Occasionally developer decides to leave your developer portal and there’s nothing that could prevent it. Developer might just find out that your APIs do not provide the needed solution to the problem at hand. Since APIs can not be solving everyones problems this situation is more than linkely to happen occasionally.

Laziness is part of being human

Laziness is default feature in humans and a virtue among developers. Kahneman’s theory suggests that people are lazy because system 2 requires efforts and we are prone to avoid efforts (5). Thus we let system 1 to take over especially when we are tired, hungry, or mentally exhausted.

Too much laziness is obviously not good for us, since then we would avoid getting necessary tasks done at all. Certain amount of laziness instead drives innovations. When something has to be done with similar steps multiple times, developer normally creates an application for that instead of doing it manually. Automation of tasks is common among developers and nowadays among common people too. Common people use If This Then That (15) and Zapier (16) to automate hundreds of thousands tasks every day. If we would not be lazy enough, we would just do the repetition of boring tasks again and again. Developers love challenges and solving problems. Creating a smart app which does our work with perfection and speed brings satisfaction.

Laziness applies to learning as well. Developers do want to learn new skills and gain knowledge but without doing it the hard way. Bad or lacking documentation makes learning how system or API works difficult and cumbersome. Hunting down what is the cost structure of given API might be the last step to choose another API. Same applies to utilization of API as well. Portal needs to provide all necessary information needed for developer to go forward. Developer loves your API solution if you can provide working code examples and instant access to API. In brief, the less developer needs to bother analytical and labour intense system 2, the more developer is likely to engage with API.

First positive experience with your API must be fast and rewarding (remember halo effect discussed above). System 1 suppresses ambiguity and doubt by constructing coherent stories from mere scraps of data. We have a bias toward believing. Because our brains are pattern recognition devices we tend to attribute causality where none exists. Providing solid information in initial view of the portal, developer might construct positive image of the service and create halo effect.

Convenience trumps features

Success among developers requires low barriers to entry. Great product with enormous amounts of features is great, but will lose the game if there’s more convenient (with less features or even inferior) solution in the market. In brief, convenience trumps features. This was evident in the case of Java and PHP. Another more recent example is SOAP and REST.

Tough audience — be honest

Developers can be skeptical, dismissive of hyperbole, willing to bench-test any product claim and unhesitatingly open to sharing their opinions with the rest of their community.

“Developers can smell bullshit, and they don’t like fluff.” (9)

“The problem is when these two worlds meet, and you have a marketing person trying to convince a developer through sugar coated, elaborate, and emotion based tactics when the developer just wants the information quickly and in a precise, easy to understand format.”(18)

Who’s behind the survey?

Survey conducted by Jarkko Moilanen, PhD (peerproduction driven ecosystems). Jarkko has conducted several surveys among hacker and 3D printing communities. For the past 6 years Jarkko has focused on API Economy and developer eXperience (DX) is his passion. Jarkko is also igniter and author of API Economy 101 book in Finnish, founder of APIOps (term coined by Jarkko 2015) and API Finland communities, organizer of APIdays Nordic 2016.

Survey structure and platform

In this research empirical focus is on open banking portals. Survey target group is application developers.

Survey has three parts.

• First set of questions is about expectations. What kind of items developers want to see when they enter portal? Before the first actual content related question, there is one background question about the professional role of the participant.
• Second set is empirical material based.
• The last part is background information about respondents. None of the last part questions are mandatory. Reason to put those last is to maximize responses to more important questions.

Survey is run on top of LimeSurvey which is installed to UpCloud sponsored cloud environment.

1st: Expectations

For the purposes of this research I have created 6 themes for developer portal evaluation framework:

Design (first impression and clarity, tone).

Devportals should have the perfect harmony of usability, content and aesthetics and present every aspect of the APIs in a well-structured, understandable way. Developer portals that inspire trust through superior production quality. It is also claimed that portals as well as marketplaces should have context awareness aka different localizations as an example(2). The tone of developer focused portals regarding text and visual design should be aligned with values and expections of the developer scene. It means always having dialogues focused on real-world problem-solving, not product promises. Too much marketing slogans is more likely to push developers away than pull them in again. It’s necessary to flush away clichés than it is with other audiences, because developers are acutely sensitive to insincerity and presumption on the part of marketers. In general it can be said that developer oriented materials are more technical than compared to for example with application enduser materials.

API Business model.

Devportals should not only have innovative business models but also found an innovative way to present them. E.g. API pricing model innovations, new ways to monetise APIs and how these are explained on the portal.

Onboarding (signup, SDK, code examples, live console, getting started, documentation).

Devportals clearly show what their APIs are about, how they work, how developers can start integrating and where they can find resources. API documentation and how they integrate with “try it out”, and sandbox functionality: the heart of developer experience. Stop focusing on your product and what it could do. Instead, think about what you’ve actually done and learned. And then share it. That’s really what it means to be authentic.

Outreach and community.

Devportals with creative solutions or initiatives to show developers that their work is appreciated. Portals with great community sections where developers can share knowledge and build connections.

Decision maker documentation.

Organizations will only use an API if they understand the value of it, that is why it is important to increase the perceived value of an API through appropriate and clear business descriptions. This theme does not apply to developers only. Sometimes decision maker is the developer but sometimes it’s the management above the developer.

As an example “software architect is looking at your widget from a top-down perspective, since he or she is charged with making macro decisions about a software product or platform, including defining its technical standards, coding standards, and the tools and platforms to use and design parameters.The development manager is closer to the ground, supervising the design of specific modules, functionalities and features, with programmers below him or her to do the actual coding.”(6)

Developers have become the new kingmakers. CIO used to make deals about software to use in golf course. Now developers just pick and deploy tools to cloud. ShadowIT was born and took over.(4,6). The book The New Kingmakers: How Developers Conquered the World paints too black and white picture of the decision making by contrasting CIO and developer. In reality the managers above developers are gatekeepers as well as the developers. More often the managers have developer background as well. Thus I can align with the book title. Tools and services used in service development are nowadays build and selected bottom up, not poured down the organization structure from the top.

https://tomwentworth.com/a-primer-on-developer-marketing-47d792d67586

This is the hard part communication-wise. Managers might need more traditional selling and marketing touch and tone, but at the same time rest of the portal should follow “developer tone” which is different. So far I have not been able to find studies iterating the differences, but in empirical tests in meetups, it has been discovered that there is a difference.

Research question: does expectations of manager and developer differ and if so how?

Take a look at your product page. Does it cover the specific technical details of your product, or does it regurgitate phrases that you are trying to rank for on Google or industry analyst-speak?

Post-integration & maintenance support (documentation, forums, email, phone, webinars, lifecycle management, changes management).

Devportals that do an exceptional job in creating trust towards their APIs by clearly, yet innovatively indicating their availability and reliability. Portals that make it easy to maintain an API integration through great release notes and other maintenance support.

The expectations related likert scale questions will be grouped by the themes above; one set of claims / theme.

2nd: Empirical part

In this section focus is on portal design (first theme in evaluation framework). Best developer portals have found the perfect harmony of usability, content and aesthetics and present every aspect of the APIs in a well-structured, understandable way. Developer portals that inspire trust through superior production quality. Developers are comfortable and like the first impression.

Focus on first impression and “feeling”.

Look at each screenshot for 10–15 seconds and answer questions

Portal candidates were selected from Open Banking scene.

• Deutche Bank https://developer.db.com/
• Nordea https://developer.nordeaopenbanking.com/
• OP https://op-developer.fi/
• Santander https://developer.santander.co.uk/sanuk/external/
• Citigroup https://sandbox.developerhub.citi.com/
• BBVA https://www.bbvaapimarket.com/
• BNP Paribas https://bnpparibas.openbankproject.com/
• BGL BNP Paribas https://developer.bgl.lu/
• Barclays https://developer.barclays.com/

Survey contains 8 portal screenshots. Each screenshot contains 5 claims (Likert scale). In addition, link to portal is also provided though you should not need it. Below the screenshot is textfield for what is good/bad in the portal.

Idea is that you look at the portal screenshot for 10–15 seconds and then choose value for the claims.

Claims:

1. I feel like I need to talk to sales people
2. Portal is associated clearly with owner company brand
3. Portal is written by the developers for the developers
4. Based on first impression I know where I have landed
5. I felt inspired by the content (in screenshot)
6. I found call for action
7. I found case studies or links to case studies in the portal

Remember that this is about first impression, not about the content.

3rd: Background questions

1. Gender: male, female, I do not want to answer
2. Location (continent level): Europe, North America, South America, Africa, Asia, Oseania
3. Age: 15 or younger, 16–20, 21–25, 26–30, 31–35, 36 or older
4. Experience in web API development: 1 year or less, 2–3 years, 4–5 years, 6–7 years, 8 or more years
5. Experience in consuming web APIs: 1 year or less, 2–3 years, 4–5 years, 6–7 years, 8 or more years
6. What is your highest education? Less than high school, High school, Some years of college, Bachelor degree, Masters degree, Doctors degree

Timeline

Getting maximum respondents is always a challenge. Now the target group is developers with interest/experience in consuming APIs. How to engage them to survey? How to reach them in the first place?

Here’s the plan and timetable:

• Blog post in which is invitation to participate Sept 15th 2018
• Planned survey run is Oct15th — Nov 1st. The survey is launched after the vacation periods.
• Use Twitter for call for action (link to blog post) with hashtags #developer #research #participate #survey #api #dx #devrel
• Use Zapier: automate tweet weekly for 4 weeks (as long as survey is open)
• Put an invitation to https://news.ycombinator.com/, link to blog post
• Tag evangelists and significant API related people in Twitter. @kinlane (Global API evangelist) is followed by lots of developers (I assume). Also Amancio Bouza, PhD is a good tag. That might help message get through. Although both of them might be followed more by business people than developers. Who are the technical global API world developer heroes? Here’s a Twitter thread containing about a hundred or so.
• Analyze results Nov 1st — Nov 15th.
• Blog about the results Nov 15th 2018.
• Write an academic paper in Jan 2019.

Preliminary research questions

• Does expectations of manager and developer differ and if so how?
• What are preferred onboarding practices?
• What are preferred first steps with new API?
• What are the first things developers and managers expect to find from portal?

Results and open access data

Results will be used in blog posts in platforms which are not decided yet. The survey data might also be used in academic paper. Most likely target academic journal is from marketing field. I would assume API Economy and B2D are not too much touched yet (literary review / check has to be done anyway). Curated list of API Economy articles.

Collected data does not contain personal information, IP addresses or contact information. Raw survey data in open format (csv) will be published in Github (under API Economy Hacklab org) for further use.

Literature

1. https://blog.hubspot.com/marketing/chartbeat-website-engagement-data-nj
2. https://nordicapis.com/case-study-rakuten-rapidapi-is-globalizing-the-api-marketplace/
3. https://martechtoday.com/b2b-developer-marketing-complicated-205934
4. https://tomwentworth.com/a-primer-on-developer-marketing-47d792d67586
5. KAHNEMAN, Daniel; EGAN, Patrick. Thinking, fast and slow. New York: Farrar, Straus and Giroux, 2011.
6. O’Grady, Stephen. The New Kingmakers: How Developers Conquered the World. “ O’Reilly Media, Inc.”, 2013.
7. Beitman, Bernard D. “Brains seek patterns in coincidences.” Psychiatric Annals 39.5 (2009).
8. Ripley, Brian D. Pattern recognition and neural networks. Cambridge university press, 2007.
9. https://www.samjarman.co.nz/dxguide/
10. Lerner, Jennifer S., Seunghee Han, and Dacher Keltner. “Feelings and consumer decision making: Extending the appraisal‐tendency framework.” Journal of consumer psychology 17.3 (2007): 181–187.
11. Johnson, Eric J., and Amos Tversky. “Affect, generalization, and the perception of risk.” Journal of personality and social psychology 45.1 (1983): 20.
12. Croskerry P, Petrie DA, Reilly JB, et al. Deciding About Fast and Slow Decisions. Acad Med. 2014;89(2):197–200.
13. Klein G. The Power of Intuition. New York, NY: Doubleday; 2004.
14. Gigerenzer G. Gut Feelings: The Intelligence of the Unconscious. New York, NY: Viking Penguin; 2007.
15. https://ifttt.com/
16. https://zapier.com/
17. Bechara, Antoine. “The role of emotion in decision-making: evidence from neurological patients with orbitofrontal damage.” Brain and cognition 55.1 (2004): 30–40.
18. My 7 Rules of Developer Marketing http://www.mikestowe.com/blog/2013/12/my-7-rules-of-developer-marketing.php
19. https://innopay.com/themes/apis/openbankingmonitor/
20. https://innopay.com/blog/mastering-open-banking-how-the-masters-in-openness-create-value/