API days 2018 summary
Part of our team was at APIdays conference last week. Cécile Hamerel, Grégoire Hébert, Olivier Lenancker, Pascal Borreli, Thomas Thorez and Arnaud Oisel were present to take care of the stand, meeting and engaging conversations with visitors curious about API Platform, Master Classes and Les-Tilleuls.coop or to attend the numerous talks of the venue. Kévin Dunglas presented the Mercure protocol. We even had the chance to meet Caroline Chuong who will join our company in January.
The venue was at the beautiful Beffroi de Montrouge, south of Paris.
The spectrum of subjects was quite broad. It went from ethics in software, API design, specification, security, authentication and management to digital identities, Open API, GraphQL, microservices, event-driven APIs and much more.
There were 4 to 5 tracks a day and 3 workshop rooms so it was impossible to see everything. The main themes we were able to attend were about :
- Consumer First
- REST, Open API and GraphQL
- Event-Driven APIs
- Ethics in software
This subject was first evoked by Paolo Malinverno (Gartner) during his opening talk “The state of the API industry”. He talked about mega trends that APIs allow such as enabling new user interactions, open banking or artificial intelligence APIs, privacy and culture being the background of all previous things. But foremost he stated that API providers are properly speaking “servers” and they have to “serve” clients. So the clients have to tell the API providers what they need and not the opposite. So we have to start a product from the client’s needs and not from the API.
It was developed in the talk given by Ankit Sobti (Postman Inc) named “Effective Consumer-Driven Contracts”. He made the remark that end-to-end tests are not possible as microservices tend to be too complex (dependency hell). So how can we possibly know when the contract between a consumer and a provider is broken? He suggested that the consumer should be setting the contract. The provider then tests that the right contract is given to all the consumers. So they are basically interfaces tests. This kind of tests was also evoked in one of the talk of Phil Sturgeon (more details in the next section). Logic tests should be undertaken by unit tests in each microservice. Tools that can help with that are Pact Foundation and Postman.
REST, Open API and GraphQL
Is GraphQL a replacement for REST or is it complementary? Should we write API documentation or specification? What is the difference?
During his speech “WeWork Open API Specification Workflow”, Phil Sturgeon (WeWork) talked about the workflow his company uses to specify their APIs and good practices we can borrow.
We should not mistake API Documentation for API Specification. According to him API documentation is static whereas API Specification evolves with the code after modifications. API specifications are written with Open API and JSON Schema for example and you can confirm that your code conforms to the specification by contract testing (see previous section). Once the specification is written, it just so happens, almost for free, to render as documentation. For this purpose, he suggested to use the very good tool ReDoc.
He stressed that we should adopt a Design-First approach, which means focusing on writing the API specification before starting anything else. At WeWork they provide the API specification directly in JIRA so that the developer can start coding directly.
He suggested other tools to use:
- Swagger Inspector to test and document legacy APIs
- APIMatic transformer (integrated in Postman) to export your API description to any format, including Open API
- Stoplight.io or Apicur.io : GUIs for Open API
- Speccy.io for linting
For tests, to check that the API corresponds to the Open API, they use JSON Schema. What JSON Schema and Open API are good at :
So they use JSON Schema for data model validation and Open API for service model validation.
JSON Schema can be returned by the API in a “Link” header, linking to the JSON Schema and allowing the client to validate.
We saw two talks about GraphQL
“GraphQL at GitHub: Building a Platform for Everyone” by Nick Van Wiggeren (GitHub), was a retrospective on the journey of GitHub from REST API to GraphQL.
Their first API appeared in 2008, the latest version of their REST API (v3, JSON only) appeared in 2011. They had their first proof of concept with GraphQL in March 2016 and released the final version of their GraphQL API in May 2017. So it was a quite fast transformation.
One of the reason they chose to open a GraphQL API was because they found that their API responses were overflowed by hypermedia links, with only few information at the end of the response body.
And over time if someone needed a new resource, it had to be added to the API, making the response even more overflowed by hypermedia.
APIs always have tradeoffs and GraphQL just offers different ones. It does not replace REST.
They found that GraphQL was good to involve clients to build on top of their API. Nick gave multiple examples of applications that have been built on top of the GraphQL API such as GitHawk.
The public GraphQL API is only a part of their transformation, the GraphQL API powers their REST v3 API, github.com and some other internal tools.
They have around 500M GraphQL requests per day.
About the challenges, they found that updating an existing API to GraphQL is difficult. It’s like adding types to an untyped code. Building the right API for multiple different consumers is hard. The GraphQL schema they had to create is quite complicated.
He gave much advice to create a GraphQL API. One of them is to establish best practices early. They also used a bot, GraphQL Doctor, to check that Pull Requests updating the GraphQL schema didn’t make any breaking changes. We should focus on the developer experience, so that they want to use our GraphQL API. They documented breaking changes, making changelogs after new releases of their GraphQL API. They have a GraphQL explorer so that consumers can explore their schema easily. Performance is tricky with GraphQL and we should pay great attention to it. Interoperability is important, for example by having the node id in the REST API responses.
I personally was expecting more insight on the difficulties they encountered. If he had more time I am sure he would have talked about it.
In his talk, “Secure and manage APIs with GraphQL”, Ozair Sheikh (IBM) states that GraphQL switches the complexity from consumers to the provider. It provides an alternative query-based approach, optimized for data-intensive operations.
A GraphQL API is likely to be more expensive than a REST one since a GraphQL transaction may invoke multiple backends. It’s important to implement throttling and rate limits. Having a good GraphQL management policy, to check the security of each query and check that it will not take too much resources. We should detect and reject requests with complex nesting, pre-calculate loads to determine if query will overwhelm backends. GitHub GraphQL API uses a point system to individually calculate cost for different query parameters.
He talked about the monetization of a GraphQL API. Since it might be easier to use for a consumer and more resource intensive, the consumer could be enforced to pay more to be able to use this API.
I wasn’t able to attend the talk “Improving the developer workflow with GitHub’s public GraphQL API” by Brian Douglas (GitHub) but it seemed interesting, if anyone attended, I would be glad to have some feedbacks.
Microservices (& service meshs)
In his speech “Taking A Break From Dodgy Dependencies”, Phil Sturgeon talks about network calls between services. We often hope for the request to happen well. But it might not be the case. A thread on a service node could take a little bit longer than usual and take extra resources. As a result other threads on the same node might fail because of this. The entire node might go down because of this. To counter this, we should at least use timeouts so that requests don’t last too long. He explains how to choose the right timeout based on existing logs. And we should not necessarily use the highest request duration. Then we should use retries to perform multiple times requests if they failed. He also explains the concept of circuit breaker : if we notice that a service is not working, we stop calling it unnecessarily for a while, before retrying later when it’s fixed. This avoids blocking thread, taking too many resources, etc… It is possible to abstract this practice in a service mesh. They basically abstract all interactions with the distributed system network from the service programmer.
He also talked about circuit breaker for the frontend. For example, we could have a “map” button, which calls a “map” service to display a “map” view. But if this service does not respond, we could hide this button to the user, leaving only the “list” view, and display it again later once the service is fixed. I don’t know if this is a really good UX choice but apparently some company use this.
About mesh, I could not attend the presentation of Laurent Doguin called “What a Mesh!” but it seemed pretty interesting.
In his talk, Cesare Pautasso (USI Lugano) asked the question “How do you backup and consistently recover your microservice architecture?”. He explained the BAC theorem which is that when Backing up a microservice architecture, you cannot have both Consistency and Autonomy. You either stop all your services at the same time and have Consistency but no Autonomy. Or you choose Autonomy over Consistency but risk losing data. Lack of Autonomy is less bad than lack of consistency, because you can still retry requests if they fail but you cannot recover lost data.
Another hot topic was event-driven APIs. I could not attend the talk “Event-Driven APIs” from Eric Horesnyi but would be glad to have a recap if someone reading this attended.
A speech of Kévin Dunglas “Deliver API Updates in Real Time with Mercure.rocks” in which he explains Mercure, a protocol allowing to push data updates to web browsers and other HTTP clients. It has similarities with WebSocket, except that WebSocket is not compatible with HTTP/2, does not have built-in authentication, reconnection or state reconciliation mechanism. Mercure has all this built-in. If you are french-speaking you can watch this talk to learn more, otherwise you will need to wait for the API days video to be released.
Fran Mendes talked about “Async API” which is an RFC specification, like Open API but for async/message-driven APIs. It’s not dependant on a specific technology, however some part of the specification depends on the protocol used. Will it support Mercure protocol ?
Ethics in software
The most interesting talks to me were about ethics in software. They were the first and last talks of the conference.
Tariq Krim (Dissident.ai) opened the conference with “Principle for the Slow Web, an ethical approach for building software”. To stress the importance of software nowadays, he reminded us how the US decided to invest a lot of money on the development of Internet during the late twentieth century while Europe was investing on clean diesel. Which should explain partly why Europe is late nowadays in every field, not only on Internet. Software is everywhere nowadays. So software engineers are writing the future (it reminded me a speech of Joel Spolsky, also saying that “developers are writing the script for the future”). So we need to decide which society we want to live in. What do we want? Do we want to be governed by technology? He mentioned three important rules according to him : Transparency, No Manipulation and Privacy.
Stéphane Bortzmeyer (AFNIC) talked to us about “Internet and human rights”. We often think that we work on practical stuff and politics does not concern us. Stéphane stated that what affects the internet will affect everyone, because we use internet everyday for everything, quoting the famous Uncle Ben : “With great power comes great responsibility”. Software is behind every activity. So politics of software is important. There is no “purely technical” decisions. He gave the example of the IETF (Internet Engineering Task Force, the standard organisation of Internet protocols) which was for a long time only “technical”, and that created the HRPC group (Human Rights and Protocol Considerations). Their work lead to the RFC 8280, a must-read according to Stéphane.
In practice what does it mean for us ? What should we do ? He gave few examples. Like personal data collection. Eventually every database will leak or has the potential to. So it’s important to minimize the data we collect (only the birth year, not the full birth date for example). He reminded personal data does not have to contain a name necessarily. As soon as there is an ID (like an IP address) that can lead to a single person, it’s personal data. So logging is personal data. Don’t keep logs too long. Do you need the full IP address or can you keep only a prefix ? He promoted his book “Cyberstrutures” talking about all that with more details.
My favorite talk was from Jean-Marc Jancovici “Our global IT system, energy saver or climate destroyer?”. He started with the fact that most people do not know what energy is. For most of people, energy is an electricity or gas bill. But it is not, it is what drives every physical change. And everything that we use nowadays comes from a physical change. The chairs we are sitting on, the computer we use, clothes we wear, … everything. The production of our way of living, of everything that we use in our everyday life, is expressed by the GDP. And this production is driven by machines which are fed with energy. That energy mainly comes from coal, oil and gas. Therefore we expect a correlation between the GDP and oil production and that is what is actually observed. So given the amount of CO2 emitted by coal and oil, currently it is not possible to have a GDP growth and a CO2 emissions limitation.
Jean-Marc Jancovici is more than skeptical about computers. They are based on rare earths which we cannot recycle given the very small amount present in each device. And given that everything nowadays is driven by computers, it seems to be more a problem than a solution. If we shut down all computing power in a second for example, we could not extract oil or coal anymore, cars would not work anymore, etc… We would be carbon neutral.
The talk was finished. Jean-Marc Jancovici began his speech twenty minutes late. The talk was an hour and a half long instead of fifty minutes. Still very few people left and all remained sit at the end not knowing what to say.
A few questions were asked, like “What should we do now ? Should we quit IT and grow potatoes ?”. He replied : “You should explain to people the problem because most people are not aware of it or think they know but mix up concepts which leads to false conclusions. Also change diet”. Another question was “I have money, where should I invest it ?”. The answer was : “In education, in what brings people closer to the food they can grow, without intermediate machines.”
If you want to know more The Shift Project (a think-tank Jean-Marc is a member of) has released a report last month : Lean ICT (Information and communications technologies): for a digital sobriety.
The conference was broad enough so that you could find a topic you didn’t know about. It could have been a little bit more technical on some topics but I guess that the twenty five minutes format was too short for that. We wished we had more time to change tracks as well. Overall it was a good conference. Thanks to the organizers. The talks on ethics were interesting to remind IT people that progress is interesting but it is important to keep others and our environment in mind while we work. This is an important part of our values at Les-Tilleuls.coop.