Observing your API traffic with Tyk, Elasticsearch & Kibana
You have your backend services for ACME.com built and ready to ship. Something is wrong — it’s Black Friday, and the conversion rate seems unusually low, despite what looks like significant increased traffic. The business is breathing down the engineers necks — money is being lost, and customer helpdesk queries have started piling in. Is the problem with the mobile app, our backend services? are we being attacked, are systems becoming overloaded due to the sheer volume of traffic arriving at the site? Where to start debugging?
I’m going to show you how you can gain these valuable insights by simply using Tyk as a transparent proxy to your underlying services, and having Tyk ship your API analytics to Elastic Search for viewing & interrogation in a beautiful Kibana dashboard.
TLDR; Feel free to jump straight into a pre-baked docker-compose https://github.com/asoorm/tyk-elasticsearch-kibana
Or if you are more of a YouTuber, I’ve prepared a walkthrough to accompany this blog post — just scroll to the bottom for the embed.
What are Tyk, Elasticsearch & Kibana (TEK Stack)?
Did I just invent a name for yet another open-source stack???
Tyk open source API Gateway is basically (a reverse proxy or load balancer). Think HA Proxy or NginX, but with a programmable API & purposefully designed for proxying & protecting your API traffic.
Because Tyk has been designed specifically as an API Gateway, your analytics are context aware — in that they are enriched specifically to provide useful information for your API traffic. Which API version is popular? Are my customers still using that deprecated API? Is it safe to retire my API. Can I capture specific information about the request, and store it with the request logs for later grepping? All this, rather than the simple request log output that you would get with your typical application load-balancer.
Out of the box, Tyk can also work with your existing Service Discovery tooling such as Consul or etcd to discover you backend services. It offers active healthchecks, uptime tests, enforced timeouts and circuit breakers, which can take unhealthy services out of the balancer, fire events or webhooks to wake up an engineer when things go really wrong or to trigger scaling events. If you need to customise Tyk for any reason to do something on the request path, you don’t need to learn Lua — Tyk offers built in Python, Go, JavaScript & gRPC support.
We will be working with the gateway https://github.com/TykTechnologies/tyk and the analytics pump https://github.com/TykTechnologies/tyk-pump.
Back to business — the engineer has now been paged, bleary eyed, she opens up her laptop — it’s BlackFriday — and there are literally gigabytes of application logs to cat, grep, awk and cut through. If only there was a simple & efficient way of quickly grepping these metrics?
Elasticsearch is an Open Core search engine, based on Lucene. It provides distributed, full-text search with a RESTful interface and is an analytics database, capable of ingesting and indexing megabytes of data per second.
Kibana lets you build your own visualisations & dashboards based on this Elasticsearch data.
Setting up the TEK Stack
DISCLAIMER:
This is far from a production setup — we are just quickly getting up and running in a local dev environment using Docker.
For our backend service, we are going to use the excellent https://httpbin.org api. HttpBin has several endpoints which allow us to test pretty much everything we need.
# example curl request to httpbin.org/get?foo=bar showing response headers and response bodycurl https://httpbin.org/get\?foo=bar -i
HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Type: application/json
Date: Thu, 09 Jan 2020 08:50:26 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
Content-Length: 232
Connection: keep-alive{
"args": {
"foo": "bar"
},
"headers": {
"Accept": "*/*",
"Host": "httpbin.org",
"User-Agent": "curl/7.67.0"
},
"origin": "151.224.16.210, 151.224.16.210",
"url": "https://httpbin.org/get?foo=bar"
}
We can setup a simple Elasticsearch & Kibana as follows:
docker network create es# Start Elasticsearch
docker run -it --rm --name elasticsearch --network es -p 9200:9200 -p 9300:9300 elasticsearch:6.8.6# Start Kibana
docker run -it --rm --name kibana --network es -p 5601:5601 \
-e ELASTICSEARCH_HOSTS=http://elastic-search:9200 \
docker.elastic.co/kibana/kibana:6.8.6
You should now be able to access your Kibana dashboard from your browser on http://localhost:5601. There is no data in your Elasticsearch database yet though, so let’s setup Tyk and start shipping some request logs.
In order to setup Tyk to proxy to our httpbin service, we need a Redis database. Redis is used to store those temporary analytics, api keys & cluster configurations. We will also need Tyk Gateway and Tyk Pump, all of which are available on Dockerhub.
Tyk Pump is a powerful analytics shipper, which can not only send analytics to Elasticsearch. Tyk Pump also has native integrations with Prometheus, StatsD, Mongo, DataDog, Logz.io, Influx, Kafka— even a CSV file to mention a few. https://github.com/TykTechnologies/tyk-pump. If you have more complex requirements, you can also send to multiple combinations of data-sinks at the same time.
There are positives & negatives about having a separate analytics pump. The big positive is separation of concerns — and this means that the pump microservice can independently scale & fail without blocking or causing problems for the gateway request flow. The downside is that you have another service to manage & monitor, but hopefully this blog post will show that configuration is a cinch.
We will use the following configs for Tyk Pump — It is also possible to configure the pump with environment variables.
cat pump.json{
"analytics_storage_type": "redis",
"analytics_storage_config": {
"type": "redis",
"host": "redis",
"port": 6379,
"optimisation_max_idle": 100,
"optimisation_max_active": 200,
"enable_cluster": false
},
"purge_delay": 5,
"pumps": {
"elasticsearch": {
"type": "elasticsearch",
"meta": {
"index_name": "tyk_analytics",
"elasticsearch_url": "http://elasticsearch:9200",
"enable_sniffing": false,
"document_type": "tyk_analytics",
"rolling_index": false,
"extended_stats": false,
"version": "6"
}
}
},
"dont_purge_uptime_data": true
}
Now we can start Tyk Pump whilst mounting the pump configurations.
docker network create tyk# create the pump container
docker create -it --network tyk --name pump --rm \
-v $(pwd)/pump.json:/opt/tyk-pump/pump.conf \
tykio/tyk-pump-docker-pub# attach the pump to the es network
docker network connect es pump# start Tyk Pump
docker start pump# follow the logs if we want
docker logs --follow pump[Jan 9 13:20:53] INFO main: Init Pump: Elasticsearch Pump
[Jan 9 13:20:53] INFO main: Starting purge loop @5(s)
[Jan 9 13:20:58] WARN redis: Connection dropped, connecting..
[Jan 9 13:21:51] INFO main: ## Tyk Analytics Pump, v0.8.3 ##
[Jan 9 13:21:51] INFO elasticsearch-pump: Elasticsearch URL: http://elasticsearch:9200
[Jan 9 13:21:51] INFO elasticsearch-pump: Elasticsearch Index: tyk_analytics
[Jan 9 13:21:51] INFO main: Init Pump: Elasticsearch Pump
[Jan 9 13:21:51] INFO main: Starting purge loop @5(s)
[Jan 9 13:21:56] WARN redis: Connection dropped, connecting..
Now that we have the analytics part setup & up and running, all we need to do now is create a Tyk API Definition (the thing that tells Tyk what to do), mount it into the Tyk docker container and start the gateway.
Example API Definition:
cat httpbin.json{
"name": "httpbin",
"slug": "httpbin",
"api_id": "httpbin",
"use_keyless": true,
"version_data": {
"not_versioned": true,
"default_version": "",
"versions": {
"default": {
"name": "default",
"use_extended_paths": true
}
}
},
"proxy": {
"listen_path": "/httpbin/",
"target_url": "http://httpbin.org/",
"strip_listen_path": true
},
"active": true,
"tag_headers": [],
"strip_auth_data": false
}
We save this API definition document into a directory — say apps, then mount it into the gateway.
docker run -it --rm --network tyk --name tyk \
-p 8080:8080 \
-v $(pwd)/apps:/opt/tyk-gateway/apps \
-e TYK_GW_ENABLEANALYTICS=true \
tykio/tyk-gateway:v2.9.2 tyk --conf tyk.standalone.conf[Jan 09 13:53:59] INFO main: Tyk API Gateway v2.9.2<-----snip----->
[Jan 09 13:53:59] INFO Loading API Specification from /opt/tyk-gateway/apps/httpbin.json
[Jan 09 13:53:59] INFO main: Detected 1 APIs
[Jan 09 13:53:59] INFO main: Loading API configurations.
[Jan 09 13:53:59] INFO main: Tracking hostname api_name=httpbin domain=(no host)
[Jan 09 13:53:59] INFO main: Initialising Tyk REST API Endpoints
[Jan 09 13:53:59] INFO main: API bind on custom port:0
[Jan 09 13:53:59] INFO Initializing HealthChecker
[Jan 09 13:53:59] INFO Checking security policy: Open api_id=httpbin api_name=httpbin org_id=
[Jan 09 13:53:59] INFO gateway: API Loaded api_id=httpbin api_name=httpbin org_id= server_name=-- user_id=-- user_ip=--
[Jan 09 13:53:59] INFO host-check-mgr: Loading uptime tests...
[Jan 09 13:53:59] INFO main: Initialised API Definitions
[Jan 09 13:53:59] INFO main: API reload complete
Our app has path based routing enabled, but we could have just as easily enabled host-based routing if we wanted to.
Let’s send a request to the HttpBin service via the gateway, and see what comes back:
curl http://localhost:8080/httpbin/get\?foo=bar -i
HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Type: application/json
Date: Thu, 09 Jan 2020 13:57:52 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Ratelimit-Limit: 0
X-Ratelimit-Remaining: 0
X-Ratelimit-Reset: 0
X-Xss-Protection: 1; mode=block
Content-Length: 272{
"args": {
"foo": "bar"
},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip",
"Host": "httpbin.org",
"User-Agent": "curl/7.67.0"
},
"origin": "172.19.0.1, 151.224.16.210, 172.19.0.1",
"url": "https://httpbin.org/get?foo=bar"
}
Checking the pump logs — we can also see that we have successfully shipped our request analytics to Elasticsearch
[Jan 9 13:57:53] INFO elasticsearch-pump: Writing 1 recordsAnd now, all we need to do is visit the Kibana dashboard and do a little setup. We need to create an index pattern.
An index pattern tells Kibana which Elasticsearch indices contain the data that you want to work with. In our example, we only have one index tyk_analytics.
Once we have created our index pattern, we can; Interactively explore our data in Discover. Analyze our data in charts, tables, gauges, tag clouds, and more in Visualize. Let’s navigate to the Discover page to see our single lonely request.
Let’s send some more requests so that we can get a slightly better & more realistic visualisation
for i in {1..10}; do curl http://localhost:8080/httpbin/get\?foo=bar; done
And maybe force an internal server error or two
curl http://localhost:8080/httpbin/status/500 -i
HTTP/1.1 500 Internal Server Error
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Length: 0
Content-Type: text/html; charset=utf-8
Date: Thu, 09 Jan 2020 14:17:08 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Ratelimit-Limit: 0
X-Ratelimit-Remaining: 0
X-Ratelimit-Reset: 0
X-Xss-Protection: 1; mode=blockThen filter our analytics to hunt down those erroneous responses. We can do this by querying response_code:500 for example.
Querying Elasticsearch for errors which have been generated by Tyk
We can even build out beautiful aggregate visualisations based on the metrics we sent to Elasticsearch.
I really haven’t scratched the surface of this powerful combination of putting an API Gateway in-front of your services, and shipping analytics to Elasticsearch for viewing in Kibana.
I don’t personally have the experience of designing Kibana dashboards, but it doesn’t really look too difficult. So I cheated and pasted in a demo example from the Elastic website for some inspiration & get your creative juices flowing.
Tyk has the ability to inspect payloads, and inject this data into your analytics pipeline, for further analyis later. It’s even possible to record the entire request and response body, and have that indexed by Elasticsearch also.
Further, you can easily tag your requests using a correlation id, user id, order id or other. Using that, it becomes trivial to track down individual operations and debug what went wrong.
Pushing the boat out further, we could grab the ip address of the client, and get Kibana to display us acoordinate map to see where on earth our client requests are coming from.
Querying analytics based on user Context
Let’s attempt one of those slightly more complex & closer to real-life examples to illustrate.
Create a new directory, at the same level as your apps directory called policies . In this directory add the file policies.json with the following contents.
cat policies/policies.json{
"5e189590801287e42a6cf5ce": {
"id": "5e189590801287e42a6cf5ce",
"rate": 100,
"per": 1,
"quota_max": -1,
"quota_renews": -1,
"quota_remaining": 0,
"quota_renewal_rate": 60,
"access_rights": {
"httpbin-jwt": {
"api_id": "httpbin-jwt",
"api_name": "httpbin-jwt",
"versions": [
"default"
]
}
},
"name": "JWT Demo Policy",
"active": true
}
}
The above is a policy definition object — which defines access rights for specified APIs. In the above, we are granting access to the httpbin-jwt api. Note the policy id 5e189590801287e42a6cf5ce as we will use it later in the API definition object.
Create a new API in apps/httpbin-jwt.json as follows:
{
"name": "httpbin-jwt",
"api_id": "httpbin-jwt",
"enable_jwt": true,
"jwt_signing_method": "hmac",
"jwt_source": "Zm9v",
"jwt_identity_base_field": "sub",
"jwt_default_policies": [
"5e189590801287e42a6cf5ce"
],
"version_data": {
"not_versioned": true,
"use_extended_paths": true,
"versions": {
"default": {
"name": "default",
"global_headers": {
"User": "$tyk_context.jwt_claims_sub"
}
}
}
},
"proxy": {
"listen_path": "/httpbin-jwt/",
"target_url": "http://httpbin.org/",
"strip_listen_path": true
},
"active": true,
"enable_context_vars": true,
"tag_headers": ["User"],
"strip_auth_data": true
}This api definition tells Tyk Gateway to protect the underlying service with JWT authentication. This isn’t really a lesson on how to use Tyk, but to give a few highlights.
When the gateway receives the request to /httpbin-json/, it identifies the API is protected with JWT auth, using a HMAC shared secret signing algorithm. I just set the shared secret to Zm9vwhich is the base64 encoded version of the word foo .
echo -n foo | base64
Zm9vWe identify the subject from the jwt sub claim. and Identify the default policy id to apply as 5e189590801287e42a6cf5ce the same as the policy id from policies.json.
We changed a few more core settings for the API.
"enable_context_vars": true,
"tag_headers": ["User"],
"strip_auth_data": trueBy enabling context variables, it allows us to use some templating magic to pull information out of the request context and inject it into the header or body for example.
Setting tag headers pulls information from the specified request headers and sends those values into your analytics.
Stripping auth data — is a good idea, because we don’t want the JWT to be leaked to the upstream.
Finally, we can perform a global header injection. We set the User header to the value of the JWT sub claim.
"global_headers": {
"User": "$tyk_context.jwt_claims_sub"
}Let’s restart the gateway, or hot-reload, but make sure that we mount the policies as a volume also.
docker run -it --rm --network tyk --name tyk \
-p 8080:8080 \
-v $(pwd)/apps:/opt/tyk-gateway/apps \
-v $(pwd)/policies/policies.json:/opt/tyk-gateway/policies/policies.json \
-e TYK_GW_ENABLEANALYTICS=true \
tykio/tyk-gateway:v2.9.2 tyk --conf tyk.standalone.conf<--- SNIPPED OUT ALL THE BORING BITS --->
INFO main: Detected 2 APIs
INFO Checking security policy: JWT api_id=httpbin-jwt api_name=httpbin-jwt
INFO Checking security policy: Open api_id=httpbin api_name=httpbin
So now, we can see that our gateway has loaded 2 services, one is the original keyless or Open api. And the other service has had a JWT security policy applied.
If we now head over to JWT.io to generate our JWT — or you can use one I created earlier, and send some requests to the new API:
curl http://localhost:8080/httpbin-jwt/get \
-H 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJmb29AYmFyLmNvbSIsIm5hbWUiOiJBaG1ldCBTb29ybWFsbHkiLCJpYXQiOjE1MTYyMzkwMjJ9.8E-X89YnHe6m_jV4Lmyy487VnkCpzuYlsLDceCz4ScU'{
"args": {},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip",
"Host": "httpbin.org",
"User": "foo@bar.com",
"User-Agent": "curl/7.67.0"
},
"origin": "172.22.0.1, 151.224.16.210, 172.22.0.1",
"url": "https://httpbin.org/get"
}
You will see that my email address foo@bar.com added to the request headers. Tyk pulled that email address out from the sub claim of the JWT. Then it striped the authorization header so it didn’t leak.
But remember, we are tagging the User header — so let’s see what it looks like in our Kibana dashboard.
That’s right — we can now use the power of ElasticSearch to query & filter our analytics based on our user’s email address — even though it was embedded inside a JWT!
We are not just limited to the sub claim of the JWT. If the JWT has any custom claims & scopes, we can pull these out too.
We also can pull out other information from the request context including:
- Information from the request body
- Information about the request path
- IP Address
- Correlation Id
- Cookies
- Headers
Even if you are not planning on using many of the features (authentication, authorization, rate-limiting, cors support, transforms etc) that an API Gateway solution has to offer, there is a lot to be said for keeping your microservices lean and void of as much non-functional boilerplate as possible. You get a LOT of value by simply configuring the gateway as a transparent reverse proxy.
No need to immediately page or bother an engineer, no need to SSH into a production box to pull out those logs — by securing Kibana in the appropriate way, your API product manager now has the tools they need to be able to do some of those preliminary investigations and identify a problem with the client request, or identify potential problems with the underlying services.
If you wish to follow along, and get started with your first TEK stack, I’ve knocked up a quick docker-compose with some code snippets in the README of a git repo for you to get started.
If you prefer to watch videos to follow along:
Have fun!