IBM Cloud
Published in

IBM Cloud

Emergent behaviour and dashboarding in Instana

Introduction

They say that the game of rugby was invented in 1823 when a young boy, William Webb Ellis, studying at Rugby Public School, “with a fine disregard for the rules of football as played in his time, first took the ball in his arms and ran with it.” (This story is apocryphal and the rules of football were probably not well regarded, in any case. But why let mere facts get in the way of a good story with an excellent lesson?)

The game of basketball was certainly invented in 1891 by James Naismith, who wanted to exercise his students with a strenuous indoor game. He adapted a game from his youth called “Duck on a Rock” and the rest, as they say, is history.

These are examples of emergent behaviour or emergent game play.

Essentially, emergent behaviour in a game is when the game environment is flexible enough to allow players to choose their own directions and challenges beyond those which the designers of the game had in mind. The myriad varieties of card games, all using the same 52 cards, showcase just how flexible that environment is. On the other hand, a checkers board and checkers tokens don’t have the same flexibility, and there are far fewer varieties of checkers. Some types of computer games are explicitly called “sandbox games” where gamers have the flexibility and the tools to find creative ways to fulfill goals — no matter what the original designers may have thought was the best way to do something.

Emergent behaviour in operational tools is the equivalent of this sandbox, where sysadmins, operators, DevOps and Site Reliability Engineers can customize their tools to solve the problems that they have in front of them, not merely the problems the tools were originally designed to solve.

Emergent behaviour — some clever examples

In this article I’ll discuss how IBM Observability by Instana APM encourages emergent behaviour, specifically in the design, development and usage of dashboards.

One of my favourite examples of emergent behaviour comes from a feature called Maintenance Mode in IBM Cloud Pak® for Watson AIOps/Event Manager (previously Netcool Operations Insight/NOI). The designed purpose of this feature is to mark specific computers as “in maintenance”. This means that if a problem occurred during this period, then the system would consider it a legitimate, even expected, issue and therefore not escalate the problem for human attention or try to resolve it automatically. Since the computer was in maintenance, any problems which arose in this period were expected.

My client, a bank, used this mechanism in an ingenious way.

Part of its responsibilities was to track the opening of certain safes. So the bank hooked up the safes to emit a message whenever they were opened, and put the safes in maintenance mode during regular working hours. This meant that the error message “the safe has been opened” would be automatically quashed during working hours and an alarm would be raised out of working hours.

Were there other ways to solve this problem? Of course. But these engineers found it easier to repurpose a tool they were familiar with than to develop a new solution using other tools.

Instana infrastructure dashboard — You can build a diagram of your datacenter/cloud environment with the skillful use of tags and labels.

Another example is that of a public transport company which used IBM monitoring tools, not to track the performance of IT transactions, but rather the speed of escalators in underground stations. They wanted to track anomalies in escalator performance to be able to send maintenance workers to fix the problem.

My colleague Matthew Duggan has used the topology mapping component of Watson AIOps (originally designed to map the relationships between IT components) to map out London Transport and the relationships between Super Mario characters!

My favourite emergent behaviour is extending tools through the use of ChatOps — triggering actions and seeing their results from within Slack, for example. I’ve used this technique to help quite a few clients get more value out of Watson AIOps.

Did someone mention tradeoff?

There are always tradeoffs — a more flexible solution will enable more emergent behaviour, but it will also present you with more options and choices. Sometimes simplicity is best and you’d like the product to just make the decisions for you.

I often find that these tradeoffs come across in four levels of possible behaviour, which I’ll illustrate based on types of game-play:

  • The One True Way — There’s definitely a right way of playing a game. For example, a board game such as Monopoly, Risk or Cataan will always be played the same way. Minor variations based on house-rules are possible (and encouraged) but these will not change the basic game-play.
  • Customizable games — The game supplies you with recognizable game pieces, but you can do what you want with them. You can also mix-and-match between toys coming from different games. Each time you play, you can give different personalities to the characters and have different plots. Still, most people will use ponies and the Triceratops as good guys and leave a T-rex as a recognizable threat.
Many pieces, one game. (by the author’s children)
  • Buildable games — The game supplies you with the building blocks of what you need and you can build anything you like out of them. It might come with instructions, but you’re free to do whatever you like with them. Admit it — you’re thinking of Lego while reading this!
  • Freeform — This is where the game isn’t really a game, but a tool-set to create more games. Think modelling clay, origami paper or 3D printers.

Instana Dashboards

During my work, I’ve seen many operations, SRE, and observability solutions and I think their emergent behaviour capabilities can be broadly described using the aformentioned four levels of capabilities (of course, most solutions will offer a mixture of these levels). Here, for example, are four different ways Instana implements dashboards:

  1. The One True Way — At this level you trust the designers of the product that they’ve come up with a fantastic solution and you merely need to “dot the i’s and cross the t’s” to customize it for your specific environment. For example, in Instana you have default dashboards, carefully curated to be as helpful as possible, which present you with the health of your applications and allow you to drill-down to investigate problems. They show everything from the monitoring Golden Signals through application topology all the way to the code layer. You can filter by things like geography, user type or any other label, but no matter which application you investigate, you’ll still be seeing the same familiar and comfortable layout during your triage and incident analysis.
    If you need to go beyond the ready-made curated dashboards, you have the Analytics dashboard which enables you to create tag-based filtering, grouping, and visualization of metrics on the fly. This allows you the flexibility of building a custom dashboard during your investigation and, simply by bookmarking it, keeping it for future use.
  2. Customizable — At this level you need to go beyond the out-of-the-box curated dashboards and want better visualization than the Analytics dashboard allows by itself. By creating a custom dashboard, based on a combination of any of the metrics Instana collects, you can customize a dedicated and unique dashboard which will present the information needed, tailored to specific requirements. This dashboard might be a deep dive into a specific application or an overall view of multiple applications. All your custom dashboards might have the same look and feel or they might be customized to match the specific roles of the people using them.
  3. Buildable — Besides building dashboards within Instana itself, you can use plugins to build Instana dashboards in external solutions. For example, you can create Instana dashboards in Grafana —leveraging both the richness of Grafana’s visualization capabilities and the capability to build dashboards which mix multiple data-sources in addition to Instana itself.
    IBM’s complementary ARM solution, Turbonomic Application Resource Management for IBM Cloud Paks, can also be used to created dashboards out of Instana data.
  4. Free-form — Instana has a powerful REST API which allows you to query whatever data you want out of the Instana and use it wherever and however you like. Perhaps you want to include some SLA/SLO data in your organizational portal? Perhaps you want to embed some metrics in a daily email/message you send to stakeholder? Maybe you want to automatically load some of the Instana reports into an Excel spreadsheet using a macro? All this and more is possible by accessing the free-form REST API.
Three types of Instana dashboards — Curated/Out of the box, Grafana plugin-based, Customized/home made in Instana

As a sysadmin/operator/SRE, you need to consider the options the solution you’ve chosen presents you with— are the defaults, the curated One True Way offered out-of-the-box, suitable for your usage? How difficult is it to customize and modify the solution to suit your own unique scenario? On the other hand, how much effort do you want to spend on modifying the system? Maybe the defaults are close enough to what you want that it’s easier to just keep using them and spend your time on something else or maybe you do want to get something that is perfect, not simply “close enough”. Another advantage of customization is that the dashboards you create are uniquely “yours” — they are special to you and your needs and therefore somehow subtly better than anything anyone else could come up with.

One of the things I like most about Instana is that each of the clients I’ve encountered has chosen a different answer and is using the Instana dashboards slightly differently — but they’ve each been able to tweak them and are very happy with them.

Conclusion

This article is rather conceptual & theoretical.

I haven’t worked long enough with Instana to see any unexpected emergent behaviour come out, as I have with Watson AIOps/Netcool, but if you have — or you have a dashboard you like — please leave a comment below or reach out to me to share the fun!

I hope to follow up with a more technical article where I demonstrate some dashboarding tips and ideas.

Another place to share knowledge and assets is the IBM AIOps & Management Community. If you’re not familiar with it, give it a look.

For future lessons and articles, follow me here as Robert Barron or as @flyingbarron on Twitter and Linkedin.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store