A search-first mindset

A skill worth pursuing to become a better professional in today’s dynamic IT world.

Has anyone used a given library before?
How are others solving a specific situation?
Who is an expert on a topic?
Did anyone run into this error?
How may I know more about exceptions being thrown by a service?

If you’re about to reach out to your coworkers with doubts like the ones mentioned above, make sure you answer YES to the following three questions first.

1. Are you aware of the different sources of knowledge with search capabilities at your disposal?

During the first years of my career I remember spending hours downloading log files through FTP, crossing fingers hoping that a given product had a robust forum or mailing list archive, checking out lots of CVS/SVN repositories to run full-text searches on them, and hating corporate team chat tools for not even saving conversations. Luckily, the situation has changed a lot.

Nowadays Stack Overflow, team collaboration products like Slack, development platforms like GitHub, and log management solutions like ELK, Splunk or AWS Cloudwatch exist. And they all have one thing in common: powerful search capabilities. In spite of this many times when being asked something by fellow developers, I keep replying with a link obtained by running a query in any of the products mentioned above (or pointing them to one of the internal documentation sites). That is good as it means that knowledge sharing and collaboration are healthy, but it would be even better if the ones asking the questions had the skill to reach the answer by running the query on their own in the first place, right?

So come on, it’s time to fight laziness, be curious, and start paying more attention to those search boxes that appear at the top of your screen. They’re there for a reason.

2. Are you familiar with the different search syntaxes and capabilities?

Once you have access to a search component, it’s essential to take some time for getting familiar with its syntax, features, and limitations too. You need to put effort into this, it’s not immediate, so please don’t get frustrated if you don’t get the desired results the first time. Searches can be a bit tricky.

Some basic initial recommendations on this item:

• Go over the official documentation sites. When doing so, make sure you’re reading the docs for the same version you’re using.
• If running searches at centralized logging solutions, avoid generic queries and ample time ranges as much as possible. Otherwise, cost, response time, and server performance may be affected.
• Pay attention to features such as running queries through APIs or scheduling notifications based on search results. They may be valuable in the future.

3. Did you take some minutes to search for the answer yourself?

Finally, it’s time to say goodbye to your ask-first mindset and dedicate some minutes to run a couple of searches by yourself. Trust me, doing this often will put you on a path to becoming a better professional. Not only will you be able to reach answers faster, by yourself, but you will also start thinking into queries you would have never thought about before!

In case you’re not convinced, here are some of the many situations in which having a search-first mindset helped me recently:

• We deploy microservices through Jenkins jobs that invoke an internal tool for taking care of blue-green deployment among other things. One day all those scripts started failing with the same not very descriptive error message. A search for that message at Github’s web console revealed the source code generating it, and the reason: A URL was no longer reachable from our Jenkins cluster. With that information, the Network Security team was contacted and fixed the root problem in a short time.
• A couple of fellow developers reported HTTP 404 when fetching some artifacts from our private package manager. A search for part of the error message at Slack guided me to some recent reports of that same issue in a NodeJS internal channel, and a pinned item detailing a bad practice when uploading artifacts that was the one causing the error (specifying a personal repository URL in the package.json file).
• A Grafana dashboard monitoring an Apache Kafka cluster notified us of increasing lag for some consumer groups. As we send all our apps logs to ELK, and we have a log statement for each message consumption, I decided to take a look at Kibana. Creating a visualization showing the number of messages consumed grouped by microservice name and thread name helped me understand that we were consuming with a single thread per process (even though we had multi-partition topics and were theoretically creating a consuming thread per partition). The culprit ended up being a Spring Boot starter creating a single thread TaskExecutor that ended up propagating as default one in some services.
• For a couple of days, end-users were reporting sporadic response delays for some actions. As the API Gateway receiving their traffic is also sending its access logs to ELK, I decided to run some queries to understand if it was a recent issue, percentage of requests and number of products affected, etc. The problem seemed to be somewhere between the gateway and our backends, so the authentication solution became a suspect. As security logs were also available at another centralized logging solution, running some searches there by tracing requests from one system to the other one lead me to stack traces showing connectivity issues contacting a node part of the security solution cluster. From there, it was just a quick message to the ones in charge of those servers for fixing the issue.

The goal of this post is not to jeopardize team collaboration due to members running searches in isolation when someone sitting a few meters away may solve those doubts in seconds. On the contrary, I’m expecting to improve team collaboration by making sure the one asking shows respect to other’s time and appetite for learning.

In the following days I’ll be writing a follow-up post focusing on the ones on the other side of a question, those receiving it. Stay tuned.