Internship Series: Developing Data Stream Processor

DSP’s graphical editor that allows you to preview events as they’re being transformed

Background

As I prepare to enter my final year of college at The University of Southern California, I am absolutely floored by the sheer amount of exciting and innovative tech companies I have had the chance to learn more about throughout my time at university.

Splunk was at the top of my list of companies I wanted to work at this summer. Although I didn’t fully understand the power of the Splunk Platform, I was keen on learning more and working at such a high-growth company.

Thanks to the University Recruiting team, I was matched to the Data Stream Processor (DSP) team, a relatively new group contained within the Splunk Cloud Platform (SCP).

DSP is a stream processing service that takes data from sources such as the Splunk Ingest REST API or the Splunk Forwarders Service, processes it with the help of a variety of functions, and sends data to a Splunk indexer.

The Internship

Throughout my interview process, I was under the impression that I would be working as a back-end Software Engineering Intern. Once my team placement was finalized and I had a chance to speak with my manager, Vidya Govindan, I quickly learned one of the most important lessons about my new team — if Splunk was considered to be a high-growth company, then the DSP team was in hyper-growth mode. Hyper-growth teams create a lot of opportunities and openings for engineers who are open to new challenges.

The rapid growth of DSP meant that even though I was supposed to focus on back-end platform work, I was now also tasked with front-end work to help support the team.

Looking back, I am extremely thankful for the opportunity to work on both ends of the stack as I not only improved my own skills but learned much more about the product as a whole.

Front-End

This was the first time I would be using React as my previous experience with front-end work was just simple JavaScript. Fortunately, a lot of the work I did was around fixing bugs that inhibited user experience — new features were being added every day and so bugs were inevitable. My mentor, Megumi Hora, was also always there to support me and made my overall learning experience much easier. Because of her help, I was able to learn a lot about the codebase and the unique aspects of React in a very short amount of time.

One of the first bugs I fixed was to prevent users from being able to enter invalid DSL into their functions. DSL (Domain-Specific Language), is a proprietary DSP language similar to a C-style programming language that compiles into Streams JSON to create a pipeline. Every function in DSP can be written in DSL and compiled into Streams JSON.

Before my changes, the system would check for a valid DSL only at compile time which would require the user to go back and fix all of the invalid DSL snippets one by one. The new approach highlighted below allows users to receive immediate feedback on the validity of their DSL and in turn, would allow them to fix the issues in a much more time-effective manner.

confirm = () => {
   const { helper } = this.props;
   const { dslMappingObject, expressionInEdit } = this.state;
   const clonedDslObj = cloneDeep(dslMappingObject);
   const alias = expressionInEdit.OUTPUT_FIELD;
   const aggregation = expressionInEdit.AGGREGATION;
   const expression = expressionInEdit.FIELD_EXPRESSION;
   
   try {
      helper.getStatements(expression);
      clonedDslObj[constants.AGGREGATIONS][expressionInEdit.i] = {
         [FIELD_EXPRESSION]: expression,
         [OUTPUT_FIELD]: alias,
         [AGGREGATION]: aggregation,
      };
      this.setState({
         dslMappingObject: clonedDslObj
      });
      const updatedDSL = this.toDSLObject(clonedDslObj);
   } catch (e) {
      this.setState({
         dslParseError: e.message
      });
   }
};

One of the other issues I worked on was integrating a DSL parser to the Aggregation function. The Aggregation function applies one or more functions on a stream of events in a specified time window such as mean, max, or median.

Originally, we used regex to parse these expressions. However, that posed a couple of issues. In the event of an incorrect parse, it would be hard to see where the mistake was without previous knowledge of regex and simply reading other people’s regex is usually a challenge.

matches = expression.match(/^as\((.+?),(?=[^,]*$)\s*”(.+?)”\)$/);

As seen above, regex is not the most intuitive approach to parsing the DSL expressions. With the implementation of the parser, we can simply return an object which holds all the attributes of the DSL.

objects = helper.getStatements(dsl);

Now, if there is an issue parsing our expression, we can look into this object and find the exact location of the parse error.

Much of my front-end work was fixing smaller issues within the platform giving me a unique opportunity to help refine the product in the midst of rapid feature addition. I was forced to become quick at debugging and applying simple yet effective fixes to any bugs that popped up.

Back-End

Thanks to my back-end mentor, Eric Prokop, the transition to platform work was relatively smooth. Not only did Eric take the time to walk me through important parts of the codebase but assigned me tickets that would further my understanding of the different components that comprise the application.

My first task was to improve the method for loading group functions into our registry. A group function is composed of other functions. The original implementation required the “inner” functions to be manually sorted in a list as some were required to be loaded before others.

This might be easy to work with when looking at 10–15 functions but as mentioned above, DSP is growing quickly, so the number of group functions will also follow that trajectory as we are able to provide more features to customers.

The new method for loading functions involves us maintaining a second list of functions that have failed at each passthrough. As long as the number of functions failed at each pass is less than the previous, we are successfully loading new functions. However, if the number of failed functions remains the same, then we have hit a wall and there is another issue with loading these group functions at which point we need to throw an error.

register() {
   List<GroupResources> failedResources = Lists.newArrayList();
   while (true) {
      for (GroupResources gr : groupResources) {
         try {…} catch (Exception e) {
            failedResources.add(gr);
         }
      }
   
      if (failedResources.isEmpty()) {
         break;
      }
      if (failedResources.size() == groupResources.size()) {
         List<String> failed = failedResources.stream().map(f ->       f.getMetadata()).collect(Collectors.toList());
         throw error;
      }
   
      groupResources.addAll(failedResources);
   }
}

Although a relatively simple fix, this issue not only helped me gain a better understanding of our product’s offerings but also helped create a foundation for future additions to the platform.

One of the more interesting tickets I had the chance to work on was creating a new scalar function for the platform called “contains-key”. Functions are essential to DSP and at the time of writing, we have over 50 different functions to help users further understand their data. This function checks to see if a given map holds a given key.

public Boolean call(Context context) {
   Map<String, Object> map = context.getArgument(“input”);
   if (map != null) {
      String key = context.getArgument(“key”);
      return map.containsKey(key);
   }
   return false;
}

Writing the function itself is only half the job. I also created tests that ensured the stability of the function with a variety of inputs.

@Test
public void testContainsKeyFunction() {
   Context context = new Context(null);
   context.addArgument(“input”, testMap);
   context.addArgument(“key”, key);
   ContainsKeyFunction containsKeyFunction = new ContainsKeyFunction();
   Boolean actualOutput = containsKeyFunction.call(context);
   
   Assert.assertEquals(actualOutput, expectedOutput);
}
@Test
public void testKeyNotFound() {
   Map<String, String> testMap = ImmutableMap.of(“String key”, “String value”);
   Context context = new Context(null);
   context.addArgument(“input”, testMap);
   context.addArgument(“key”, “missing key”);
   ContainsKeyFunction containsKeyFunction = new ContainsKeyFunction();
   
   Assert.assertFalse(containsKeyFunction.call(context));
}
@Test
public void testMissingMap() {
   Context context = new Context(null);
   context.addArgument(“input”, null);
   context.addArgument(“key”, “foo”);
   ContainsKeyFunction containsKeyFunction = new ContainsKeyFunction();
   
   Assert.assertFalse(containsKeyFunction.call(context));
}

The “contains-key” function, although straightforward, is valuable especially when layering it with some of the other function offerings. As the DSP product continues to grow, the number and complexity of functions will do so as well.

Final Thoughts

I knew an internship at Splunk would be filled with both fun and unique challenges but I could have never imagined how much I would end up loving my time here at this company.

I met some of the most hardworking, intelligent, and kindhearted people I have ever come across. I became closer to both full-time employees and other interns scattered throughout different teams. I was able to learn and contribute to my own growth as an engineer and an independent thinker.

I want to give a special thank you to Vidya, Eric, and Megumi for making this internship one of the best experiences I have had to date. I have some very fond memories that I will carry with me forever.

Raghav Jain is a Full-Stack Software Engineering Intern on Splunk’s Data Stream Processor team. He is a rising senior at The University of Southern California majoring in Computer Science.