From PowerPoints to Fancy Web App
And Everything In Between
This past summer, I interned at NETSCOUT Systems, Inc., a leading provider of application and network performance management products. My time there afforded me the perfect opportunity to dig deep into the magical powers of Ruby on Rails.
The problem.
Every Monday, managers from each of NETSCOUT’s software engineering locations hold a “scrum of scrums” meeting to go over their team’s progress that week. Prior to the meetings, they would each spend an hour or so pulling data from Rally, an online agile process management software that the company uses. They would then manually generate graphs and charts in PowerPoint slides, type and format talking points or notes for the slides, send these slides to the other managers through email, and download the other managers’ slides to their machines. A lot of repeated work every week — sounds like it can be automated (D.R.Y.)!
The learning.
When my boss asked me to create a Ruby on Rails web application, I was ecstatic! And clueless. I did know enough Ruby from a Codecademy course, and I did have experience designing and building websites. But Rails? Full-stack development? Forget it. I’ve always been intimidated by anything outside of my front-end “expertise,” if you can even call it that. But, the boss said so, so time to try.
The most important piece of knowledge I gained was not how to Rails, but rather how to learn. I’ll write up a post about getting over my fears of learning later.
The coding.
Newly empowered with Rails, I opened terminal and typed:
rails new rally-app-prototypeThus began the journey of countless hours, days, and weeks spent customizing a dashboard template, figuring out how Highcharts JS works, testing different queries (most of which did not work at all) to different APIs, updating the UI, saving chart data and manager notes into a database, laboriously trying to parse data that I had stored in the databases, fixing a feature that had broken the entire app, updating the UI again, documenting usage and features, deploying the app, realizing I need to implement everything as a background process, overhauling the code, securing and redeploying the app, and fixing more bugs. Fun stuff.
The solution.
What came out of the infinite StackOverflow searches and endless frustrations was 300+ commits to GitHub and a tool that saves managers five man-hours per week. Pats self on back.
Here’s what the homepage looks like:
After managers logged in through HTTP Basic Authentication, they can play around with the tool and start generating reports.
Step 1:
Managers would click on “Generate report” and be greeted by the form on the left
Step 2:
Managers would select the Project and Release (not depicted in this screenshot), which would automatically fill the Iteration choices from a query to Rally’s Web Services API
Step 3:
After the form is submitted, Resque/Redis takes over through background processing
The Implementation.
To actually generate the reports, I had to do quite a bit of processing. First I queried Rally’s Web Services API and Lookback API to get data about the selected iteration and release. Then I ran calculations on the retrieved data, which involved integrating legacy code that the managers had been using and writing new analysis code. This data was all saved to a database (sqlite3 for development and pg for production) as strings (ew). Processing? Done.
To actually show anything meaningful in the report, I had to turn those strings back into arrays in order to feed data into Highcharts JS scripts.
“That’s not that hard,” you say.
That’s what I thought too, until two hours later when I was still trying to parse the strings I was retrieving from my database.
My “refresh_chart_variables” function was filled with line after line of code that parsed different data columns in the database. I must have tried at least 10 different ways of parsing the strings until I got one of the strings to finally parse correctly. The following is a small snippet of what I had to go through to parse just two strings…
# Parse wsi_score array
@wsi_score = @report.wsi_score.tr(‘”[]‘, ‘’).split(“,”).map(&:to_i)# Parse user story names (need "-+-" to escape all valid characters)
@story_name = @report.story_name.split('-+-", "-+-')
@story_name[0] = @story_name[0].sub('["-+-', '')
@story_name[@story_name.length-1] = @story_name[
@story_name.length-1].reverse.sub(']"-+-', '').reverse
The Iteration Burndown Analysis Chart took me quite a few days to get right. I first tried querying the Rally Web Services API for the right pieces of data. Nope: wrong data values. Next I tried web scraping with Ruby’s Mechanize gem. Nope: chart always loaded after Mechanize scraped the page. Then I tried Rally’s Lookback API. Nope: wrong data values again. Finally, by using Rally’s deprecated App SDK LoginKey method, I was able to pull the burndown chart. Whew.
Finally, I caught a fun break playing around with JavaScript and JQuery to implement features such as color selection buttons for the User Story table.
Here’s how a sample report turned out:
The Pitch.
Every product needs a pitch, so naturally, my boss had me create a slide to show the benefits of switching from PowerPoints to the new web app.
I had a lot of fun showing the managers a quick demo of the app’s functionalities and features — I also got great feedback. After the call, I went back to work adding some requested features, securing the app, and prepping it for deployment to the company’s accounts.
The End (not really).
A few of the managers got a chance to test the tool out during their scrum of scrums meeting the following Monday. We actually discovered a bug in the original script (which was used to generate one of the main metrics), and I was able to pinpoint and patch the issue. That was pretty satisfying (but also an arduous process of combing through hundreds of lines of code).
The End (really).
I continued making small changes here and there, never really 100% content with the final product. I’m always thinking about improving little things or adding new features, but I guess I shouldn’t focus on the tiny details too much. (Bugs, on the other hand? I’ll fix those.) Ultimately, this app was a great learning experience and a stepping stone to even bigger projects in my future. I’ve learned a lot through this process, and I’ve listed some of The Lessons at the end of this write-up.
The Failures.
Not every feature is going to be put into production, whether due to bugs, inefficiencies, or lack of necessity. Many of the ideas I had couldn’t be put into the web app, so I thought I’d share some of them here:
- Pagination for the reports table: It would’ve been nice to have each weeks’ reports displayed on its own “page” in the load reports table, but due to usability, I decided to leave them all on the same page and add a filter tool instead. It ended up being more sleek this way, too.
- Pretty PDF exports: Sometimes managers might need to save a PDF of their report to send to colleagues (they can’t send the actual report since the web app is encrypted) or to create a backup. I tried integrating a few Ruby gems, but the PDF generation process was so slow due to the Burndown iframe and Highcharts metrics.
- Rich text editor: When managers are editing their talking points, having a rich text editor might make their reports more organized. However, the other intern at NETSCOUT suggested that it was too “fancy” of a feature for not a lot of use. I went with allowing HTML tags instead.
- Login-independent burndown chart: In the current version of the reporting tool, managers have to be logged in to their Rally accounts (or input their credentials to the burndown chart iframe) since I used a workaround with Rally’s APIKey to display the chart. Using the correct LoginKey method would have costed the company an extra seat on the license — not worth.
The Lessons.
Lesson 1: Always start with clean code. If you’ve using a version control system before, you know that it’s a pain in the butt to go back and edit commits from a week ago. Imagine forgetting to hide credentials from the very first commit. Trying to fix that mistake after 300+ commits? Forget it. I ended up pushing the final application (which used Rails environment variables as placeholders instead of hard coding them into the files) to the company’s GitHub, but it would’ve been nice to have the commit history there as well. Lesson learned.
Lesson 2: Don’t Repeat Yourself (D.R.Y.) — or in this case, Don’t Repeat Others. There are so many open source projects and documentation on using different technologies available online, so try and take advantage of them. It would’ve taken me ages to understand the Rally APIs well enough to write my own Ruby wrapper for it. However, it just so happened that someone on GitHub had already done this for me. Perfect!
Lesson 3: Focus on functionality before optimization. After my demo for the managers, my boss suggested I automatically populate one of the selection fields. I said that when I tried it originally, it had taken a few extra seconds. A coworker replied that I should be thinking about functionality instead of optimization when I’m developing something like this — he said managers wouldn’t care about the extra few seconds if it provided some benefit.
Lesson 4: GitHub is a wonderful tool. So many times I had to revert to a previous commit because I had broken something when I was developing. Total. Life saver. The issue tracking capabilities are another amazing feature not only for collaboration, but even for keeping records of the major fixes that have been made. I spent a few hours going through a Git tutorial, and I posted my notes (Comprehensive GitHub Cheat Sheet)!
Lesson 5: Customer support is another life saver. So many times I was stuck on either a Rally API issue I couldn’t figure out or needed help deploying to and configuring my Heroku app. Every single time I submitted a ticket to their respective support teams, I received a response within a few days.
Lesson 6: You’ll get so much better at something if you put in the time to practice. After the initial code-with-me tutorials and this project, I can now push out a simple app in an evening (check out my Rails Lyrics Analyzer here)! (Disclaimer: initial working prototype was launched in a single evening, but I will still be adding updates and new features!)
