Slaying the Serpent
A web developers first experience with Python and back end scripting.
The Problem
Spreadsheets, lots of spreadsheets
Internal worklows of the marketing team at Grundfos Global revolve around a lot of setup and roll out of campaigns to Grundfos sales companies around the world. This setup involves a lot repetitive tasks, as the same campaign needs to be setup in multiple languages.
There is no automated way to do this and it takes hours.
This has been the focus of the development team in marketing here at Grundfos — automate wherever possible.
Of course, web automation means scripting and scripting means back end programming languages. I needed a crash course in the chosen back end language of this office — Python.
Python is a programming language that lets you work quickly and integrate systems more effectively. -python.org
For practice, I decided to automate the json of my Ecademy Dashboard webapp (mentioned in my last post). This would use a combination of Python and Selenium. Selenium is a web driver meant for automated testing of web apps. It gives you full control over automating actions in a browser.
The Solution
Using a web driver to auto-download json data
My Ecademy Dashboard needs json data from 31 different countries, each a separate webpage. The constant problem I’ve had is that I cannot pull this json directly by linking to the webpage without my Ecademy Dashboard being on Grundfos Servers (CORS error, cross-domain).
With Python and Selenium I can automate a script that navigates to each page, then copies and downloads the json locally from each country. Then all I do is update the ../json folder in my Ecademy Dashboard. This still isn’t as nice as 100% real-time data, but it’s super easy to update myself whenever needed.
I would upload a video or images of it working, because it’s cool to watch a program you’ve built execute in your browser, but it includes passwords and usernames so I won’t.
But feel free to checkout the finished product:
http://christopherbabb.com/grundfoscoursestatus/#
I’ve done a few updates to the Ecademy Dashboard, mostly a counter that shows how many courses are active in each country. This took a bit of JS functionality, because it counts every iteration of an active course in each country as the program loops through the JSON, but it wasn’t hard to figure out.
Next Steps
Optimization
Next, I would like to optimize the JS, which is still very heavy. The DRY principle (don’t repeat yourself) could certainly be applied in my JS file. Currently, each country has a separate Fetch function in the JS. It would be best if this Fetch function could loop through an array of countries. I would have to make the JS more dynamic — it would need to fetch json from the countries array, then append dynamically to the correct country from that array.
After this, there is some cool optimization to be done in the Python program, including: executing in a headless browser, modulating the code and even the possibility of multi-threading, where each instance the code downloads json data, could be using it’s own thread on a computer — dramatically increasing performance speed. This is certainly not front end developer territory, but now that it’s opened up to me I’m interested.
