Automating mitmproxy and Improving Hulu’s Build Loading Tool for Roku
By Chandler Underwood, Software Developer
Web-connected applications such as Hulu’s have complex and constant network traffic to a variety of services. A proxy is a tool that sits between a web client and the internet and allows requests to be observed, rerouted, and changed. This is valuable for understanding the network call patterns of an app, as well as changing network requests to satisfy testing requirements.
Because Roku devices do not have a native proxy solution, we needed to configure our app to use mitmproxy, an open-source HTTPS proxy. We then automated this setup for mitmproxy and added it to our open source Roku build loading tool roku-dev-cli. Developers were already using roku-dev-cli to deploy builds and monitor logs, so this extra proxy functionality is convenient for them to use. This benefits Roku developers because:
- The developer doesn’t have to manually copy files and modify their app code every time they want to use a proxy.
- Allows the modification of responses from any service to trigger complex test scenarios.
- Allows the developer to mock APIs that don’t even exist yet and remove dependencies on other teams
Removing some of the barriers to thorough testing such as these allow our developers and QA engineers to focus on quality and deliver better features sooner.
Finding a Proxy
There are very few documented processes for using an HTTPS proxy with Roku apps, so finding one to work was a process of trial and error. Charles Proxy is a tool used by other teams at Hulu so it was the first attempted solution. Charles Proxy failed to decrypt traffic, a problem that other Roku developers had also encountered. An inconspicuous error was being thrown about an unexpected header for all network calls. Finding this error insurmountable, a different solution was required.
Next, we evaluated mitmproxy, an open source command-line driven HTTPS proxy. mitmproxy has a wide range of features, including a web GUI interface to inspect traffic, a dump to create logs of traffic history, and a Python API to write add-ons in the form of automated scripts to modify and redirect requests or responses. One challenge we saw while integrating mitmproxy was certain network calls from the Roku OS or the native video player could not have their HTTPS certificate overridden. This meant that large sections of the app and device would stop working completely while using a proxy. The hosts for these calls had to be identified and then excluded from the proxy so they would be passed through and not decrypted with an incorrect certificate. After verifying mitmproxy could decrypt the traffic coming from the Hulu app, we investigated how we could use these features as part of our regular testing and development flow.
Integration with Build Loader
The setup for mitmproxy with the Hulu app was a multi-step process. It involved starting mitmproxy with several arguments and four different URLs to ignore, then copying a certificate file into your app code, and finally editing the app code to use the new certificate. In order to save developers time, we added this setup to the existing build loading tool roku-dev-cli. The build tool is now able to do everything to support using mitmproxy:
- Copy the mitmproxy certificate into the zip file so it will be deployed with the app.
- Update the app code to use the certificate file that mitmproxy generates. The build tool uses a regex to find all BrightScript calls in the app code to setCertificatesFile and overrides them with the mitmproxy certificate name.
- Start mitmproxy with all required arguments.
- With these tasks complete, all network traffic will be passed through and stripped by mitmproxy. The build loader also has optional arguments to specify test scripts to run with the app.
Observing network calls in real time is helpful in debugging, but being able to intercept and modify any request or response allows developers to test a variety of network scenarios. mitmproxy’s Python add-ons expose a python API that allows developers to programmatically modify any network request/response that passes through the proxy. Here is an example of a test script that picks an error at random to return to the app for authorization calls:
This is a simple but powerful example of a test script for mitmproxy. Other examples include; changing artwork, throttling network connection speed, and validating ad beacons are being sent with all required properties.
Documentation for mitmproxy script add-ons can be found here.
To route the Roku network traffic through a proxy, a laptop has to share its internet connection and forward Roku traffic to a specific port where the proxy is listening. This setup is different across operating systems and is currently a manual process and ongoing pain point of using mitmproxy. Adding an OS agnostic enhancement to roku-dev-cli to verify and set up port forwarding would make this tool a complete solution to HTTPS proxying on Roku Apps.
We have open sourced the automated setup of mitmproxy for Roku apps as part of our roku-dev-cli build loader so any Roku developers can use it. It is hosted on github and detailed setup instructions can be found under the “Network Proxying” section. We’re excited for other Roku developers to adopt our tool, it is extremely convenient as a build-loader and provides a ton of value as a proxy tool. We are also open to accepting feedback in the form of pull requests or issues filed on the roku-dev-cli GitHub repo.
If you’re interested in working on projects like these and powering play at Hulu, see our current job openings here