Troubleshooting web-based issues can quickly become exercises in patience and extreme outside-the-box thinking, especially when there are multiple technologies, codebases, and partners at play. That’s why any tools that allow us to troubleshoot our case-specific problems can be so important. Here at GumGum, our Ad Server team has created an internal tool called “Badges” that allow various teams here to easily see useful data that is collected or created while requesting and serving our ads.
The badges that we generate for our pages and ads have become so useful, in fact, that it was requested that we allow clients to also be able to use these badges in a limited capacity. Our clients could be anyone from the publishers whose pages we serve ads on, to the advertisers whose ads we serve, and the DSPs (Demand-side platform) we work with in the programmatic ad space.
The desired outcomes were for our internal users that work with our clients to be able to generate a link, and for our clients to go to this link to see a specific ad and its badge. This ad badge would display only a subset of the data that would normally be shown. We would also give clients the ability to select a location from within the ad badge itself to “spoof” their location. However, this link would only be valid for a predetermined amount of time.
In order to achieve this, some updates and additions were needed from both the backend and frontend portions of the team.
The backend basically needs to process 2 types of requests. The first is the request made by our internal customers, i.e., GumGummers, that work with the clients, to generate a forcelink for them. A forcelink is a term we use at GumGum to mean a link used to force an ad. This is a very handy tool because many ads can be served on a website but sometimes we want to troubleshoot a very specific ad. A forcelink allows the user to get the exact ad they want to appear instead of desperately refreshing the page until the ad shows up. The second type of request is the request from the forcelink made by the clients to visualize the forced ad and the badge. To serve our internal customers, we need to provide them an ability to make the request to get the forcelink, and that’s simply an endpoint on the backend they can call. We already know the output of the endpoint is the forcelink, but what is the input and is there anything we should store in the database?
The backend needs to produce a forcelink that is associated with an ad to force. So the input should include the ad ID. And since the ad needs a place to render, a page URL is also needed in the input. The output is pretty simple, i.e. the forcelink. Now do we have anything to store in the database? Before tackling this question, let’s first figure out what the forcelink should look like and what will happen when our external customer clicks the link?
How the forcelink works
When our client clicks the force link, a GET request is sent directly to the backend. With the information of this forcelink, the backend should know what ad to force. Therefore, there must be some identifier as part of the forcelink to make the lookup possible. So do we have to generate an identifier and store it in the database on request of a forcelink? Not necessarily! We can make the identifier by encrypting the ad ID and then decrypt it to get the ad ID with which we can look up the ad to be forced. In fact, we need to encrypt more than the ad ID. We need to also encrypt the page URL as we want to restrict the ad to only render on the given page. Besides, we need to encrypt the expiration time because we want the forcelink to expire after a certain amount of time.
An alternative to encryption is to generate a UUID as the identifier, and store it along with the ad ID, page URL, and expiration time in the database. We prefer the latter for the following reasons:
- We can easily find out how many forcelinks we have issued
- We don’t have to manage expired forcelinks ourselves as the database (we use DynamoDB) can do the cleanup for us
- All identifiers have the same structure whereas in the encryption approach some can be longer than others
- The identifier is totally uninterpretable compared with the encryption approach
- It’s more extensible in that if there are new requirements on the forcelink we can simply add new attributes in the table schema but it will get messier and messier if we use encryption
With this approach, the forcelink looks something like this: https://example.com/sports-today#eaft=693693ef-42dd-4802-bc81-7a1d22aec542, with
eaft standing for external ad force token. Note this parameter is appended as a URL fragment so it’s not sent to the server of the webpage.
We don’t want every forcelink to hit the database, so we can place a caching layer in front of DynamoDB. The cache strategy we use is cache aside which basically has the following pattern:
// for writedb.write(data)cache.write(data)
// for readdata = cache.read(key)if data is present: return datadata = db.read(key)if data is present: cache.write(data)return data
Choosing (and Re-Choosing) the Structure
When first deciding how to limit the information we would display in the client-facing ad badges, a few approaches came to mind quickly, and it became a matter of figuring out which would be the most efficient while retaining the ability to be understood and maintained by other developers quickly and easily. The current method of implementation needed to be considered first in order to best decide how to proceed.
To generate our internal ad badges, we have a factory function that serves as the prototype. Then, for each of our main ad products, we extend that prototype in order to modify and/or add functionality to the ad badge based on how the ad product itself works. The overarching ad badge as well as the product-specific badges each have their own file in our codebase in order to make maintaining the code more organized and human-readable.
The initial thought was to extend each of the product-specific badges in new factory functions for the client-facing versions of each badge. This was relatively quick and easy to set up and would keep the code for the various types of badges very siloed. However, it was quickly evident that the files for these new modules would have very little code because there were very few modifications needed by each specific badge. In reviewing, it seemed like adding the extra handling for the generation of the client-facing versions of the badges and the addition of the new files for each new module was overkill and ended up being too clunky for what we were trying to accomplish.
Instead, we would try setting a new property on the badge objects as they were created to serve as a flag that would signify that the badge should be client-facing. Then, before displaying any of the information or parts of the interface that we would want to omit from client-facing badges, there would be a simple check for this flag on the badge object. Though this technique results in less modular code, it still ended up being an efficient solution since there were only minor modifications needed on the original ad badges.
Sending, Receiving, and Handling New Data
Now that the method for generating the client-facing ad badges had been determined, the next piece of the puzzle to consider was how to handle the necessary data exchange between our frontend and backend. Since the links would be generated by the backend by appending a parameter to the URL, the frontend would need to send the value from that parameter to the backend with the ad requests for that page.
After our backend has received one of these ad requests with the value of the URL parameter included, they could then validate whether client-facing badge data should be sent. If it should, an object containing this data is added within the JSON object that is returned to the frontend as an ad response. The inclusion of this object could then be used to set the flag on new badge objects.
Before any badges are created, there is some processing that needs to be done to the data. We very briefly experimented with keeping this processing the same with the client-facing badge data being used in place of the badge data used for internal users. Very quickly, this proved not ideal and we would need to separate some of our functionality in order to handle this new type of badge data. We also need to check if our badge modules have been included on the page, so that if they are not, we can load a script that will enable our badges to be rendered.
Shiny New Feature: Location Spoofing
One feature of this client-facing badge that had drawn some excitement was the ability for a user to spoof their location according to our location cookie. This functionality had already been created for a standalone page that is used by GumGummers. However, having this exist within the badge would prove much more convenient and would also be more client-friendly.
In order to bring this functionality into the badge, we wanted to be able to utilize on the frontend what was already in place for the standalone page in order to create as little new work for our backend team as possible. This was not 100% achievable since the backend team needed to make updates for us to be able to make AJAX requests to get and set the location cookie values. Essentially, the backend needed to include a bunch of CORS response headers to break the same-origin policy. Two articles that helped us to a great extent are Same-Origin Policy and Cross-Origin Resource Sharing.
After these updates were made, setting the location data only required successfully sending a request. However, parsing the location data after retrieving it proved to be a heftier task. The request to the standalone page does not return any standard type of payload but just the HTML markup for the content of the page. Therefore, the two options for parsing the information needed were to use a series of regular expressions or to assign the HTML to a dummy element and use DOM traversal to extract the necessary content.
In the end, it was actually a combination of both approaches that was the most successful. Regular expressions were useful for isolating the different sections of the page’s content. Then, by assigning these sections to dummy HTML elements (HTML element objects that are created but never added to the DOM), we could more accurately find and store the various location data values.
Now, whereas the updates for the ad badge modules themselves were relatively small, the other necessary updates did touch many parts of our codebase at several points in the ad rendering process. Therefore, thorough testing by our team would be required for all ad products.
To help facilitate this testing, I (Lisa) created a test page that would demo the new functionality. I started by collecting ad responses from our backend for each ad product and storing them in local JSON files. Then, I created a page that would load each ad response, make any necessary modifications to the responses, and render the ads and their ad badges. To make troubleshooting easier, I set up a system to simply specify which ad product or products to load so that product-specific issues could be isolated. This allowed the frontend part of the team to test the updates while the backend finalized the work that was needed for generating and validating the link.
Once the functionality was put in place on the backend to generate links, I was able to make one more useful update to the test page: a live link generator. A page URL could be pasted in along with an ID for an ad, and a link would be created and opened for that page so that testers could use any ad and view it on any page including our actual publishers’ pages.
The external badge gives our clients a fantastic resource to render a specified ad and peek into what’s going on behind the scenes. It’s also a great tool for them to test and debug in that if they make a change they can see the impact immediately without dependency on GumGum. To implement this feature, both the frontend and the backend were pulled in. This project demonstrated good collaboration between them and healthy iterations on software development which is not uncommon to see.
On the backend, we supported the request to generate a forcelink which is passed to the clients for use. We also supported the request from the forcelink itself which forces the ad and collects the data for the frontend to display the badge and render the ad. The forcelink is forged by appending a url fragment/hash that looks like a url parameter with the value being a UUID. Every UUID is associated with and used to look up an ad force entity which contains an ad ID, a page URL, and an expiration timestamp. A database layer and caching layer were used to store these ad force entities.
On the frontend, we began checking the page URL for the parameter appended by backend when the link was generated and would send the value with any ad requests made to them. Then, when receiving ad responses from the backend, we began checking for the inclusion of client-facing badge data. If it is present, we set a flag on the object we create for the ad badge. With that flag set, we omit any sensitive data and include the Location Switcher functionality that lets a user “spoof” the physical location the ad requests are being made from.
We hope you like this article. Don’t forget to check out our other blog posts. Keep in touch!