Faster purchasing through software development

How I reduced a 20-minute task to 5 minutes through 60 minutes of work

Mood

One of the most satisfying parts of software development is writing code to solve a problem for which it is entirely unnecessary. At least once a week I look at some very minor inconvenience and think, I could write a script to solve this! Often I come to my senses before I implement these ridiculous solutions. It helps that I have other hobbies to fill the time and distract me from my bad ideas. For example, I cross stitch!

I started cross stitching about a year ago and I absolutely love it! Cross stitch is very relaxing, and the results make great gifts. Many of the patterns I stitch are “off the shelf” from creators on Etsy, but I’ve even gotten into designing custom patterns of my favorite things.

Now I won’t dive too deep into the stimulating world of cross stitch patterns, but I at least need to tell you that all patterns use a standard list of colors identified by numbers from the crafting brand DMC. You might have noticed that the Wikipedia article I linked to is in French. DMC is a French company and their American Wikipedia page is rather sparse. Anyway, I have tried using threads from brands other than DMC, but the color bleeds out of the thread and onto the background. Often patterns consist of several very similar colors that may not render right on a display, so it’s tough to tell exactly what colors to use if I’m not going by the DMC number. It’s a lot easier just to use the DMC colors. Well, easier with the exception of one step: visiting the DMC website.

Did you go to the website? No? Well, go back and do it! I think it’s really important to fully understand and appreciate the experience. Go ahead, I’ll wait…

…

…

I feel like I have to say that I think DMC is a great brand that makes high-quality products. In no way do I want to put them down, but wow, that website is slow. It gives me flashbacks of trying to load that Aicha video on eBaum’s World back in high school. (If you’re paying attention, I bet you can figure out how old I am!)

If you still didn’t want to go to the website, check out this lighthouse report

I started to write a section here about why a company with a great product can end up with a not so great website, but I decided to put those thoughts into a different post since they’re only tangentially related. I’ll link that post here once it’s finished.

The problem: this is going to take a minute

Ok, so the website is very slow. Lots of websites are slow. So why not just put up with it? Well, I did for my first order, which was for 13 different colors of thread. My next project requires a whopping 26 different colors of thread! This means that for each color, I have to:

1. Look at my pattern to see what number I need
2. Search for the color’s number
3. Add the desired amount of that color thread to my cart
4. Check the pattern again to make sure I searched for the right number. Spoiler alert: I didn’t
5. Remove the color from the cart
6. Search again, this time for the right color
7. Repeat

This is a bit of a dramatic description, as I don’t get EVERY color wrong the first time. But the point is still the same: this is going to take a minute. Maybe even twenty minutes.

The solution: Node.js

I figured that there is a good chance that when an item is added to the cart, an API call is being made. If I can find that endpoint, I might be able to replicate the call externally in a Node.js application. I can pass in the numbers of the colors I need, have the program make the calls, and get something back. I’ll get into that later on. But first, I need to find the API call I’m looking for and see if it’s even possible to gather the information I need to call the endpoint outside of the browser.

Made possible by Chrome dev tools

I want to see what the website was doing under the hood when I add a thread to the shopping cart. Fortunately, Chrome developer tools make this easy. I can check out all of the network calls that are happening. Finding the call I want — adding the item to the cart — is more of a manual process.

Like a needle in a haystack

I checked for new network calls that occur after adding thread to the cart. After longer than I care to admit, I was able to glean that the API call I needed was to “https://www.dmc.com/js/ajax/update-panier.php". It turns out that “panier” means “basket” in French. DMC is a French company and the underlying code is written in French, which added another layer of complexity. On the bright side, I now know a bit of French!

Here is the information I need to make the call:

In the console in Chrome developer tools, I created a form with the required form data and submitted it using a POST request to the endpoint:

let addToCartFormData = new FormData();
addToCartFormData.append(“produit_id”, 9003292);
addToCartFormData.append(“produit_ref”, 117S-304);
addToCartFormData.append(“produit_attribut_id”, 9002860);
addToCartFormData.append(“quantite”, 1);
addToCartFormData.append(“action”, “add”);
let url = “https://www.dmc.com/js/ajax/update-panier.php";
await fetch(url, { method: “POST”, body: addToCartFormData });

The result was 200, which meant that this goofy plan might just work! I also confirmed that the item was visible in the cart when I went to the checkout page.

Making the call

With the post request in hand, I left the Chrome browser and started a Node.js app. Here is the gist of what I want to achieve:

1. Start with an array of objects, each one representing a color of thread and a quantity of thread
2. In a loop, add each color to the cart using the API endpoint
3. Once all colors have been added to the cart, display the webpage with the shopping cart so I can check out.

I can’t make the post request outside of the DMC website due to CORS restrictions; I need a browser pointing at the DMC website. Also, properly populating the shopping cart requires session cookies, account credentials, and requests to multiple pages. Throughout this process, the context needs to be preserved between all those steps. If I tried to avoid using a headless browser, I’d be manually handling cookies and other fine details to make sure the items were actually added to the cart, which would be horrible. Puppeteer allows us to avoid all that. The puppeteer package gives us access to a browser which acts totally normally, but is controllable by an automated script.

const puppeteer = require(“puppeteer”);
(async () => {
   const browser = await puppeteer.launch({ headless: false });
   const page = await browser.newPage();
   console.log(“navigating to home page”);
   await page.goto(“https://www.dmc.com/us/");
})();

The cross stitch pattern I’m using requires 2 of each color, so I made a default quantity of 2 to pass with requests. Then, I declare an array of objects called produit_refs (not a typo, it’s French!) Each object represents one color I need to purchase. I’ve included one example of a product reference with a different quantity to showcase how the function could work with varying quantities.

const puppeteer = require(“puppeteer”);
(async () => {
   const browser = await puppeteer.launch({ headless: false });
   const page = await browser.newPage();
   console.log(“navigating to home page”);
   await page.goto(“https://www.dmc.com/us/");
   let default_quanity = 2;
   const produit_refs = [
    { id: “BLANC” },
    { id: 221 },
    { id: 3340 },
    { id: 825 },
    { id: 321, quantity: 1 }
  ];
})();

For each color, I call an addToCart function, passing the id of the color and the quantity desired. I also pass the browser page to the function. This ensures that all POST requests occur on the same site. Once all of the colors are in the cart, I navigate to the cart so I can check out and make the purchase.

const puppeteer = require(“puppeteer”);
(async () => {
  const browser = await puppeteer.launch({ headless: false });
  const page = await browser.newPage();
  console.log(“navigating to home page”);
  await page.goto(“https://www.dmc.com/us/");
  let default_quanity = 2;
  const produit_refs = [
    { id: “BLANC” },
    { id: 221 },
    { id: 3340 },
    { id: 825 },
    { id: 321, quantity: 1 }
  ];
  for (ref of produit_refs) {
    console.log(ref);
    await addToCart(page, ref.id, ref.quantity || default_quanity);
  }
  console.log(“navigating to shopping cart… … …”);
  
  await page.goto(“https://www.dmc.com/us/shopping_cart.html");
})();

The addToCart function

In order to add an item to the cart, I need to recreate the POST request I made in the console of Chrome dev tools.

async function addToCart(page, produit_ref, quantity) {
  // adds color to shopping cart
  console.log(“adding color to cart”);
  const addToCart = await page.evaluate(
  
  async (produit_ref, quantity) => {
    const produit_id = 9003292; // The id for all six-strand threads
    const prod_attribut = // ???
    let url = “https://www.dmc.com/js/ajax/update-panier.php";
    let addToCartFormData = new FormData();
    addToCartFormData.append(“produit_id”, produit_id);
    addToCartFormData.append(“produit_ref”, produit_ref);
    addToCartFormData.append(“produit_attribut_id”, prod_attribut);
    addToCartFormData.append(“quantite”, quantity);
    addToCartFormData.append(“action”, “add”);
    return await fetch(url, { method: “POST”, body: addToCartFormData });
    }, 
    produit_ref,
    quantity
  );
}

I run the code in the browser using page.evaluate. This allows me to make basically the same call I made when I wrote in the Chrome console.

A lot has gone on so far, so let’s recap:

1. This ecommerce website is much too slow for my liking. Lighthouse agrees.
2. I decide I’ll make a simple Node.js app to add multiple items to the cart so I don’t have to add them one by one myself.
3. I use Chrome developer tools to check network requests. I comb through a myriad of requests until I find the one that adds an item to the cart
4. I add an item to the cart through a POST request in developer tools for proof of concept
5. I recreate this post request in a Node.js app using puppeteer to manage a Chrome browser.
6. Once all items are added to the cart, I’ll have a browser window open to the cart so I can check out and make my purchase.

If you’re looking at all of this and wondering why the heck I didn’t just add the thread from the website like a regular person, then strap in because the worst is yet to come.

In the above code, you can see that I have most of the data that I need to complete the form. The product id is the same for all six-strand threads regardless of color. The product ref is the color’s unique number as it is identified externally. I can get this number from the cross stitch pattern I’m using. The product attribute is also unique to a color, but it is not customer-facing. I need a way to find the attribute id for each color I want to add to the cart. Find out how I get the attribute id in part 2 of Faster purchasing through software development!