Using worker_threads in Node.js Part 2
Earlier this month, I published “Using worker_threads in Node.js”. Since that time, Node.js 11.7.0 shipped. With Node.js 11.7.0, you no longer need the --experimental-workers flag to use the worker_threads module. So it’s even easier to use worker_threads than it was when I wrote that first article.
Robert Jefe Lindstädt left a comment requesting more sophisticated real-world examples. I’m not sure this is more sophisticated than the prime number example in the previous article. But it is more of a real-world example!
I used to have a website that would solve Six Degrees Of Kevin Bacon queries, but for music rather than movies. Give it two musicians, and it would tell you how to connect them. It did this based on who recorded with whom on particular tracks. One thing I discovered is that there wasn’t a good source for track-by-track recording data. So I had to curate my own. A data dump is at https://github.com/Trott/music-routes-data.
Let’s say you wanted to connect Carrie Brownstein to Michael Jackson:
- Carrie Brownstein played on Wild Flag’s “Black Tiles” with Janet Weiss.
- Janet Weiss played on Bright Eyes’ “Clairaudients (Kill Or Be Killed)” with Nate Walcott.
- Nate Walcott played on Pete Yorn’s “Social Development Dance” with Joey Waronker
- Joey Waronker played on Paul McCartney’s “A Certain Softness” with Paul McCartney
- Paul McCartney played on Michael Jackson’s “The Girl Is Mine” with Michael Jackson
The data is a simple undirected and unweighted graph. The algorithm is a breadth first search from each endpoint. Find all people one step away from Carrie Brownstein. Are any of them Michael Jackson? If not, find all people one step away from Michael Jackson. Are any of them also one step away from Carrie Brownstein? If not, find all people two steps away from Carrie Brownstein. If no overlap yet, find all people two steps away from Michael Jackson. And repeat until you find someone that is in both sets of connections.
I wrote some code to do this:
Unfortunately, it’s single-threaded. This means it will be slow:
$ node index.js
search duration: 4144.005ms
Carrie Brownstein played on "Racehorse" with Janet Weiss
Janet Weiss played on "Clairaudients (Kill Or Be Killed)" with Mike Mogis
Mike Mogis played on "Social Development Dance" with Joey Waronker
Joey Waronker played on "A Certain Softness" with Paul McCartney
Paul McCartney played on "The Girl Is Mine" with Michael Jackson
$ 4 seconds isn’t too bad, I suppose. But it can get a lot longer:
$ node index.js 8876 8992
search duration: 27410.035ms
Derek Holt played on "She's Falling Apart" with Dweezil Zappa
Dweezil Zappa played on "Trouble Every Day" with Steve Vai
Steve Vai played on "Fishing" with John Lydon
John Lydon played on "Bad Baby" with Martin Atkins
Martin Atkins played on "The Bushmaster" with David Yow
David Yow played on "Seasick" with David Wm. Sims
David Wm. Sims played on "Soul Machine" with Jim Kimball
Jim Kimball played on "Now I Agree" with Drew Thomas
Drew Thomas played on "Running Into Walls" with Tony Bono
Tony Bono played on "Insult To Injury" with Joe Cangelosi
Joe Cangelosi played on "Isolation" with Mille Petrozza
Mille Petrozza played on "World Beyond" with Rob Fioretti
$ Climax Blues Band guitarist Derek Holt to German thrash-metal bassist Rob Fioretti? Over 27 seconds! Unacceptable!
To the rescue, worker_threads!
Here’s the main thread code:
And here’s the worker thread code:
As far as worker_threads go, there isn’t anything that wasn’t covered in the previous article. Things do get a bit more complicated with the messaging. But there’s nothing new. Let’s see if we’ve improved performance.
$ node main.js
search duration: 724.048ms
Carrie Brownstein played on "Glass Tambourine" with Mary Timony
Mary Timony played on "All Dressed Up In Dreams" with Stephin Merritt
Stephin Merritt played on "The Dead Only Quickly" with Neil Hannon
Neil Hannon played on "Do They Know It's Christmas?" with Paul McCartney
Paul McCartney played on "The Girl Is Mine" with Michael Jackson
$ Nice! From over 4 seconds to under 724ms. Let’s see how we do on the 27-second route!
$ node main.js 8876 8992
search duration: 2767.005ms
Derek Holt played on "She's Falling Apart" with Dweezil Zappa
Dweezil Zappa played on "Smoke On The Water" with Steve Madaio
Steve Madaio played on "All By Myself" with Jim Keltner
Jim Keltner played on "Couldn't Call It Unexpected No. 4" with Marc Ribot
Marc Ribot played on "Bridge To The Beyond" with Mike Patton
Mike Patton played on "When The Stars Begin To Fall" with Duane Denison
Duane Denison played on "Soul Machine" with Jim Kimball
Jim Kimball played on "Monday's Highs" with Drew Thomas
Drew Thomas played on "William" with Tony Bono
Tony Bono played on "Battle Scars" with Joe Cangelosi
Joe Cangelosi played on "Prevail" with Frank Blackfire
Frank Blackfire played on "World Beyond" with Rob Fioretti
$ Whoa! That’s a roughly tenfold improvement! From over 27 seconds to under 3 seconds! And we’re only using two threads!
This may not quite be the complex example sought by the commenter in the last article. But I hope it is a step in that direction. And there’s always the possibility of a Part 3….
Thanks, Anna Henningsen, for reviewing a draft of this post.
