Seeing is Believing: Spark’s Web UI

A beginner’s guide to understanding Spark’s GUI

What? I didn’t know Spark had a GUI either. Apparently it does, so let’s get to know it better. As a brief introduction, the UI allows us to monitor stages for our Spark applications. This means that instead of imagining how operations break down, we can actually see it. We can see how Spark makes the decomposition of our jobs, stages and tasks.

Things might vary depending on the way we deploy Spark, but usually the driver will launch a web UI. This web UI will run by default on port 4040. The cool thing about this UI, is that we can see things like all the Spark SQL queries, a list of scheduler stages and tasks, memory usage, information about the executors, and some other stuff that might be interesting.

OK so let’s get started. First I made an empty text file and filled it with the lyrics for Eminem’s “My Name Is” song. I did that because I was kinda curious about how many times he says “Slim Shady” and because why not.

[Chorus: Eminem]
Hi, my name is, what? My name is, who?
My name is, chka-chka, Slim Shady
Hi, my name is, huh? My name is, what?
My name is, chka-chka, Slim Shady
Hi, my name is, what? (Excuse me) My name is, who?
My name is, chka-chka, Slim Shady
(Can I have the attention of the class for one second?)
Hi, my name is, huh? My name is, what?
My name is, chka-chka, Slim Shady

[Verse 1: Eminem & Dr. Dre]
Hi, kids, do you like violence? (Yeah, yeah, yeah)
Wanna see me stick nine-inch nails through each one of my eyelids? (Uh-huh)
Wanna copy me and do exactly like I did? (Yeah, yeah)
Try 'cid and get fucked up worse than my life is? (Huh?)
My brain's dead weight, I'm tryna get my head straight
But I can't figure out which Spice Girl I want to impregnate (Oh)
And Dr. Dre said, "Slim Shady, you a basehead" (Uh-uh)
"Then why's your face red? Man, you wasted"
Well, since age 12, I felt like I'm someone else
'Cause I hung my original self from the top bunk with a belt
Got pissed off and ripped Pamela Lee's tits off
And smacked her so hard I knocked her clothes backwards like Kris Kross
I smoke a fat pound of grass, and fall on my ass
Faster than a fat bitch who sat down too fast
Come here, slut; "Shady, wait a minute, that's my girl, dawg"
I don't give a fuck, God sent me to piss the world off

[Chorus: Eminem]
Hi, my name is, what? My name is, who?
My name is, chka-chka, Slim Shady
Hi, my name is, huh? My name is, what?
My name is, chka-chka, Slim Shady
Hi, my name is, what? My name is, who?
My name is, chka-chka, Slim Shady
Hi, my name is, huh? My name is, what?
My name is, chka-chka, Slim Shady

[Verse 2: Eminem]
My English teacher wanted to flunk me in junior high (Shh)
Thanks a lot, next semester I'll be 35
I smacked him in his face with an eraser, chased him with a stapler
And stapled his nuts to a stack of paper (Ow)
Walked in the strip club, had my jacket zipped up
Flashed the bartender, then stuck my dick in the tip cup
Extraterrestrial, running over pedestrians in a spaceship While they're screaming at me, "Let's just be friends"
99 percent of my life, I was lied to
I just found out my mom does more dope than I do (Damn)
I told her I'd grow up to be a famous rapper
Make a record about doin' drugs and name it after her
(Oh, thank you)
You know you blew up when the women rush your stands
And try to touch your hands like some screamin' Usher fans
(Ahh, ahh, ahh)
This guy at White Castle asked for my autograph (Dude, can I get your autograph?)
So I signed it, "Dear Dave, thanks for the support, asshole"

[Chorus: Eminem]
Hi, my name is, huh? My name is, who?
My name is, chka-chka, Slim Shady
Hi, my name is, what? My name is, who?
My name is, chka-chka, Slim Shady
Hi, my name is, huh? My name is, who?
My name is, chka-chka, Slim Shady
Hi, my name is, what? My name is, who?
My name is, chka-chka, Slim Shady

[Verse 3: Eminem]
Stop the tape, this kid needs to be locked away (Get him)
Dr. Dre, don't just stand there, operate
I'm not ready to leave, it's too scary to die (Fuck that)
I'll have to be carried inside the cemetery and buried alive
(Huh, yup)
Am I comin' or goin'? I can barely decide
I just drank a fifth of vodka, dare me to drive? (Go ahead)
All my life I was very deprived
I ain't had a woman in years and my palms are too hairy to hide (Whoops)
Clothes ripped like the Incredible Hulk
I spit when I talk, I'll fuck anything that walks (Come here)
When I was little, I used to get so hungry I would throw fits
How you gonna breastfeed me, Mom? You ain't got no tits
I lay awake and strap myself in the bed
With a bulletproof vest on and shoot myself in the head (Bang)
'Cause I'm steamin' mad (Grr)
And by the way, when you see my dad (Yeah?)
Tell him that I slit his throat in this dream I had

[Chorus: Eminem]
Hi, my name is, what? My name is, who?
My name is, chka-chka, Slim Shady
Hi, my name is, huh? My name is, what?
My name is, chka-chka, Slim Shady
Hi, my name is, who? My name is, huh?
My name is, chka-chka, Slim Shady
Hi, my name is, huh? My name is, who?
My name is, chka-chka, Slim Shady

Then I ran pyspark in my console. I read the file I just created. Then I filtered it by the string “Slim Shady” to see how many times that string shows up in the text file. Then I just count it.

>>> song_lyrics = spark.read.text("./eminem-my_name_is.txt")
>>> filtered = song_lyrics.filter(song_lyrics.value.contains("Slim Shady"))
>>> filtered.count()
17

By going to localhost:4040 we can already see Spark’s UI. As an introductory exploration, let’s check out the DAG for the code we just executed. Lets first see the jobs so we can get a simple understanding of how it works.

Now if we click on Stage 0, we should see how that stage went. The UI show’s us how the stage is composed and details about it. This will come in very handy later when we want to optimize our Spark code.

Concepts like “Exchange” or tasks being executed in parallel will become crucial, but for now the point was to become aware of Spark’s GUI and get to know it a little bit. I would recommend you try doing this exercise with something you’re interested in. Try not to deviate so much from the topic, but make it interesting.