How to run Spark History Server on Windows
At the time these lines are written Apache Spark do not contain an out of the box script to run its History Server on Windows.
The History Server is usually needed when after Spark has finished its work. Many times you need it after running spark-submit. When spark-submit is running, you can monitor spark activity through the active monitor on port 4040. But many times you want to monitor after the fact. For this use-case and many others you’ll need the Spark History Server.
Spark’s Standalone Mode cluster manager also has its own web UI. If an application has logged events over the course of its lifetime, then the Standalone master’s web UI will automatically re-render the application’s UI after the application has finished.
If Spark is run on Mesos or YARN, it is still possible to reconstruct the UI of a finished application through Spark’s history server, provided that the application’s event logs exist.
The default Spark installation comes with built-in scripts: start-history-server.sh and stop-history-server.sh. On Windows you’ll need to run the .cmd files of Spark not .sh.
According to what I saw, there is no .cmd script for Spark history server. So basically it needs to be run manually.
I have followed the history server Linux script and in order to run it manually on Windows you’ll need to take the following steps:
- All history server configurations should be set at the spark-defaults.conf file (remove .template suffix) as described below
- You should go to spark config directory and add the spark.history.* configurations to %SPARK_HOME%/conf/spark-defaults.conf. As follows:
spark.eventLog.enabled true
spark.history.fs.logDirectory file:///c:/logs/dir/path
- After configuration is finished run the following command from %SPARK_HOME%
bin\spark-class.cmd org.apache.spark.deploy.history.HistoryServer
- The output should be something like that:
16/07/22 18:51:23 INFO Utils: Successfully started service on port 18080. 16/07/22 18:51:23 INFO HistoryServer: Started HistoryServer at http://10.0.240.108:18080 16/07/22 18:52:09 INFO ShutdownHookManager: Shutdown hook called
- In order to quit use ctrl+c
Now you have history server working on the following URL(use the URLfrom your command output):
Hope that it helps! :-)
Follow me on:
Medium | Twitter | Linkedin | Stackoverflow | GitHub
