Kubernetes did not work with Apache Spark and How to name your Kubernetes Services.

It was frustrating when spark master couldn’t start and keeps dying.

`java.lang.NumberFormatException: For input string: “tcp://100.68.168.187:8080”
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.parseInt(Integer.java:615)
at scala.collection.immutable.StringLike$class.toInt(StringLike.scala:272)
at scala.collection.immutable.StringOps.toInt(StringOps.scala:29)
at org.apache.spark.deploy.master.MasterArguments.<init>(MasterArguments.scala:45)
at org.apache.spark.deploy.master.Master$.main(Master.scala:1049)
at org.apache.spark.deploy.master.Master.main(Master.scala)
`

This was a lot of suffering seeing a service that worked a week ago isn’t working now.

spark master’s service name is “spark-master”, ( a week ago this was “spark-hdfs”)

After hours of ups and downs of helm charts, I happen to change the service name from “spark-master” back to “spark-hdfs” and updated the chart, with no hope that this would work, but to my surprise, voila it worked!

Hmmm, it worked but why?
I started digging into it, started with Dockerfile, then into shell files, then went looking into the error message itself, it says **at org.apache.spark.deploy.master.MasterArguments.<init>(MasterArguments.scala:45)**, well going into the apache spark’s github page and search for the same Class MasterArguments you’ll get it [here](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/master/MasterArguments.scala#L45), look at it ( at this point of time when I’m writing this blog that line says has this piece of code
` port = System.getenv(“SPARK_MASTER_PORT”).toInt
`)

well next thing I did was, login to the spark master container which failed, open spark-shell, the only way I can use scala, get the Map of System Env Values,

`
import scala.collection.JavaConverters._
val s=System.getenv.asScala
for ( (key,value) <- s){
println(key,value)
}
`

Do the same thing for the spark master container which worked, here’s what I found.

Failed Container | Successful Container
(SPARK_MASTER_PORT_7077_TCP_PROTO,tcp)|(SPARK_HDFS_PORT_7077_TCP_PROTO,tcp)
(SPARK_MASTER_PORT_7077_TCP_PORT,7077)|(SPARK_HDFS_PORT_7077_TCP_PORT,7077)
(SPARK_MASTER_PORT_8080_TCP_PORT,8080)|(SPARK_HDFS_PORT_8080_TCP_PORT,8080)
(SPARK_MASTER_SERVICE_PORT_WEB_SPARK_MASTER,7077)|(SPARK_HDFS_SERVICE_PORT_WEB_SPARK_MASTER_UI,8080)
(SPARK_MASTER_PORT_8080_TCP_ADDR,100.68.168.187)|(SPARK_HDFS_PORT_8080_TCP_ADDR,100.67.122.173)
(SPARK_MASTER_PORT,tcp://100.68.168.187:8080)|(SPARK_HDFS_PORT,tcp://100.67.122.173:8080)
(SPARK_MASTER_PORT_8080_TCP_PROTO,tcp)|(SPARK_HDFS_PORT_8080_TCP_PROTO,tcp)

You see that they are totally different, i.e, SPARK_MASTER_**** and SPARK_HDFS_****
Let’s look back at our service names first one was spark-master second one was spark-hdfs!

Lol, the real culprit was Kubernetes all the way!
Kubernetes generates environment variables in every pod!

***

When a Pod is run on a Node, the kubelet adds a set of environment variables for each active Service. It supports both Docker links compatible variables (see makeLinkVariables) and simpler {SVCNAME}_SERVICE_HOST and {SVCNAME}_SERVICE_PORT variables, where the Service name is upper-cased and dashes are converted to underscores.

***

Link [here](https://kubernetes.io/docs/concepts/services-networking/service/#environment-variables)

So, for our service spark-master, kubernetes would generate an env varibale called SPARK_MASTER_PORT=tcp://100.68.168.187:8080, but in turn SPARK_MASTER_PORT was an internal variable for APACHE SPARK!
It worked for service spark-hdfs, because kubernetes would generate an env variable called
SPARK_HDFS_PORT=tcp://100.68.168.187:8080, which,… is not referenced by APACHE SPARK, so it worked!

P.S: Be careful while naming your kubernetes services.

- Sai Varun Reddy!