Spark FSI tunning

1. GC

For some application jobs, GC is a big problem, like:

recommendation_product_order(application_1584475276480_0002):

the GC time almost 15% of the whole execution time.

I noticed you used G1GC, like:

Used: -XX:+UseG1GC -XX:MaxGCPauseMillis=10000 -XX:ParallelGCThreads=45

and I checked the GClogs in the given folder, like:

it seems just execution logs, different from the common GC logs, like:

you may try to add some arguments like:

-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:/tmp/gc4_6.log -XX:+UseParallelGC -XX:+UseParallelOldGC -XX:NewRatio=1 -XX:SurvivorRatio=8 -XX:ParallelGCThreads=6

/tmp/gc4_6.log is your GC_logs location.

By the way, I tried ParallelGC on our local cluster, it seems better. While it is hard to say it will better for your cluster, you need to get the GC logs or you can try different GC methods.

2. Offheap size or executor size not enough

For recommendation_transaction(application_1584475276480_0011) in stage 2, some tasks failed, like:

Noticed that,

the off-heap size is 38g, but the spark.yarn.executor.memoryOverhead is only 4g.

you need to increase spark.yarn.executor.memoryOverhead size.

3. Data skew and spill

For some application , it has data skew,

it mainly happened in ALS.scala(https://github.com/apache/spark/blob/branch-2.2/mllib/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala) in the spark mllib.

In ALS.scala,

while srcBlockId=”client_id”and dstBlockID=”product_id”, REPARTITION=1120 further bigger than dstBlockID .

That makes data skew happened.

It seems can not be avoided. You may try to increase REPARTITION numbers to check whether it works or not.

4. Storage level

I noticed storage level is MEMORY_AND_DISK，it will spill to spark.local.dir directory.

Cause I don’t find it in the spark-defaults.conf file or init.sql, It maybe use the default dir:/tmp. Or maybe configured in Cloudera, like:

This dir affects the disk location whether it on SSD or it on PEMM. It caused different execution times.

5. Memory useLegacyMode

When running ModelTraining related workload, a sharp drop in memory. Like, when running recommendation_transaction we have 110 executors and each of them have 16GB storage memory .