| 2015-12-11 12:05:17 |
Vitalii Gridnev |
description |
Execution of Spark jobs on Spark plugin with swift inputs/outputs ends in "Done with Error" state even if swift username and password are provided.
Error from job execution logs:
Exception in thread "main" org.apache.hadoop.fs.swift.exceptions.SwiftConfigurationException: Missing mandatory configuration option: fs.swift.service.sahara.username
This problem has a workaround - ":" symbol can be added to spark.executor.extraClassPath config parameter during cluster template creation. it fixes reading from swift, but not writing.
The following error occurs if swift selected as an output:
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 7, spark-cluster-spark-worker-001.novalocal): java.io.IOException: Permission denied |
Execution of Spark jobs on Spark plugin with swift inputs/outputs ends in "Done with Error" state even if swift username and password are provided.
Error from job execution logs:
Exception in thread "main" org.apache.hadoop.fs.swift.exceptions.SwiftConfigurationException: Missing mandatory configuration option: fs.swift.service.sahara.username
This problem has a workaround - ":" symbol can be added to spark.executor.extraClassPath config parameter during cluster template creation. it fixes reading from swift, but not writing. |
|
