Execution of Spark jobs on Spark plugin with swift inputs/outputs ends in "Done with Error" state even if swift username and password are provided.
Error from job execution logs:
Exception in thread "main" org.apache.hadoop.fs.swift.exceptions.SwiftConfigurationException: Missing mandatory configuration option: fs.swift.service.sahara.username
This problem has a workaround - ":" symbol can be added to spark.executor.extraClassPath config parameter during cluster template creation. it fixes reading from swift, but not writing.
The following error occurs if swift selected as an output:
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 7, spark-cluster-spark-worker-001.novalocal): java.io.IOException: Permission denied
Execution of Spark jobs on Spark plugin with swift inputs/outputs ends in "Done with Error" state even if swift username and password are provided.
Error from job execution logs:
Exception in thread "main" org.apache. hadoop. fs.swift. exceptions. SwiftConfigurat ionException: Missing mandatory configuration option: fs.swift. service. sahara. username
This problem has a workaround - ":" symbol can be added to spark.executor. extraClassPath config parameter during cluster template creation. it fixes reading from swift, but not writing.
The following error occurs if swift selected as an output:
Exception in thread "main" org.apache. spark.SparkExce ption: Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 7, spark-cluster- spark-worker- 001.novalocal) : java.io. IOException: Permission denied