Comment 0 for bug 1520630

Revision history for this message
Andrey Pavlov (apavlov-n) wrote :

Execution of Spark jobs on Spark plugin with swift inputs/outputs ends in "Done with Error" state even if swift username and password are provided.

Error from job execution logs:

Exception in thread "main" org.apache.hadoop.fs.swift.exceptions.SwiftConfigurationException: Missing mandatory configuration option: fs.swift.service.sahara.username

This problem has a workaround - ":" symbol can be added to spark.executor.extraClassPath config parameter during cluster template creation. it fixes reading from swift, but not writing.

The following error occurs if swift selected as an output:

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 7, spark-cluster-spark-worker-001.novalocal): java.io.IOException: Permission denied