Databricks Loader: Setting config formats.fileFormat = "PARQUET" thows an UnsupportedFileSystemException

[dkbrkjni]

Running the Databricks loader image (snowplow/transformer-kinesis:4.0.0) in AWS Fargate with following configuration:

{
    input = {
        region = ${AwsRegion}
        streamName = ${EnrichedEventsStreamName}
        appName = ${AppName}
    }
    output = {
        region = ${AwsRegion}
        path = "s3://"${S3Bucket}"/events/widerow"
    }
    queue = {
        region = ${AwsRegion}
        type = "sqs"
        queueName = ${EventTransformationQueueName}
    }
    formats = {
        transformationType = "widerow"
        fileFormat = "PARQUET"
    }
    windowing = "5 minutes"
}

Throws following error:

org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "s3"

[pondzix]

Hey @dkbrkjni, thank you for reporting the issue! Indeed, there is a problem with accessing s3 with parquet files. Adding hadoop-aws as a runtime dependency should resolve the issue. This week we're planning to release new version of loader containing the fix!

Note: in transformer output configuration, s3 should be replaced with s3a as this is the schema currently supported by hadoop-aws module.

[dkbrkjni]

Hi pondzix

That sounds great!
Thanks for looking into and fixing this quickly.

[istreeter]

Hi @dkbrkjni we released version 4.0.2 which should fix the problem you were having.

I will close this issue for now, but if you're still having any problems with 4.0.2 then please let us know.

Thank you again for telling us about this issue!