Invalid field schema option provided-DatabricksDeltaLakeSinkConnector

Bency — Thu, 24 Mar 2022 19:01:42 GMT

I have configured a Delta Lake Sink connector which reads from an AVRO topic and writes to the Delta lake . I have followed the docs and my config looks like below .

{

"name": "dev_test_delta_connector",

"config": {

"topics": "dl_test_avro",

"input.data.format": "AVRO",

"connector.class": "io.confluent.connect.databricks.deltalake.DatabricksDeltaLakeSinkConnector",

"name": "dev_test_delta_connector",

"kafka.auth.mode": "SERVICE_ACCOUNT",

"kafka.service.account.id":"****",

"delta.lake.host.name": "******",

"delta.lake.http.path": "*********",

"delta.lake.database":"dl_test_db",

"delta.lake.token": "*********",

"delta.lake.table.auto.create": "true",

"delta.lake.table.format":"kafka_${topic}",

"staging.bucket.name": "dl-test-bucket",

"s3.region":"eu-west-2",

"staging.s3.access.key.id":"*************",

"staging.s3.secret.access.key":"**********",

"confluent.topic.bootstrap.servers":"****************",

"flush.interval.ms": "100",

"tasks.max": "1"

}

I am expecting the connector to automatically create the table , but it errors out as :

"trace": "org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception.\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:568)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:326)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:228)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:196)\n\tat org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:184)\n\tat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:750)\nCaused by: org.apache.kafka.connect.errors.ConnectException: Invalid field schema option provided\n\tat io.confluent.connect.databricks.deltalake.DatabricksDeltaLakeSinkTask.convertFieldSchemaToTableSchema(DatabricksDeltaLakeSinkTask.java:368)\n\tat io.confluent.connect.databricks.deltalake.DatabricksDeltaLakeSinkTask.autoCreateTable(DatabricksDeltaLakeSinkTask.java:309)\n\tat io.confluent.connect.databricks.deltalake.DatabricksDeltaLakeSinkTask.recordsToPutToS3(DatabricksDeltaLakeSinkTask.java:146)\n\tat io.confluent.connect.databricks.deltalake.DatabricksDeltaLakeSinkTask.put(DatabricksDeltaLakeSinkTask.java:98)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:546)\n\t... 10 more\n"

Any help on this would be appreciated , thanks

Re: Invalid field schema option provided-DatabricksDeltaLakeSinkConnector

Hubert-Dudek — Thu, 24 Mar 2022 19:10:25 GMT

I understand that config is on confluent side (Confluent Databricks AWS Delta Lake Sink Connector).

Common issue is S3 problem (roles). Can you check that anything is saved in S3 staging bucket?

You can also contact confluent support.

Re: Invalid field schema option provided-DatabricksDeltaLakeSinkConnector

Hubert-Dudek — Thu, 24 Mar 2022 19:13:37 GMT

Generally you need to be sure that confluent can connect to S3/Databricks resources (ip/roles) and this message from confluent is not much helpful 🙂

Re: Invalid field schema option provided-DatabricksDeltaLakeSinkConnector

Bency — Thu, 24 Mar 2022 19:23:17 GMT

@Hubert Dudek , Should I be configuring anything with respect to schema in the connector config ?

Because I did successfully stage some data from another topic of a different format(JSON_SR) into delta lake table , but its with AVRO topic that I get this error.

topic Re: Invalid field schema option provided-DatabricksDeltaLakeSinkConnector in Data Engineering

Invalid field schema option provided-DatabricksDeltaLakeSinkConnector

Re: Invalid field schema option provided-DatabricksDeltaLakeSinkConnector

Re: Invalid field schema option provided-DatabricksDeltaLakeSinkConnector

Re: Invalid field schema option provided-DatabricksDeltaLakeSinkConnector