topic Delta live tables target schema in Data Engineering

Delta live tables target schema

shagun — Thu, 15 Jun 2023 14:20:46 GMT

The first time i run my delta live table pipeline after setup, I get this error on starting it :-

------------------------------------

org.apache.spark.sql.catalyst.parser.ParseException:

Possibly unquoted identifier my-schema-name detected. Please consider quoting it with back-quotes as `my-schema-name`(line 1, pos 34)

== SQL ==

CREATE DATABASE IF NOT EXISTS my-schema-name

----------------------------------^^^

at org.apache.spark.sql.errors.QueryParsingErrors$.unquotedIdentifierError(QueryParsingErrors.scala:382)

at org.apache.spark.sql.catalyst.parser.PostProcessor$.exitErrorIdent(ParseDriver.scala:388)

--------------------------------------------

The pipeline works fine, if i leave the target schema field blank and run the pipeline.

I then update the pipeline with target schema my-schema-name and then successive pipeline updates are successful. But everytime i run the pipeline 1st time after setting it up, I get the above errror.

Re: Delta live tables target schema

Anonymous — Sat, 17 Jun 2023 09:41:14 GMT

Hi @Shagun Bhardwaj

Great to meet you, and thanks for your question!

Let's see if your peers in the community have an answer to your question. Thanks.

Re: Delta live tables target schema

shagun — Tue, 20 Jun 2023 11:41:31 GMT

Hey .. so the problem is fixed now. The issue was use of "-" in target schema name. When i changed it from "my-schema-name" to "my_schema_name" it works fine.

Re: Delta live tables target schema

BenTendo — Wed, 20 Dec 2023 15:18:23 GMT

This still errors on internal databricks spark/python code like

deltaTable.history()

@shagun wrote:
The first time i run my delta live table pipeline after setup, I get this error on starting it :-
------------------------------------
org.apache.spark.sql.catalyst.parser.ParseException:
Possibly unquoted identifier my-schema-name detected. Please consider quoting it with back-quotes as `my-schema-name`(line 1, pos 34)

== SQL ==
CREATE DATABASE IF NOT EXISTS my-schema-name
----------------------------------^^^

at org.apache.spark.sql.errors.QueryParsingErrors$.unquotedIdentifierError(QueryParsingErrors.scala:382)
at org.apache.spark.sql.catalyst.parser.PostProcessor$.exitErrorIdent(ParseDriver.scala:388)
at
--------------------------------------------
The pipeline works fine, if i leave the target schema field blank and run the pipeline.
I then update the pipeline with target schema my-schema-name and then successive pipeline updates are successful. But everytime i run the pipeline 1st time after setting it up, I get the above errror.

Please fix! or just support dashes!