Databricks Community

Magnus · 11-01-2023

Hi,I'm implementing a DLT pipeline using Auto Loader to ingest json files. The json files contains an array called Items that contains records and two of the fields in the records wasn't part of the original schema, but has been added later. Auto Loa...

Magnus · 07-11-2023

I'm using Auto Loader as part of a Delta Live Tables pipeline to ingest json files, and today it failed with this error message:om.databricks.sql.transaction.tahoe.DeltaAnalysisException: Found invalid character(s) among ' ,;{}()\n\t=' in the column ...

Magnus · 11-23-2022

I'm using Auto Loader in a SQL notebook and I would like to configure file notification mode, but I don't know how to retrieve the client secret of the service principal from Azure Key Vault. Is there any example notebook somewhere? The notebook is p...

Magnus · 11-09-2022

I have a notebook with the code below, where I try to do an upsert into a dimension table and only include one column from the source table. I get an error even though I think the syntax matches what I see in the docs. How can I write this in the cor...

Magnus · 11-28-2023

I used a schema hint in the Auto Loader notebook to explicitly define the two fields that caused the error. Then it worked. Thanks @Kaniz_Fatma !

Magnus · 07-14-2023

I could not get the DeltaTable solution to work in combination with Auto Loader/DLT/Unity Catalog, since it expects a table location and I want the framework to handle that.I also tried withColumnRenamed, but I can't get it to work either. I still ge...

Magnus · 05-12-2023

Hi @Cedric Law Hing Ping,Are there any plans to support input_file_name in Unity Catalog? I'm using Unity Catalog in Delta Live Tables (DLT), which is in preview, and would like to let DLT handle what cluster is used and still be able to use input_f...

Magnus · 11-27-2022

Thanks @Geeta Sai Boddu! It worked when I converted the notebook to Python and used the Secrets utility. I was hoping that it could be done in a SQL notebook, but maybe that's not supported.

Databricks Community

User Stats

User Activity

FIELD_NOT_FOUND when selecting field not part of original schema

Auto Loader fails when reading json element containing space

How to retrieve Auto Loader client secret from Azure Key Vault?

How to specify which columns to use when using DLT APPLY CHANGES INTO

Re: FIELD_NOT_FOUND when selecting field not part of original schema

Re: Auto Loader fails when reading json element containing space

Re: input_file_name() not supported in Unity Catalog

Re: How to retrieve Auto Loader client secret from Azure Key Vault?