Databricks

08-16-2022

I tried to load a static table as source to a streaming dlt pipeline. I understand this is not optimum, but it provides the best path toward eventually having a full streaming pipeline. When I do I get the following error:pyspark.sql.utils.Analysis...

08-05-2022

I have a notebook used for a dlt pipeline. The pipeline should perform an extra task if the pipeline is run as a full refresh. Right now, I have to set an extra configuration parameter when I run a full refresh. Is there a way to programmatically...

07-07-2022

From a notebook I can import the log4j logger from cs and write to a log like so:log4jLogger = sc._jvm.org.apache.log4jLOGGER = log4jLogger.LogManager.getLogger(__name__)LOGGER.info("pyspark script logger initialized")But this does not work in a Delt...

06-30-2022

I'm new to spark and Databricks and I'm trying to write a pipeline to take CDC data from a postgres database stored in s3 and ingest it. The file names are numerically ascending unique ids based on datatime (ie20220630-215325970.csv). Right now auto...

06-30-2022

When trying to ingest parquet files with autoloader with the following codedf = (spark .readStream .format("cloudFiles") .option("cloudfiles.format","parquet") .load(filePath))I get the following error:java.lang.UnsupportedOperationException:...

08-10-2022

@Debayan Mukherjee I would like to automate this from within a notebook that is a part of multiple pipelines. Both solutions require knowing the `pipeline_id`. How can I access the pipeline id from within a pipeline run?Reference Links for future ...

08-10-2022

While I can confirm that schema inference is supported in DBR 11.1, it is still not supported in either the DLT Current or Preview runtimes which is where I need it. Womp womp.

07-11-2022

Thank you Kaniz. I have seen that article, but it doesn't address how to write log entries which is what I am trying to do.To record my own log messages would I write directly to `event_log_raw`? What is Databricks doing under the hood?

Databricks

User Stats

User Activity

How to set pipelines.incompatibleViewCheck.enabled = false

Access workflow settings from within a notebook.

How to write log entries from a Delta Live Table pipeline.

How to get autoloader to load files in order

When trying to ingest parquet files with autoloader I get an error stating that schema inference is not supported, but the parquet files have schema data. No inference should be necessary. Is this right?

Re: Access workflow settings from within a notebook.

Re: When trying to ingest parquet files with autoloader I get an error stating that schema inference is not supported, but the parquet files have schema data. No inference should be necessary. Is this right?

Re: How to write log entries from a Delta Live Table pipeline.