Databricks Community

ChristianRRL · 11-21-2024

Hi there,I've used databricks workflows to explicitly create tasks with known input parameters (either user input or default parameters). But I'm wondering, what if I want the output of one task to be a list of specific ID's (e.g. id = [7,8,10,13,27]...

ChristianRRL · 11-18-2024

Title may be self-explanatory. Basically, I'm curious to ask if it's possible (and if so how) to add `_metadata` and `_rescued_data` fields to a view "using json".e.g. %sql CREATE OR REPLACE VIEW entity_view USING json OPTIONS (path="/.../.*json",mu...

ChristianRRL · 11-15-2024

Hi there,I'm aware that nowadays newer runtimes of Databricks support some great features, including primary and foreign key constraints. I'm wondering, if we have clusters that are running older runtime versions, are there Upserting patterns that ha...

ChristianRRL · 11-15-2024

Hi there,I'm trying to understand if there's an easy way to create VIEWS and TABLES (I'm interested in both) *WITH* a provided schema file. For example, I understand that via dataframes I can accomplish this via something like this:df = spark.read.sc...

ChristianRRL · 10-29-2024

Hi there, I ran into a peculiar case and I'm wondering if anyone else has run into this and can offer an explanation. We have a DLT process to pull CSV files from a landing location and insert (append) them into target tables. We have the setting "cl...

ChristianRRL · 11-18-2024

Forgot to add this as reference:JSON Files - Spark 3.5.3 DocumentationThe JSON Files documentation doesn't seem to show any OPTIONS that may enable the functionality that I'm looking for.. But please feel free to correct me if there's a way to achiev...

ChristianRRL · 11-15-2024

This is another great tip. Thank you for sharing!One small follow-up, I looked into this a while back but didn't have luck with it at the time. Regarding variables in Spark SQL, is there a way to initialize a variable outside of the %sql cell? For ex...

ChristianRRL · 11-15-2024

Oh WOW. Although I'm a bit disappointed that Spark SQL doesn't have this functionality built-in, this is an excellent tip!I haven't turned into a udf quite yet, but I have the proper building blocks. For example, below I can extract the raw schema St...

ChristianRRL · 11-01-2024

Hi.. I've tried to respond 3 times already but there seems to be an issue with DBX Community and each time my post shows as successful, and I refresh the page and it looks good... but then I check in later (e.g. 30 mins later) and my post is GONE! .....

ChristianRRL · 10-31-2024

Great feedback!Can you please provide a bit more context or example about which DLT logs to monitor? I tried looking into logs, but likely I'm not digging in the right place and the logs I found were completely overwhelming to dig through.

Databricks Community

User Stats

User Activity

Databricks Workflows - Generate Tasks Programmatically

CREATE view USING json and include _metadata, _rescued_data

PKEY Upserting Pattern With Older Runtimes

Spark SQL: USING JSON to create VIEWS/TABLES with existing schema file

DLT Potential Bug: File Reprocessing Issue with "cloudFiles.allowOverwrites": "true"

Re: CREATE view USING json and include _metadata, _rescued_data

Re: Spark SQL: USING JSON to create VIEWS/TABLES with existing schema file

Re: Spark SQL: USING JSON to create VIEWS/TABLES with existing schema file

Re: DLT Potential Bug: File Reprocessing Issue with "cloudFiles.allowOverwrites": "tr

Re: DLT Potential Bug: File Reprocessing Issue with "cloudFiles.allowOverwrites": "tr