Databricks

JJ_LVS1 · 03-22-2023

Hi, I'm trying to create a calendar dimension including a fiscal year with a fiscal start of April 1. I'm using the fiscalyear library and am setting the start to month 4 but it insists on setting April to month 7.runtime 12.1My code snipet is:start_...

JJ_LVS1 · 02-17-2023

I'm trying to parse incoming stream files in DLT which have variable length records. I'm getting the error:Queries with streaming sources must be executed with writeStream.start();Notebook code@dlt.table ( comment="xAudit Parsed" ) def b_table_pa...

JJ_LVS1 · 02-15-2023

Hi All, I need to install a spark-xml package from a notebook cell (hoping it will work on a DLT cluster). Maven Package: com.databricks:spark-xml_2.12:0.16.0Can anyone help me with the command to install from the notebook cell? Fairly new to all thi...

JJ_LVS1 · 03-23-2023

Thanks for the reply. Not sure why I hadn't thought of doing this. I'll add these changes in and see how it turns out.

JJ_LVS1 · 02-23-2023

I would call this one closed. Don't see a way to make this work given how the spark-xml library functions.

JJ_LVS1 · 02-23-2023

I was just going through this as well and require micro-batch operations. Can't see how this will work with DLT right now so I've switched back to structured streaming. I hope they add this functionality otherwise it limits DLT to more basic strea...

JJ_LVS1 · 02-17-2023

Hi, Have not solved his issue yet. This maven spark-xml library does not install via pip so exploring options to execute the install via databricks cli in a notebook cell (which I don't know how to do so still trying to make it work).

JJ_LVS1 · 02-16-2023

Because I need this on a DLT cluster, there is no way to interface with it using CLI since it's the cluster created at runtime. I think I can run CLI commands within a notebook cell so I'll try to figure out how to do that. Please reply if you know...

Databricks

User Stats

User Activity

FiscalYear Start Period Is not Correct

DLT Notebook Error - Queries with streaming sources must be executed with writeStream.start();

Command to install spark package from notebook cell

Re: FiscalYear Start Period Is not Correct

Re: Command to install spark package from notebook cell

Re: Using foreachBatch within Delta Live Tables framework

Re: Command to install spark package from notebook cell

Re: Command to install spark package from notebook cell