Databricks Community

@szymon_dybczak your solution was crisp.@SuMiT1 since you have mentioned your json is dynamic, get one of your json body into a variable. json_body = df.select("content").take(1).collect(0)then get the schema of the json,schema = schema_of_json(json_...

@Hritik_Moon Try to read the file as delta. path/delta_file_name/- parquet files- delta_log/since you are using spark, use this, spark.read.format("delta").load("path/delta_file_name").Delta internally stores the data as parquet and delta log contain...

@Hritik_Moon 1. check if your data is cached, this you can see in sparkUI > storage tab.2. if it is not cached, try to add a action statement after you cache. eg : df.count(). Data is cached with the first action statement it encounters. Now check in...

This is a great initiative! As a data engineer, I always appreciate learning new optimization strategies. Recently, I stumbled upon Monkey Mart while researching resource-efficient architectures—funny how inspiration comes from unexpected places. Loo...

  You’re running into a Databricks SQL results delivery limit—the UI (and even “Download results”) isn’t meant to stream 1.5M × (id, name, 5,000-double array) back to your browser. That’s why SELECT * “works” on Snowflake’s console but not in the DBS...

Hi @KKo ,The simplest way is to have a parametrized notebook which you can pass a name of your catalog as your parameter. Then you can use that parameter to prepare appropriate SQL statements responsible for creating catalogs/schemas/tables.Alternati...

@elliottatreef Can you try to set the Environment version on the source notebook and then trigger the job?On notebook -> Serverless -> configuration -> Environment version drop down. Then, in your job, making sure it’s assigning to the Serverless com...

It seems the duplicates are caused by the task_change_time from the job_tasks table. Even though the table definition shows task_change_time is the time last time the task was modifed.. But it is capturing different times and it is SCD type 2 table. ...

Give this a shotCreate a view in SQL Server that converts geometry to Well-Known Text before federating:-- Create view in SQL ServerCREATE VIEW dbo.vw_spatial_converted ASSELECTid,location_name,location.STAsText() AS geom_wkt,location.STSrid() AS sri...

@stevewb Driver crash is very generic. We may need to dig deeper here to understand the root cause. Can you raise a support ticket with us? 

Hi @surajitDE ,At the moment, DLT doesn’t support linking existing email notification groups or Teams workflows directly. You can only add individual email addresses in the DLT UI.If you have a group email alias, you can use it as a single address so...