Data Engineering

Forum Posts

Sorted by:

by NathanE • New Contributor II

11-17-2023 1:00:47 AM

1684 Views
1 replies
1 kudos

Time travel on views

Hello,At my company, we design an application to analyze data, and we can do so on top of external databases such as Databricks. Our application cache some data in-memory and to avoid synchronization issues with the data on Databricks, we rely heavil...

Data Engineering

1684 Views
1 replies
1 kudos

11-17-2023 1:00:47 AM

View Replies

Latest Reply

karthik_p
Esteemed Contributor

11-17-2023 7:00:45 AM

1 kudos

@NathanE As you said, based on below article it may not support currenlty https://docs.databricks.com/en/sql/user/materialized-views.html, but at the same time looks as Materialized View is built on top of table and It is synchronous operation ( when...

1 kudos

11-17-2023 7:00:45 AM

by Chris_sh • New Contributor II

11-15-2023 8:35:51 AM

500 Views
0 replies
0 kudos

Enhancement Request: Add Additional context to DLT MERGE Error Messages.

In Delta Live tables I ran into a situation where the Schema in the source system showed as an integer but the DLT table had already inferred the schema during a prior pull as a string. This caused a merge conflict and the pipeline failed to run.This...

Data Engineering

500 Views
0 replies
0 kudos

11-15-2023 8:35:51 AM

by Rubini_MJ • New Contributor

11-09-2023 12:21:34 AM

8603 Views
1 replies
0 kudos

Resolved! Other memory of the driver is high even in a newly spun cluster

Hi Team Experts, I am experiencing a high memory consumption in the other part in the memory utilization part in the metrics tab. Right now am not running any jobs but still out of 8gb driver memory 6gb is almost full by other and only 1.5 gb is t...

Data Engineering

Databricks Mosaic's grid_polyfill() is taking longer to explode the index when run using PySpark

Pyspark Configuration: pyspark --packages io.delta:delta-core_2.12:2.4.0,org.apache.hadoop:hadoop-aws:3.3.4,io.delta:delta-storage-s3-dynamodb:2.4.0 --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" --conf "spark.sql.catalog.spark...

Data Engineering

Delta Lake

Explode

mosaic

spark

1036 Views
0 replies
0 kudos

11-13-2023 9:05:34 PM

by alj_a • New Contributor III

11-12-2023 10:49:19 AM

2030 Views
2 replies
2 kudos

Resolved! Delta Live Table - not reading the changed record from cloud file

Hi,I am trying to ingest the data from cloudfile to bronze table. DLT is working fist time and loading the data into Bronze table. but when i add new record and change a filed in existing record the DLT pipeline goes success but it should be inserted...

Data Engineering

Databricks Delta Live Table

2030 Views
2 replies
2 kudos

11-12-2023 10:49:19 AM

View Replies

Latest Reply

alj_a
New Contributor III

11-12-2023 7:06:29 PM

2 kudos

Thank you Emil. I tried all the suggestions. .read works fine it picks up the new data or changed data. but my problem is it is bronze table as target. in this case my bronze table has duplicate records. However, let me look at the other options to ...

2 kudos

11-12-2023 7:06:29 PM

1 More Replies

by NathanSundarara • Contributor

11-12-2023 5:48:23 PM

1537 Views
0 replies
0 kudos

Lakehouse federation bringing data from SQL Server

Did any one tried to bring data using the newly announced Lakehouse federation and ingest using DELTA LIVE TABLES? I'm currently testing using Materialized Views. First loaded the full data and now loading last 3 days daily and recomputing using Mate...

Data Engineering

dlt

Lake house federation

1537 Views
0 replies
0 kudos

11-12-2023 5:48:23 PM

by rt-slowth • Contributor

10-09-2023 4:40:46 PM

3233 Views
2 replies
1 kudos

CRAS in @dlt

The Delta Table created as a result of the Dataframe returned by @dlt.create_table is confirmed to be overwritten when checked with the DECREASE HISTORY command.I want this to be handled as a CRAS, or CREATE AS SELECT, but how can I do this in python...

Data Engineering

3233 Views
2 replies
1 kudos

10-09-2023 4:40:46 PM

View Replies

Latest Reply

siddhathPanchal
Contributor

11-10-2023 1:40:06 AM

1 kudos

Hi @rt-slowth You can review this open source code base of Delta to know more about the DeltaTableBuilder's implementation in Python. https://github.com/delta-io/delta/blob/master/python/delta/tables.py

1 kudos

11-10-2023 1:40:06 AM

1 More Replies

by dbx_deltaSharin • New Contributor II

10-10-2023 2:46:23 AM

2054 Views
2 replies
1 kudos

Resolved! Open sharing protocol in Datbricks notebook

Hello,I utilize an Azure Databricks notebook to access Delta Sharing tables, employing the open sharing protocol. I've successfully uploaded the 'config.share' file to dbfs. Upon executing the commands: client = delta_sharing.SharingClient(f"/dbfs/p...

Data Engineering

DELTA SHARING

2054 Views
2 replies
1 kudos

10-10-2023 2:46:23 AM

View Replies

Latest Reply

Manisha_Jena
New Contributor III

11-10-2023 1:12:31 AM

1 kudos

Hi @dbx_deltaSharin, When querying the individual partitions, the files are being read by using an S3 access point location while it is using the actual S3 name when reading the table as a whole. This information is fetched from the table metadata it...

1 kudos

11-10-2023 1:12:31 AM

1 More Replies

by dowdark • New Contributor

10-09-2023 6:07:29 AM

1866 Views
2 replies
0 kudos

UPDATE or DELETE with pipeline that needs to reprocess in DLT

i'm currently trying to replicate a existing pipeline that uses standard RDBMS. No experience in DataBricks at allI have about 4-5 tables (much like dimensions) with different events types and I want to my pipeline output a streaming table as final o...

Data Engineering

1866 Views
2 replies
0 kudos

10-09-2023 6:07:29 AM

View Replies

Latest Reply

Manisha_Jena
New Contributor III

11-10-2023 12:54:00 AM

0 kudos

Hi @dowdark, What is the error that you get when the pipeline tries to update the rows instead of performing an insert? That should give us more info about the problem Please raise an SF case with us with this error and its complete stack trace.

0 kudos

11-10-2023 12:54:00 AM

1 More Replies

by svrdragon • New Contributor

11-04-2023 4:35:05 AM

1643 Views
0 replies
0 kudos

optimizeWrite takes too long

Hi , We have a spark job write data in delta table for last 90 date partition. We have enabled spark.databricks.delta.autoCompact.enabled and delta.autoOptimize.optimizeWrite. Job takes 50 mins to complete. In that logic takes 12 mins and optimizewri...

Data Engineering

1643 Views
0 replies
0 kudos

11-04-2023 4:35:05 AM

by pavlos_skev • New Contributor III

10-26-2023 5:18:25 AM

4866 Views
2 replies
0 kudos

Resolved! Invalid configuration value detected for fs.azure.account.key only when trying to save RDD

Hello,We have encountered a weird issue in our (old) set-up that looks like a bug in the Unity Catalog. The storage account which we are trying to persist is configured via External Volumes.We have a pipeline that gets XML data and stores it in an RD...

Data Engineering

4866 Views
2 replies
0 kudos

10-26-2023 5:18:25 AM

View Replies

Latest Reply

pavlos_skev
New Contributor III

10-31-2023 5:42:00 AM

0 kudos

I will post here what worked resolving this error for us, in case someone else in the future encounters this.It turns out that this error appears in this case, when we were using the below command while the directory 'staging2' already exists. To avo...

0 kudos

10-31-2023 5:42:00 AM

1 More Replies

by Mohammad_Younus • New Contributor

10-26-2023 7:35:31 PM

4096 Views
0 replies
0 kudos

Merge delta tables with data more than 200 million

HI Everyone,Im trying to merge two delta tables who have data more than 200 million in each of them. These tables are properly optimized. But upon running the job, the job is taking a long time to execute and the memory spills are huger (1TB-3TB) rec...

Data Engineering

4096 Views
0 replies
0 kudos

10-26-2023 7:35:31 PM

by Hubert-Dudek • Esteemed Contributor III

10-23-2023 1:29:32 PM

7959 Views
1 replies
1 kudos

The perfect table

Unlock the Power of #Databricks: The Perfect Table in 8 Simple Steps!

Data Engineering

7959 Views
1 replies
1 kudos

10-23-2023 1:29:32 PM

View Replies

Latest Reply

jose_gonzalez
Moderator

10-26-2023 10:24:05 AM

1 kudos

Hi @Hubert-Dudek, Thank you for sharing this great post

1 kudos

10-26-2023 10:24:05 AM

by boriste • New Contributor II

07-28-2023 6:44:44 AM

8637 Views
11 replies
10 kudos

Resolved! Upload to Volume inside unity catalog not possible?

I want to upload a simple csv file to a volume which was created in our unity catalog. We are using secure cluster connectivity and our storage account (metastore) is not publicly accessable. We injected the storage in our vnet. I am getting the fol...

Data Engineering

8637 Views
11 replies
10 kudos

07-28-2023 6:44:44 AM

View Replies

Latest Reply

jeroenvs
New Contributor III

10-09-2023 4:38:07 AM

10 kudos

@Ahdri We are running into the same issue. It took a while to figure out that the error message is related to this limitation. Any updates on when we can expect the limitation to be taken away? We want to secure access to our storage accounts with a ...

10 kudos

10-09-2023 4:38:07 AM

10 More Replies

by harish446 • New Contributor

10-23-2023 4:20:34 AM

1186 Views
1 replies
0 kudos

Can a not null constraint be applied on a identity column

I had a table creation script as follows for example: CREATE TABLE default.test2 ( id BIGINT GENERATED BY DEFAULT AS IDENTITY(), name String)using deltalocation "/mnt/datalake/xxxx" What are the possible ways to apply not n...

Data Engineering

data engineering

Databricks

Delta Lake

Delta tables

spark

1186 Views
1 replies
0 kudos

10-23-2023 4:20:34 AM

View Replies

Latest Reply

Krishnamatta
New Contributor III

10-23-2023 6:59:23 AM

0 kudos

Hi Harish,Here is the documentation for this issuehttps://docs.databricks.com/en/tables/constraints.html

0 kudos

10-23-2023 6:59:23 AM