cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
rsamant07
by • New Contributor III
  • 1995 Views
  • 3 replies
  • 2 kudos

Serverless SQL Cluster giving error with Powerbi

Power bu Giving this error while accessing delta table using serverless sql endpoint. Error while using path /mnt/xyz/_delta_log/00000000000000000000.checkpoint for resolving path '/xyz/_delta_log/00000000000000000000.checkpoint' within mount at '/mn...

Data Engineering
Reply
  • 1995 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Rahul Samant​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback...

  • 2 kudos
2 More Replies
477061
by • Contributor
  • 2777 Views
  • 3 replies
  • 0 kudos

Resolved! Renamed table cannot be written to or deleted from

I have renamed a table, however on trying to write to it (or delete from it) I get the following error: `java.io.FileNotFoundException: No such file or directory: s3a://.../hive/warehouse/testing.db/renamed_table_name/_delta_log/00000000000000000002....

Data Engineering
Reply
  • 2777 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Noopur_Nigam
Databricks Employee
  • 0 kudos

Hi @477061 Could you please try to test it in DBR 11.1 and see if the issue persists for you?

  • 0 kudos
2 More Replies
pantelis_mare
by • Contributor III
  • 5677 Views
  • 6 replies
  • 2 kudos

Resolved! Delta log statistics - timestamp type not working

Hello team!As per the documentation, I understand that the table statistics can be fetched through the delta log (eg min, max, count) in order to not read the underlying data of a delta table.This is the case for numerical types, and timestamp is sup...

max value image.png
Data Engineering
Reply
  • 5677 Views
  • 6 replies
  • 2 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 2 kudos

are you sure the timestamp column is a valid spark-timestamp-type?

  • 2 kudos
5 More Replies
KKo
by • Contributor III
  • 5189 Views
  • 3 replies
  • 4 kudos

Resolved! Reading multiple parquet files from same _delta_log under a path

I have a path where there is _delta_log and 3 snappy.parquet files. I am trying to read all those .parquet using spark.read.format('delta').load(path) but I am getting data from only one same file all the time. Can't I read from all these files? If s...

Data Engineering
Reply
  • 5189 Views
  • 3 replies
  • 4 kudos
View Replies
Latest Reply
KKo
Contributor III
  • 4 kudos

@Werner Stinckens​ Thanks for the reply and explanation, that was helpful to understand the delta feature.

  • 4 kudos
2 More Replies
soundari
by • New Contributor
  • 2204 Views
  • 1 replies
  • 1 kudos

Resolved! Identify the partitionValues written yesterday from delta

We have a streaming data written into delta. We will not write all the partitions every day. Hence i am thinking of running compact spark job, to run only on partitions that has been modified yesterday. Is it possible to query the partitionsValues wr...

Data Engineering
Reply
  • 2204 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Deepak_Bhutada
Contributor III
  • 1 kudos

Hi @Gnanasoundari Soundarajan​ Based on the details you provided, you are not overwriting all the partitions every day which means you might be using append mode while writing the data on day 1. On day 2, you want to access those partition values and...

  • 1 kudos
Labels