- 2651 Views
- 3 replies
- 1 kudos
How can I optimize my data pipeline?
Delta Lake provides optimizations that can help you accelerate your data lake operations. Here’s how you can improve query speed by optimizing the layout of data in storage.There are two ways you can optimize your data pipeline: 1) Notebook Optimizat...
- 2651 Views
- 3 replies
- 1 kudos
- 1 kudos
some tips from me:Look for data skews; some partitions can be huge, some small because of incorrect partitioning. You can use Spark UI to do that but also debug your code a bit (get getNumPartitions()), especially SQL can divide it unequally to parti...
- 1 kudos
- 2992 Views
- 4 replies
- 1 kudos
List of Databricks tables created by a user
Hi team,Could you please confirm below clarifications1. How can we get the list of tables created by a user in particular workspace?2. How can we get the list of tables created by user from multiple workspaces? ( Same user has access to 10 workspace...
- 2992 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi Arun, hope your query is answered. Please select the best answer or let us know if any further questions.
- 1 kudos
- 1328 Views
- 1 replies
- 0 kudos
Reading multi-dimensional json files
So I've been having some issues reading a json file that's been provided to the business with another nesting layer, so instead of a json being an:'array of objects' -> [ {} ,{} ,{} ] It's an 'array of arrays of objects' -> [ [ {}, {} ,{} ], [ {} ,{}...
- 1328 Views
- 1 replies
- 0 kudos
- 0 kudos
You can use the explode function to flatten the array to rows, can you post a simple example of your data?
- 0 kudos
- 1057 Views
- 2 replies
- 0 kudos
"desc history" shows versions older than the default logRetentionDuration of 30 days
I have a cdc enabled table where no data changes were made since July 28. Then updates started occurring from November 22 onwards. The first checkpoint occurred on Nov 28. Based on the corresponding timestamp of checkpoint and log files, it looks lik...
- 1057 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Laval Liahkim, could you please try running the VACUUM with 30 days retention?Please confirm when you last run the cmd with the 30-day retention period. Also, when you created this table and do you see old data files were deleted?Also, when disk...
- 0 kudos
- 2299 Views
- 3 replies
- 0 kudos
How to debug Workflow Jobs timing out and DLT pipelines running forever?
So I'm the designated data engineer for a proof of concept we're running, I'm working with one infrastructure guy who's setting up everything in Terraform (company policy). He's got the setup down for Databricks so we can configure clusters and run n...
- 2299 Views
- 3 replies
- 0 kudos
- 0 kudos
@Espen Solvang - Just thought of checking with you, could you please let us know if you require further assistance on this?
- 0 kudos
- 2151 Views
- 2 replies
- 0 kudos
How can I run a streaming query on a new table with tbl property: change data feed enabled?
In Databricks on AWS, I am trying to run a streaming query (trigger=Once) with delta.enableChangeDataFeed=true in the table definition as instructed, but this always fails with :ERROR: Some streams terminated before this command could finish! com.d...
- 2151 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @daniel e Can you try running the select command on table changes from 0th version and see if you get output?SELECT * FROM table_changes('tableName', 0)Also, Please share the streaming query that you are running.
- 0 kudos
- 3074 Views
- 4 replies
- 2 kudos
Truncate delta live table and try to repopulate it in the pipeline
Has anyone attempted to truncate a delta live gold level table that gets populated via a pipeline and then tried to repopulate it by starting the pipeline. I have this situation wherein i need to reprocess all data in my gold table, so i stopped the ...
- 3074 Views
- 4 replies
- 2 kudos
- 2 kudos
Please can you confirm if the job is still failing with the same error even after “FULL REFRESH ALL” option? If so please share the full stack trace and is it failing in any of the below steps?Creating updateWaiting for resourcesInitializingResetting...
- 2 kudos
- 1697 Views
- 2 replies
- 3 kudos
Does exist the ability to run jobs with integration tests from the Databricks interface?
Currently, Nutter could be run inside a common CI/CD pipeline from GitLab, but need the possibility to run jobs with integration tests from the Databricks interface.How to use Nutter directly from Databricks?Does exist any integration test examples a...
- 1697 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Dmitrii Kalashnikov,You could find examples and more details here https://github.com/alexott/databricks-nutter-repos-demo
- 3 kudos
- 3150 Views
- 5 replies
- 1 kudos
Resolved! ApprodxQuantile does not seem to be working with delta live tables (DLT)
HI,I am tying to use the approxQuantile() function and populate a list that I made, yet somehow, whenever I try to run the code it's as if the list is empty and there are no values in it.Code is written as below:@dlt.table(name = "customer_order_silv...
- 3150 Views
- 5 replies
- 1 kudos
- 1 kudos
Maybe try to use (and the first test in the separate notebook) standard df = spark.read.table("customer_order_silver") to calculate approxQuantile.Of course, you need to set that customer_order_silver has a target location in the catalog, so read us...
- 1 kudos
- 3567 Views
- 2 replies
- 0 kudos
Resolved! facing issue mentioned in body when connecting event hub with databricks , followed earlier discussion on this but no solution
ERROR: Query termination received for [id=37bada03-131b-4fbb-8992-a427263fef2c, runId=cf3d7c18-780e-43ae-aed0-9daf2939b823], with exception: java.lang.IllegalArgumentException: Input byte array has wrong 4-byte ending unit at java.util.Base64$Decoder...
- 3567 Views
- 2 replies
- 0 kudos
- 0 kudos
The issue could be due to the mismatch in the eventHub jar and the dependencies added. Also, not all the required dependencies may be added.Suggestions:Using the azure_eventhubs_spark_2_12_.jar eventHub spark jar along with the following dependencies...
- 0 kudos
- 3318 Views
- 4 replies
- 7 kudos
I'd like to ingest data into my ADLS from sql server in an incremental manner using Delta Live Tables.
I'd like to ingest data into my ADLS from sql server in an incremental manner using Delta Live Tables. I do not want to use any staging tables. I was using CDC, While I call dlt.apply_changes, its asking me to specify source and target. SInce source ...
- 3318 Views
- 4 replies
- 7 kudos
- 7 kudos
If you have a CDC feed, looks like we can use this: https://docs.databricks.com/workflows/delta-live-tables/delta-live-tables-cdc.html
- 7 kudos
- 2087 Views
- 3 replies
- 2 kudos
Error while optimizing the table . Failure of InSet.sql for UTF8String collection
count of the table : 1125089 for october month data , So I am optimizing the table. optimize table where batchday >="2022-10-01" and batchday<="2022-10-31"I am getting error like : GC overhead limit exceeded at org.apache.spark.unsafe.types.UTF8St...
- 2087 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Nagini Sitaraman To understand the issue better I would like to get some more information. Does the error occur at the driver side or executor side? Can you please share the full error stack trace? You may need to check the spark UI to find wher...
- 2 kudos
- 11691 Views
- 2 replies
- 13 kudos
Understanding Rename in Databricks Now there are multiple ways to rename Spark Data Frame Columns or Expressions. We can rename columns or expressions...
Understanding Rename in DatabricksNow there are multiple ways to rename Spark Data Frame Columns or Expressions.We can rename columns or expressions using alias as part of selectWe can add or rename columns or expressions using withColumn on top of t...
- 11691 Views
- 2 replies
- 13 kudos
- 2356 Views
- 2 replies
- 2 kudos
Issue connecting to SQL warehouse spark thrift server
we have a library that allows dotnet applications to talk to databricks clusters (https://github.com/clearbank/SparkSqlClient). This communicates with the clusters over the spark thrift serverAlthough this works great for clusters in the "data scienc...
- 2356 Views
- 2 replies
- 2 kudos
- 2 kudos
I have tried those connection details however it they give me 400 errors when trying to connect directly using the hive thrift server contract (https://github.com/apache/hive/blob/master/service-rpc/if/TCLIService.thrift). I do not get the issues whe...
- 2 kudos
- 1392 Views
- 2 replies
- 1 kudos
Unexpected workspace setup dialog in the account
Greetings,Recently we were doing cleanups in AWS and removed some Databricks related resources that were used only once for setting up our workspace and were not used since then.Since there is no plan to create any other workspaces the decision was t...
- 1392 Views
- 2 replies
- 1 kudos
- 1 kudos
The resources that were cleaned up were just the ones that were used for the initial setup of the workspace, everything else important for the day to day operation are in place and we are actively using the workspace, therefore there is no plan to de...
- 1 kudos