- 2437 Views
- 1 replies
- 1 kudos
Resolved! Can I delete specific partition from a Delta Live Table?
if I have created a Delta Live Table with partition on a column (lets say a date column) from a Stream Source, can I delete the partition for specific date values later to save on cost & to keep the table lean? if I can, then -1- how to do it?2- do I...
- 2437 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @gauravchaturved , You can remove the partition by filtering it in your source code and triggering a full refresh in your pipeline. There is no need to run vacuum, as DLT has maintenance clusters that perform OPTIMIZE and VACUUM operations on y...
- 1 kudos
- 4947 Views
- 3 replies
- 2 kudos
Unable to connect with Databricks Serverless SQL using Dbeaver
I am trying to connect to databricks serverless SQL pool using DBeaver as mentioned in the documentation below:https://learn.microsoft.com/en-us/azure/databricks/dev-tools/dbeaverI am trying to use the Browser based authentication i.e (OAuth user-to-...
- 4947 Views
- 3 replies
- 2 kudos
- 4963 Views
- 3 replies
- 2 kudos
Resolved! Databricks Expectations
In the example in https://docs.databricks.com/en/delta-live-tables/expectations.html#fail-on-invalid-records, it wrote that one is able to query the DLT event log for such expectations violation. In Databricks, I can use expectation to fail or drop r...
- 4963 Views
- 3 replies
- 2 kudos
- 2 kudos
That's right, the "reason" would be "x1 is negative" in your example and "valid_max_length" in the example JSON payload that I shared.If you are looking for a descriptive reason, you would name the expectation accordingly such as: @Dlt.expect_or_fail...
- 2 kudos
- 7510 Views
- 8 replies
- 0 kudos
Databricks Autoloader File Notification Not Working As Expected
Hello Everyone,In my project I am using databricks autoloader to incrementally and efficiently processes new data files as they arrive in cloud storage.I am using file notification mode with event grid and queue service setup in azure storage account...
- 7510 Views
- 8 replies
- 0 kudos
- 0 kudos
Hi @Sambit_S , I misread inputRows as inputFiles which aren't the same thing. Considering the limitation on Azure queue, if you are already at the limit then you may need to consider to switching to an event source such as Kafka or Event Hub to get b...
- 0 kudos
- 1204 Views
- 0 replies
- 0 kudos
Databricks Platform Cleanup and baseline activities.
Hi Team, Kindly share the best practices for managing Databricks Platform Cleanup and baseline activities.
- 1204 Views
- 0 replies
- 0 kudos
- 4070 Views
- 3 replies
- 0 kudos
SHOW GROUPS is not giving groups available at the account level
I am trying to capture all the Databricks groups and their mapping to user/ad group(s).I tried to do this by using show groups, show users, and show grants by following the examples mentioned in the below article but the show groups command only fetc...
- 4070 Views
- 3 replies
- 0 kudos
- 0 kudos
Yes, I can use the Rest API but I am looking for a SQL or Programming way to do this rather than doing the API calls and building the Comex Datatype Dataframe and then saving it as a Table.ThanksRamana
- 0 kudos
- 2817 Views
- 2 replies
- 1 kudos
Predicting compute required to run Spark jobs
Im working on a project to predict compute (cores) required to run spark jobs. Has anyone work on this or something similar before? How did you get started?
- 2817 Views
- 2 replies
- 1 kudos
- 1 kudos
@kseyser good day, This documentation might help you in your use-case: https://docs.databricks.com/en/compute/cluster-config-best-practices.html#compute-sizing-considerations Kind regards, Yesh
- 1 kudos
- 7628 Views
- 1 replies
- 2 kudos
Resolved! Advice for generic file processing for ingestion of multiple data formats
Hello,We are using delta live tables to ingest data from multiple business groups, each with different input file formats and parsing requirements. The input files are ingested from azure blob storage. Right now, we are only servicing three busines...
- 7628 Views
- 1 replies
- 2 kudos
- 2 kudos
Hello @Lea , I'd like to inform you that our platform does not currently provide a built-in feature for ingesting multiple or interchangeable file formats. However, we highly value your input and encourage you to share your ideas through Databricks' ...
- 2 kudos
- 13301 Views
- 2 replies
- 1 kudos
Resolved! Migration Azure to AWS
Hello, today I use Azure Databricks, I want to migrate my wordspaces to AWS Databricks. What is the best practice, which path should I follow?, I didn't find anything in the documentation.thanks.
- 13301 Views
- 2 replies
- 1 kudos
- 1 kudos
Hello, as I already have a working Databricks environment on Azure, the best way would be to use tool-databricks-migrate?
- 1 kudos
- 11530 Views
- 1 replies
- 0 kudos
SFTP connection using private key on Azure Databricks
I need to connect to a server to retrieve some files using spark and a private ssh key. However, to manage the private key safely I need to store it as a secret in Azure Key Vault, which means I don't have the key as a file to pass down in the keyFil...
- 11530 Views
- 1 replies
- 0 kudos
- 3906 Views
- 3 replies
- 0 kudos
Autoloader ignore one folder in path
Hi everyone!I am trying to setup Autoloader to read json file with specific name from all subfolders under the path except one.Could someone advice how this can be achieved? For example, I need to read from .../*/specific_name.json, but ignore test f...
- 3906 Views
- 3 replies
- 0 kudos
- 0 kudos
I think you can use REGEXP to achieve this. This might not be the best way, but it should get the job done. It's all about filtering that file in the df from getting loaded. Try something like thisdf.select(“*”,”_metadata”).select(“*”,”_metadata.file...
- 0 kudos
- 4385 Views
- 3 replies
- 2 kudos
How to find that given Parquet file got imported into Bronze Layer ?
Hi Team,Recently we had created new Databricks project/solution (based on Medallion architecture) having Bronze-Silver-Gold Layer based tables. So we have created Delta-Live-Table based pipeline for Bronze-Layer implementation. Source files are Parqu...
- 4385 Views
- 3 replies
- 2 kudos
- 2 kudos
Hello @Devsql , It appears that you are creating DLT bronze tables using a standard spark.read operation. This may explain why the DLT table doesn't include "new files" during a REFRESH operation. For incremental ingestion of bronze layer data into y...
- 2 kudos
- 5714 Views
- 0 replies
- 2 kudos
Delta Lake Liquid Clustering
Support for liquid clustering is now generally available using Databricks Runtime +15.2 Getting started with Delta Lake Liquid clustering https://lnkd.in/eaCZyhbF#DeltaLake #Databricks
- 5714 Views
- 0 replies
- 2 kudos
- 5219 Views
- 2 replies
- 0 kudos
Asynchronous API calls from Databricks Workflow job
Hi all,I have many API calls to run on a python Databricks notebook which I then run regularly on a Databricks Workflow job. When I test the following code on an all purpose cluster locally i.e. not via a job, it runs perfectly fine. However, when I ...
- 5219 Views
- 2 replies
- 0 kudos
- 0 kudos
I actually got it too work though I do see that if I run two jobs of the same code in parallel the async execution time slows down. Do the number of workers of the cluster on which the parallel jobs are run effect the execution time of async calls of...
- 0 kudos
- 1856 Views
- 1 replies
- 1 kudos
Cluster termination using python script, sending return code 1
i have used a cluster termination logic for terminating a cluster , the issue is , the cluster is not terminating gracefully , returns a return/exit code 1The cluster is completing all the spark jobs, but it goes on long running state, hence i create...
- 1856 Views
- 1 replies
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
