- 5202 Views
- 1 replies
- 0 kudos
Databricks asset bundles error "failed to instantiate provider"
Hi all,I'm trying to deploy with databricks asset bundles. When running bundle deploy, the process fails with following error message:failed execution pid=25092 exit_code=1 error="terraform apply: exit status 1\n\nError: failed to read schema for dat...
- 5202 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi,sorry for my late answer and thank you for your help. I was on parental leave in the meantime.I solved the problem by installing the latest Databricks extension. With the new extension and updated Databricks-CLI I got the same error.However: The l...
- 0 kudos
- 8409 Views
- 4 replies
- 0 kudos
How to deal with delete records from the source Files in DLT .
Can apply_changes feature deal with deleted records in incoming source Files?By delete I mean record is being removed (Not a soft delete with Flag).If not, how to automate with deleting records from Bronze Streaming table based on source Files.
- 8409 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi Manoj,Did you get the solution or design change for this problem. We have 200K files on to S3 bucket and when there is change in upstream app we get new feed, feed name is fixed. On DLT we should have only new records from replaced file but in dlt...
- 0 kudos
- 2152 Views
- 1 replies
- 0 kudos
Credential passthrough and Hive metastore table access controls are deprecated
Hello,Based on the recent platform release, the credential passthrough will be deprecated for runtime 15.0 and later. Our current setup involves using Databricks alongside AWS Glue and Athena, i.e. registering delta tables in AWS Glue and running oth...
- 2152 Views
- 1 replies
- 0 kudos
- 2413 Views
- 2 replies
- 0 kudos
Connecting to ADLS azure storage and reading csv file from adls
Hi Friends , How are you all? I am Anwitha. I am a newbie in azure data bricks community. I am stuck with this error while practising with azure storage account. Could anyone throw a light on this error and provide a solution?I tried connecting to or...
- 2413 Views
- 2 replies
- 0 kudos
- 0 kudos
Ermm, you might want to edit your post and remove that storage account access key! In combination with the storage account name (that is also in your post), everyone will be able to access it if it isn't behind a firewall! Btw, you might want to stud...
- 0 kudos
- 1429 Views
- 3 replies
- 0 kudos
Error while reading steaming source from DLT Pipeline
Hi All, I am getting this error, when I am reading the streaming sourceFull load - It loadsIncremental load - am facing this errorThis is the piece of code that am using def gim_suppliers_ln(): logger.info("Starting __cn_gim_suppliers") overall...
- 1429 Views
- 3 replies
- 0 kudos
- 9510 Views
- 10 replies
- 2 kudos
Resolved! Data Explorer minimum permissions
What are the minimum permissions are required to search and view objects in Data Explorer? For example, does a user have to have `USE [SCHEMA|CATALOG]` to search or browse in the Data Explorer? Or can anyone with workspace access browse objects and, ...
- 9510 Views
- 10 replies
- 2 kudos
- 2 kudos
Circling back to this. With one of the recent releases you can now GRANT BROWSE at the catalog level! Hopefully they will be rolling this feature out at every object level (schemas and tables specifically).
- 2 kudos
- 1486 Views
- 1 replies
- 0 kudos
Delta Sharing Azure Databricks Private Storage
I am looking for details on how Delta Sharing is configured on Azure. If your storage account is private, do you need to whitelist the IP for the recipient? Recipient is not using Databricks.
- 1486 Views
- 1 replies
- 0 kudos
- 3424 Views
- 2 replies
- 0 kudos
Resolved! Differences between Spark SQL and Databricks
Hello,I'm using a local Docker Spark 3.5 runtime to test my Databricks Connect code. However I've come across a couple of cases where my code would work in one environment, but not the other.Concrete example, I'm reading data from BigQuery via spark....
- 3424 Views
- 2 replies
- 0 kudos
- 0 kudos
@dollyb That's because when you've added another dependency on Databricks, it doesn't really know which one it should use. By default it's using built-in com.google.cloud.spark.bigquery.BigQueryRelationProvider.What you can do is provide whole packag...
- 0 kudos
- 1280 Views
- 0 replies
- 0 kudos
Optimized option to write updates to Aurora PostgresDB from Databricks/spark
Hello All, We want to update our postgres tables from our spark structured streaming workflow on Databricks. We are using foreachbatch utility to write to this sink. I want to understand an optimized way to do this at near real time latency avoidi...
- 1280 Views
- 0 replies
- 0 kudos
- 1462 Views
- 1 replies
- 0 kudos
Azure Devops - Entra ID - AWS Databricks
Hi, I need to integrate Azure Devops repos with AWS Databricks, but not via personal token.I need it via main service, integrated with Azure Entra ID, using Azure Databricks when I go to create main service, "Entra ID application ID" appears, but in ...
- 1462 Views
- 1 replies
- 0 kudos
- 1882 Views
- 1 replies
- 0 kudos
Resolved! unity catalog with external table and column masking
Hi everbody, I am facing a issue with spark structured steaming. here is a sample of my code: df = spark.readStream.load(f"{bronze_table_path}") df.writeStream \ .format("delta") \ .option("checkpointLocation", f"{silver_checkpoint}") \ .option("me...
- 1882 Views
- 1 replies
- 0 kudos
- 0 kudos
My first message was not well formatted. i wrote : df = spark.readStream.load(f"{bronze_table_path}") df.writeStream \ .format("delta") \ .option("checkpointLocation", f"{silver_checkpoint}") \ .option("mergeSchema", "true") \ .trigger(availabl...
- 0 kudos
- 2606 Views
- 1 replies
- 0 kudos
Cannot get past Query Data tutorial for Azure Databricks
I created a new workspace on Azure Databricks, and I can't get past this first step in the tutorial: DROP TABLE IF EXISTS diamonds; CREATE TABLE diamonds USING CSV OPTIONS (path "/databricks-datasets/Rdatasets/data-001/csv/ggplot2/diamonds.csv", hea...
- 2606 Views
- 1 replies
- 0 kudos
- 0 kudos
Struggling with this as well. So using dbfs:/ with CREATE TABLE statement works on AWS, but not Azure?
- 0 kudos
- 5586 Views
- 1 replies
- 0 kudos
Measure size of all tables in Azure databricks
Hi Team,Currently I am trying to find size of all tables in my Azure databricks, as i am trying to get idea of current data loading trends, so i can plan for data forecast ( i.e. Last 2 months, approx 100 GB data came-in, so in next 2-3 months there ...
- 5586 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Retired_mod,1- Regarding this issue i had found below link:https://kb.databricks.com/sql/find-size-of-table#:~:text=You%20can%20determine%20the%20size,stats%20to%20return%20the%20sizeNow to try above link, I need to decide: Delta-Table Vs Non-De...
- 0 kudos
- 1908 Views
- 1 replies
- 0 kudos
Asset Bundles webhook not working
Hello,The webhook notifications in databricks jobs defined in the asset bundles are not taken into account and therefore not created. For instance, this is not working:resources: jobs: job1: name: my_job webhook_notifications: on...
- 1908 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Retired_mod ,Thank you for your help.However we did check the job configuration multiple time. If we substitue 'webhook_notifications' with 'email_notifications' it works, so the syntax is correct. Here is a sample of our configuration:For the...
- 0 kudos
- 2292 Views
- 1 replies
- 0 kudos
Access historical injected data of COPY INTO command
Dear Community,I'm using the COPY INTO command to automate the staging of files that I get in an S3 bucket into specific delta tables (with some transformation on the fly).The command works smoothly, and files are indeed inserted only once (writing i...
- 2292 Views
- 1 replies
- 0 kudos
| User | Count |
|---|---|
| 1628 | |
| 790 | |
| 510 | |
| 349 | |
| 287 |
