Facing issues databricks asset bundle, All jobs are getting Deployed into specified targets Instead of defined target following was files i am using resourser yaml and databricks yml file , i am using Databricks CLI v0.240.0 , i am using databricks b...
Hi,Sometime ago, I got a message that there will be a change (starting from 01/31/2025) in "deletion behavior of Materialized View and Streaming Tables defined by Delta Live Table", but when I remove dlt pipeline, it also removes related tables, will...
Hi @MariuszK, The users will need to explicitly call DROP MATERIALIZED VIEW to delete MVs and DROP TABLE to delete STs, when deleting DLT pipelines. https://home.databricks.com/account-alert-deletion-behavior-change-for-materialized-view-and-streamin...
We have instance pools setup with a maximum capacity and are looking at ways to monitor the usage to help with our capacity planning.I have been using the system tables to track how many nodes are being used within a pool at a point in time but it ap...
Hii there!!Currently, we are exploring options for reporting on Salesforce. We extract data from salesforce via databricks and store it in delta lake.Is there a connector by which data can be pulled from databricks into Tableau/CRM analytics??I know ...
Hello @ruby Were you able to resolve this? I know it's been a while, but I believe we now have multiple ways to connect Tableau and Databricks.1. Use the Native Databricks Connector for TableauTableau has a built-in Databricks connector, making it ea...
We used delta sharing (authentication type: token) to generate the config.share file and share with external users not from our organisation, the users faced the "FileNotFoundError" error while using python "delta_sharing.load_as_pandas" method to re...
Hello @tonykun_sg,It looks like ADLS Gen2 might be restricting access to the data through an ACL, which is why Databricks allows access but the underlying files remain protected. Could you check with your team to temporarily enable access for testing...
I am trying to create a SQL task in Workflows. I have my query which executes successfully in the SQL editor, and it is saved in a repo.However, when I try to execute the task, the below error shows.Query text can not be empty: BAD_REQUEST: Query tex...
It ended up being that the query wasn't actually saved. Once I manually clicked save, the query preview showed and the task ran successfully. I'm really surprised that was the reason. I had moved the query around to different folders and closed and r...
Hi,(1) On SQL warehouse, I created a table in unity catalog from data source file vw_businessmetrics_1000.json in ADLS blob.USE CATALOG `upreport`;USE SCHEMA `test_genie`;-- Create the external table from the JSON fileCREATE EXTERNAL TABLE IF NOT EXI...
Hi @Ayushi_Suthar Thank you very much. I tried with the Classic and Pro and it did not work.My solution is: drop the table and recreate as a delta table then loading data from json to the delta table. Now it works. Probably, the SQL warehouse only su...
Hi All,I am trying to create table using DLT and would like to specify the path where all the files should reside.I am trying something like this:dlt.create_streaming_table( name="test", schema="""product_id STRING NOT NULL PRIMARY KEY, ...
tengo un inconveniente igual. no me gusta guardar con un nombre aleatorio dentro de __unitystorage java.lang.IllegalArgumentException: Cannot specify an explicit path for a table when using Unity Catalog. Remove the explicit path:
I have a Databricks pyspark query that has been running fine for the last two weeks but am now getting the following error despite no changes to the query: OutOfMemoryError: GC overhead limit exceeded.I have done some research on possible solutions a...
When you say: "I have a Databricks pyspark query that has been running fine for the last two weeks but am now getting the following error despite no changes to the query: OutOfMemoryError: GC overhead limit exceeded."Can you tell us how do you execut...
What are the best practices for implementing non-stop streaming in a Medallion Architecture with a Star Schema?Use Case:We have operational data and need to enable near real-time reporting in Power BI, with a maximum latency of 3 minutes. No Delta li...
@g96g I've setup a near real-time (30-minute latency) streaming solution that ingests data from SQL Server into Delta Lake.Changes in the source SQL Server tables are captured using Change Data Capture (CDC) and written to CSV files in a data lake.A ...
com.databricks.backend.common.rpc.DriverStoppedException: Driver down cause: driver state change (exit code: 143)facing this cluster issue while i deploy and run my workflow through asset bundle. i have tried everything to update in spark configurati...
We started getting a very weird error at random from Databricks. This is from cells that routinely work, and after it happens once it will happen on every cell. It appears to be including full text of a .py file we're importing, that I've had to remo...
@TKr Hey everybody - sorry that you experienced these issues. We identified the issue and reverted the feature causing it. Things should be back to normal already.I'm glad to hear that. Are you a Databricks employee?Referring to your question, we did...
Hello,What are the benefits of not "registering" Raw data into Unity Catalog when the data in Raw will be in its original format, such as .csv, .json, .parquet, etc?An example scenario could be:Data arrives at Landing as .zip; The zip will be verifie...
Thanks @Rjdudley I meant to say, the scenario is:Data arrives at Landing as .zip; The zip will be verified for correctness, and then unzipped, the extracted files will be saved to Raw as-is, in a pre-defined folder structure. Unity Catalog will not...
I am trying to send a message to a service bus in azure. But I get following error:ServiceBusError: Handler failed: DefaultAzureCredential failed to retrieve a token from the included credentials.This is the line that fails: credential = DefaultAzure...
@Sega2 - Can you explicitly useClientSecretCredential and try
Hi all,as per the below article -https://community.databricks.com/t5/technical-blog/optimize-costs-for-your-data-and-ai-workloads-with-azure-and-aws/ba-p/662411. it is possible to choose the number of spot instances using 'availability' parameter. Bu...
Hello @noorbasha534, Thanks for your question! 1. 'availability' Parameter: The 'availability' parameter in Azure Databricks controls whether the compute uses on-demand or spot instances. The values for this parameter are: ON_DEMAND_AZURE: This value...
| User | Count |
|---|---|
| 1612 | |
| 768 | |
| 348 | |
| 286 | |
| 252 |
