- 725 Views
- 3 replies
- 5 kudos
Resolved! user standard serverless with asset bundle on Azure
Anyone running into issues with using standard serverless with Asset bundle we tried all options with below line performance_target: STANDARDbut it ignore above value and uses performance optimized cluster which is not expected any lead with ri...
- 725 Views
- 3 replies
- 5 kudos
- 5 kudos
resources: jobs: my_dabs: performance_target: STANDARD Please check whether it is on the correct level in the YAML. Also consider updating the CLI. I've just tested it, and it worked properly.
- 5 kudos
- 456 Views
- 3 replies
- 4 kudos
Resolved! Getting below error when trying to create a Data Quality Monitor for the table. ‘Cannot create Monit
Getting below error when trying to create a Data Quality Monitor for the table.‘Cannot create Monitor because it exceeds the number of limit 500'.
- 456 Views
- 3 replies
- 4 kudos
- 4 kudos
Maybe this is not a situation, but trial accounts have lower quotas. Tried to find the quota related to the monitor in "databricks resource-quotas list-quotas" but couldn't find it.Your account contact in databricks can probably adjust it or find wit...
- 4 kudos
- 1947 Views
- 4 replies
- 4 kudos
Resolved! How to create a Unity Catalog Connection to SQL Server using service principal??
I am trying to use the Databricks Python SDK (v 0.63.0) to create a Unity Catalog connection to a Azure-hosted SQL Server database using an Azure service principal to authenticate. I have successfully done this via the Workspace UI, but I am trying t...
- 1947 Views
- 4 replies
- 4 kudos
- 4 kudos
Hi @ScottH ,You need to configure it in following way (I've tested it and it works). In a place where a red arrow is pointing you need to provide your own tenant_id:
- 4 kudos
- 4115 Views
- 5 replies
- 0 kudos
Unity Catalog is missing column in Catalog Explorer
I have just altered one of my tables and added a column.ALTER TABLE tpch.customer ADD COLUMN C_CUSTDETAILS struct<key:string,another_key:string,boolean_key:boolean,extra_key:string,int_key:long,nested_object:struct<more:long,arrayOne:array<string>>>A...
- 4115 Views
- 5 replies
- 0 kudos
- 0 kudos
I had this error just recently. I did basically the same table transformation 4 times over the course of 2 days. Added two new fields to two different tables using the same SQL syntax. It worked 3 out of 4 times, but for 1 the column is not available...
- 0 kudos
- 934 Views
- 3 replies
- 2 kudos
Lakebridge code conversion from oracle to databricks sql
Hi Community,I’m working on migrating several Oracle views to SparkSQL using the Databricks Labs Lakehouse Bridge tool. I’m facing issues while converting the code .For oracle views and materialized views .Problems I’m encountering:The converted SQL...
- 934 Views
- 3 replies
- 2 kudos
- 2 kudos
@Louis_Frolio - will there be similar guidelines for other 'source' code too such as T-SQL, Teradata. Any such comprhensive docs on Laeebridge to cover per source.
- 2 kudos
- 1078 Views
- 1 replies
- 2 kudos
Resolved! Moving files using DBUtils is so slow
I am using the platform DBUtils.fs.mv() on databricks clusters, and facing issues with move operation slowness.I move files in UC Volumes or ADLS storage abfss links, which work but is so slow.I mean it takes hours to transfer files that used to take...
- 1078 Views
- 1 replies
- 2 kudos
- 2 kudos
Hello @murtadha_s , here are some helpfult tips and hints to help you further diagnose the slowness. Totally expected behavior here: object-storage moves with dbutils.fs.mv will be much slower than HDFS. Under the hood, dbutils isn’t doing an atom...
- 2 kudos
- 387 Views
- 1 replies
- 0 kudos
Serverless Compute – ADLS Gen2 Authorization Failure with RBAC
We are facing an authorization issue when using serverless compute with ADLS Gen2 storage. Queries fail with:Code AbfsRestOperationException: Operation failed: "This request is not authorized to perform this operation.", 403 AuthorizationFailureDetai...
- 387 Views
- 1 replies
- 0 kudos
- 0 kudos
private link from serverless, as probably you are not allowing public internet access. Configure private connectivity to Azure resources - Azure Databricks | Microsoft Learn you need to add both dfs and blob
- 0 kudos
- 14166 Views
- 7 replies
- 3 kudos
Resolved! SQL Warehouse Serverless - Not able to access the external tables in the hive_metastore
I have DLT tables created under the hive_metastore with external data stored in ADL gen2.The ADL blob storage is mounted into /mnt/<storage-account>The tables are successfully created and accessible from my notebooks, as well the ADL storage.I have c...
- 14166 Views
- 7 replies
- 3 kudos
- 3 kudos
we can use terraform to create NCC (Network Connectivity Configuration). It will create a private endpoint on storage account and approve it manually because it is not auto approved.
- 3 kudos
- 1042 Views
- 2 replies
- 1 kudos
Resolved! Multiple Databricks Issues: Spark Context Limit, Concurrency Load, API Character Limit & Job Timeout
I am encountering multiple issues in our Databricks environment and would appreciate guidance or best-practice recommendations for each. Details below:1. [MaxSparkContextsExceeded] Too many execution contexts are open right now (Limit 150)Error: [Max...
- 1042 Views
- 2 replies
- 1 kudos
- 1 kudos
I would like to add my experience with 3. Databricks API 10k Character LimitWe had a similar issue, and this limit cannot be changed. Instead review concepts of sharing the input/output between Databricks and caller using cloud storage like ADLS. Pro...
- 1 kudos
- 607 Views
- 4 replies
- 1 kudos
Reset committed offset of spark streaming to capture missed data
I have a very straightforward setup between Azure Eventhub and DLT using the kafka endpoint through spark streaming.There were network issues and the stream didn't pick up some event, but still progressed (and committed) the offset for some reasonAs ...
- 607 Views
- 4 replies
- 1 kudos
- 238 Views
- 1 replies
- 0 kudos
Authentication Temporarily Unavailable
This has happened alot in the previous weeks although both azure and Databricks showed no issues at the time the error was recieved by both Databricks python SDK and Java SDK, now I started creating a retry mechnaism to retry those errors selectively...
- 238 Views
- 1 replies
- 0 kudos
- 0 kudos
My PoV please, We use Databricks REST API; and we have faced 401 or Azure Front door related auth issues. Like you said, we use retry mechanism. Runtime errors are recorded and retry attempts are made if the tasks are idempotent; otherwise user inter...
- 0 kudos
- 367 Views
- 3 replies
- 2 kudos
Compression Export to volume is not working as expected
I am trying to write data into a volume using below table.coalesce(1) .write .mode("overwrite") .format(file_format) .option("header", "true") .option("delimiter", field_delimiter) .option("compre...
- 367 Views
- 3 replies
- 2 kudos
- 2 kudos
It sounds like Spark is splitting your output into many small files (one per row) despite coalesce(1). Can you try setting spark.sql.files.maxRecordsPerFile , this limits how many records can be written into a single output file; if this is set to 1 ...
- 2 kudos
- 1888 Views
- 17 replies
- 5 kudos
Resolved! remote_query() is not working
I am trying to experiment with remote_query() function according to the documentation. The feature is in public preview, so I assume it should be available to everyone now.select * from remote_query( 'my_connection', database => 'mydb', dbtable...
- 1888 Views
- 17 replies
- 5 kudos
- 5 kudos
Hi @Coffee77 are you giving the foreign catalog details in the remote query function? coffee77.sampleDB
- 5 kudos
- 500 Views
- 4 replies
- 0 kudos
Archive of legacy system into Databricks with structure and semi-structured data
We are currently exploring using Data Bricks to store and archive data from a legacy syste. The governance features of Unity Catalogue will give us the required capabilities to ensure we meet our legal, statutory and policy requirements for data rete...
- 500 Views
- 4 replies
- 0 kudos
- 0 kudos
Classic 360 Custoemr View case and Databricks is certainly the right platform to do so.Strcutred Data - Stores in the Delta TablesEmail and PDFs - stored in Volumes, however, metadata as path to the volumes stored in delta table against customer-idIn...
- 0 kudos
- 933 Views
- 5 replies
- 2 kudos
Want to see logs for lineage view run events
Hi All,I need your help, as I am running jobs it is getting successful, when I click on job and there we can find lineage > View run events option when click on it. I see below steps.Job Started: The job is triggered.Waiting for Cluster: The job wait...
- 933 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi there, I vibe-coded* a query where I was able to derive most of your events from the system tables: system.lakeflow.jobssystem.lakeflow.job_run_timelinesystem.lakeflow.job_task_run_timeline If you have SELECT access to system tables, this could b...
- 2 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
