Currently trying to connect a table on external location and it fails with error -[UNAUTHORIZED_ACCESS] Unauthorized access: PERMISSION_DENIED: request not authorized SQLSTATE: 42501which seems like a pretty straight forward error but unable to find ...
goal: implement event driven architecture without trigger on file arrivalI would like to know whether materialzied view can update itself when its source table which is external table updated.given that the source external table referencing data in d...
it seems that my idea is a bad idea because it seems that materialzied view doesn't support incremental udpate for external locationIncremental refresh for materialized views - Azure Databricks | Microsoft Learn
HiMy organization has many notebooks that reference tables in schemas with the three part pathcatalog.schema.tablenameWith a lack of foresight we hardcoded all of these paths in the code and now the inevitable is happening and there is a need to rest...
Hi @IONA ,Definitely, I would say that’s even a common practice. Create a feature branch and make the necessary changes there. But once a day, merge into that feature branch all the changes that have appeared on your main branch. That way, you will a...
We've been using auto loader to ingest data from a storage account on Azure (format "cloudFiles").Today, we're starting to see failures during the setup of event notification:25/09/11 19:06:28 ERROR MicroBatchExecution: Non-interrupted exception thro...
Hello @Malthe @Saska @MehdiJafariWhen it was showing such error, did you try any alternative methods like using databricks service credentials. For example stream = spark.readStream.format("cloudFiles") .option("cloudFiles.format", "json") .option...
Use Case:Kafka real time steaming network telemetry logsIn a real use case approx. 40 TB of data can be real time streamed in a dayArchitecture:issue encountered: when i try to simulate kakfa like streaming in databricks itself , as this is a free e...
Hi @ManojkMohan ,Unfortunately there is no workaround here. Free Edition supports only serverless compute. And serverless has following streaming limitation - one of which you just encountered - there is no support for default or time-based trigger i...
[Issue] DLT finishes processing, but cluster remains active due to log write errorHi everyone, I'm running into a problem with my DLT pipeline and was hoping someone here could help or has experienced something similar.Problem DescriptionThe pipeline...
Can you please try one more option. If you’re on Preview, move to Current (or vice versa). Sometimes the regression only exists in one channel.
HiI'm sure many of you have power bi reports that use the native Databricks connector to pull data from a schema to fuel a wonderful dashboard. If the source schemes was moved to a different catalog or a table renamed then the pbi connection would be...
Hi @IONA ,You can parametrize your data source using power query parameters. Just define parameter for catalog, schema etc. I have such setup in one of the project. We parametrized our PBI semantic models and when we are deploying semantic model to a...
HiWe have a couple of unity catalogs. In each are schema's, some used, some old, same named temp_data etc. etc Within those schema are tables with the same erratic approach. The result of a couple of years of bad housekeeping. (who not experienced th...
Did anyone know when we will be going to receive the partner tech summit badge?
I have written data to a table using clusterByAuto set to trueBut the clustering keys are not selected automatically when i do a desc detail on the table.Screenshot belowWhy are clustering columns not being selected automatically?Repro steps:Create ...
Hi @RevanthV ,As @K_Anudeep correctly suggested it could be the case that your table is to small to benefit from liquid clustering.Another possibility it that you're using runtime lower than 15.4 LTS.
Hello Databricks Support Team,I am reaching out to request assistance with upgrading the drivers and configuring authentication methods for our Snowflake–Databricks integration.We would like to explore and implement one of the recommended secure auth...
Hello @Vinil Greatings for the day! For Snowflake connectivity from Databricks, OAuth is generally recommended as the most secure and scalable method, especially for production environments. You can find more details over here: https://docs.databrick...
Hello Community,We have been trying to migrate our jobs from Classic Compute to Serverless Compute. As part of this process, we face several challenges, and this is one of them.When we try to execute the existing jobs with Serverless Compute, if the ...
Hello @Ramana ,The above error occurs when the Spark SQL optimiser is unable to resolve a query within the fixed maximum number of rule-application iterations (default 1000) in its internal logical plan "Resolution" phase. This typically happens with...
Hello @Soumenkumar ,I believe OGG has a Databricks target that stages to cloud storage (ADLS on S3) and then runs MERGE into Delta tables. This is designed for UC and documented in the official Oracle Docs.In Unity Catalog, create a storage credentia...
This function logs every time it writes to stdout "Wrote n bytes." I want to disable its logging as I have thousands of files I'm writing and it floods the log with meaningless information. Does anyone know if it's possible?
Hello @ToBeDataDriven ,If it's a notebook cell, you can silence the output of the cell by using %%capture %%capture dbutils.fs.put("dbfs:/FileStore/anudeep/datasets/word_count/tmp/3.txt","foo") If you want to do it in your code, then there is no flag...
Hi Everyone,Today I gave databricks exam for and I got 64 questions and my result was exactly 70.00%(As per databricks the pass percentage is 70 or above). but still the status was showing Failed and I couldn't get certified.Can you anyone help me on...
Same story and get the same reply from supportI cant even determine how far I am to the passing scoretelling u fail, then u failed
| User | Count |
|---|---|
| 1616 | |
| 784 | |
| 349 | |
| 324 | |
| 287 |
