- 4899 Views
- 3 replies
- 10 kudos
Resolved! How to retrieve Auto Loader client secret from Azure Key Vault?
I'm using Auto Loader in a SQL notebook and I would like to configure file notification mode, but I don't know how to retrieve the client secret of the service principal from Azure Key Vault. Is there any example notebook somewhere? The notebook is p...
- 4899 Views
- 3 replies
- 10 kudos
- 10 kudos
Hi @Magnus Johannesson​ , you must use the Secrets utility (dbutils.secrets) in a notebook or job to read a secret.https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-utils#dbutils-secretsHope it helps!
- 10 kudos
- 5674 Views
- 3 replies
- 2 kudos
FutureWarning: ``databricks.feature_store.entities.feature_table.FeatureTable.keys`` is deprecated since v0.3.6
I'm getting this message with the following code:from databricks import feature_store fs = feature_store.FeatureStoreClient() fs.create_table( name='feature_store.user_login', primary_keys=['user_id'], df=df_x, description='user l...
- 5674 Views
- 3 replies
- 2 kudos
- 2 kudos
Yes, it's a nice thing to do. You can report it here: https://community.databricks.com/s/topic/0TO3f000000CnKrGAK/bug-report and if it's more urgent or blocking for you, you can also open a ticket to the help center: https://docs.databricks.com/resou...
- 2 kudos
- 762 Views
- 0 replies
- 0 kudos
patientportaltvobgynÂ
patientportaltvobgyn
- 762 Views
- 0 replies
- 0 kudos
- 4639 Views
- 4 replies
- 9 kudos
Resolved! How to call a few child notebooks from master notebook parallelly?
Planning using dbutils.notebook.run() to call all the child notebooks in the master notebook, but they are executed sequentially.
- 4639 Views
- 4 replies
- 9 kudos
- 9 kudos
Hi @andrew li​ Yes, They do run on the cluster on which the master notebook is running. Specifically, we are multithreading the Spark driver with Futures to enable parallel job submission. You can check out more on threads and futures for deeper unde...
- 9 kudos
- 3218 Views
- 0 replies
- 0 kudos
The CDC Logs from AWS DMS not apply correctly
I have a dms task that processing the full-load and replication ongoing tasksfrom source (MSSQL) to target (AWS S3)then use delta lake to handle the CDC logsI've a notebook that would insert data into mssql continuously (with id as primary key)then d...
![204293406-01bf6cc1-bb6f-42bb-9bfe-e9b1f5135ae9[1]](/t5/image/serverpage/image-id/1103i5E2E130C9E462F86/image-size/medium?v=v2&px=400)
- 3218 Views
- 0 replies
- 0 kudos
- 9869 Views
- 1 replies
- 4 kudos
Databricks Jobs API not returning notebook run results?
Calling a databricks notebook using the Rest API, can confirm that it is executing the notebook, but is not accepting my parameters or returning a notebook output. Any ideas on what I am doing wrong here?My code and notebook function are below, tryin...
- 9869 Views
- 1 replies
- 4 kudos
- 4 kudos
Resolved this by using dbutils within the notebook being called from the API.# databricks notebook function data = dbutils.widgets.get('data') # pulls base_parameters from API call def add_test(i): result = i + ' COMPLETE' return result ...
- 4 kudos
- 1663 Views
- 0 replies
- 0 kudos
How to query a MySQL Table from Databricks?
I wanted to query a MySQL Table using Databricks rather than reading the complete data using a dbtable option, which will help in incremental loads.remote_table = (spark.read .format("jdbc") .option("driver", driver) .option("url", URL) .option("quer...
- 1663 Views
- 0 replies
- 0 kudos
- 3527 Views
- 4 replies
- 1 kudos
Hi @Kaniz Fatma​ and @Nadia Elsayed​ , i have taken databricks data engineer associate exam on nov 27th . in result mail it is mentioned i have obtain...
Hi @Kaniz Fatma​ and @Nadia Elsayed​ ,i have taken databricks data engineer associate exam on nov 27th . in result mail it is mentioned i have obtained below 70% in assessment but as per the section wise results i have gained more than 70% . Can you ...
- 3527 Views
- 4 replies
- 1 kudos
- 22711 Views
- 4 replies
- 11 kudos
The functionality of table property delta.logRetentionDuration
We have one project requirement where we have to store only the 14 days history for delta tables. So for testing, I have set the delta.logRetentionDuration = 2 days using the below commandspark.sql("alter table delta.`[delta_file_path]` set TBLPROPER...
- 22711 Views
- 4 replies
- 11 kudos
- 11 kudos
Hi, by default there is a safety interval enabled. So if you set a retentionperiod lower than that interval (7 days), data in that interval will not be deleted.You have to specificall override this safety interval by setting spark.databricks.delta.r...
- 11 kudos
- 12796 Views
- 3 replies
- 10 kudos
Resolved! Number of partitions in Spark UI Simulator experiment
I am learning how to optimize Spark applications with experiments from Spark UI Simulator. There is experiment #1​596 about data skew and in command 2 there is comment about how many partitions will be set as default:// Factor of 8 cores and greater ...
- 12796 Views
- 3 replies
- 10 kudos
- 10 kudos
Hi @Bartosz Maciejewski​ Generally we arrive at the number of shuffle partitions using the following method.Input Size Data - 100 GBIdeal partition target size - 128 MBCores - 8Ideal number of partitions = (100*1028)/128 = 803.25 ~ 804To utiltize the...
- 10 kudos
- 2726 Views
- 3 replies
- 4 kudos
Hi @Vidula Khanna​  i have taken databricks data engineer associate exam on nov 27th . in result mail it is mentioned i have obtained below 70% in ass...
Hi @Vidula Khanna​ i have taken databricks data engineer associate exam on nov 27th . in result mail it is mentioned i have obtained below 70% in assessment but as per the section wise results i have gained more than 70% . Can you please check and up...
- 2726 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @Harish Raj​ Please bear with us!I already have shared this issue with the respective person in the Academy team. Thanks for your patience!
- 4 kudos
- 5885 Views
- 4 replies
- 4 kudos
Data bricks issue writing to Azure synapse
Process is to pickup data from an external table in Azure Synapse and combine with other tables and write to another synapse tables. Data for external table is in Azure storage. It was fine for months ,all of sudden last week it errored out with err...
- 5885 Views
- 4 replies
- 4 kudos
- 4 kudos
Also do check out this blog from this stack overflow question.https://stackoverflow.com/questions/66747544/databricks-write-back-to-azure-synapse-errorhttps://chinnychukwudozie.com/2020/11/13/write-data-from-azure-databricks-to-azure-synapse-analytic...
- 4 kudos
- 23608 Views
- 7 replies
- 4 kudos
Resolved! Invalid JDBC url
Hello. I am trying to establish a connection between DBeaver and Databricks. I followed the steps in DBeaver integration with Databricks | Databricks on AWS, but I get the following error while testing the connection: Could anyone provide any insight...
- 23608 Views
- 7 replies
- 4 kudos
- 4 kudos
Hi @ELENI GEORGOUSI​ Glad to hear! It's a request that mark an answer as best.Thanks...
- 4 kudos
- 7146 Views
- 3 replies
- 6 kudos
Unable to create materialized view in Unity enabled Catalogues.
Hi Team,I was going through one of the videos of Databricks Sql Serverless and it say there is materialized view support . We can create materialized view .I tried same on my cluster of Sql Warehouse gives below error:
- 7146 Views
- 3 replies
- 6 kudos
- 6 kudos
Materialized views is in private preview right now afaik. Please talk to your account or customer success team at Databricks in order to sign up and enable it for your workspace. Thanks!
- 6 kudos
- 2534 Views
- 0 replies
- 2 kudos
java.lang.SecurityException: Could not verify permissions for OverwritePartitionsDynamic RelationV2 - Delta tables dynamic partition overwrite on Databricks ACL enabled clusters
I'm working on Databricks ACL enabled clusters, and having trouble performing dynamic partition overwrite to Delta tables.I have created a test table using the following query:CREATE TABLE IF NOT EXISTS test_01 ( id STRING, name STRING, c...
- 2534 Views
- 0 replies
- 2 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 566 | |
| 349 | |
| 287 |
