- 2390 Views
- 2 replies
- 0 kudos
Query on using secret scope for dbt-core integration with databricks workflow
Hello all,I am currently configuring dbt-core with Azure Databricks Workflow and using Azure Databricks M2M (Machine-to-Machine) authentication for this setup. I have the cluster ID and cluster secret ID stored in Databricks secret scope.I am seeking...
- 2390 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Ram-Dev7 , Thank you for reaching out to our community! We're here to help you. To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedbac...
- 0 kudos
- 2376 Views
- 1 replies
- 2 kudos
DLT Serverless costs
I recently started checking out the serverless delta live tables. In my understanding serverless continuous jobs (with autoloader) would only do something when new files arrive. However for serverless 4 pipelines running continuously, I spend in two ...
- 2376 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @mbaas , That does not sound right. So, were you able to compare the jobs, stages and spot the difference. Are there more tasks added, or with compute perspective you do not find a difference at all but only the cost? Also, it may be required to ...
- 2 kudos
- 4604 Views
- 2 replies
- 2 kudos
Maximum Number of Parameters in Databricks SQL Queries
Hello Databricks Community,I'm working with Databricks SQL and encountered an issue when passing a large number of parameters in a query. Specifically, I attempted to pass 493 parameters, but I received the following error message:BAD_REQUEST : Too m...
- 4604 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @rumfox ,I would assume there is such a limit, the error is pretty clear. But the weird thing is, I cannot find any mention in documentation about it
- 2 kudos
- 1306 Views
- 1 replies
- 1 kudos
Delta Live Tables Unity Catalog Insufficient Permissions
I am receiving the following error when I try to run my DLT pipeline with unity catalog enabled.```raise Py4JJavaError( py4j.protocol.Py4JJavaError: An error occurred while calling o950.load. : org.apache.spark.SparkSecurityException: [INSUFFICIENT_P...
- 1306 Views
- 1 replies
- 1 kudos
- 1 kudos
I have also tried granting all permissions on the schema to myself and to all users and neither helped
- 1 kudos
- 852 Views
- 0 replies
- 0 kudos
BigQuery as foreign catalog - full object structs
Hi -We have mounted BigQuery, hosting Google Analytics data, as a foreign catalog.When querying the tables, objects are returned as strings, with all keys obfuscated by "f" or "v", likely to avoid replicating object keys across all records and hence ...
- 852 Views
- 0 replies
- 0 kudos
- 1043 Views
- 0 replies
- 0 kudos
Significant Delay in Deploying View on Unity Catalog Compared to Hive Metastore
Hi everyone,I'm experiencing a significant delay when deploying a view in the Unity Catalog compared to the Hive Metastore. Specifically, the deployment on the Unity Catalog takes 20 to 30 minutes, whereas the same deployment on the Hive Metastore co...
- 1043 Views
- 0 replies
- 0 kudos
- 2028 Views
- 3 replies
- 0 kudos
Decrease frequency of Databricks Asset Bundle API
We are using DABs for our deployment and to invoke any workflow. Behinds the scenes, it calls below API to get the status of workflow. Currently, it checks every few seconds. Is there a way to decrease this frequency from seconds to minutes. GET /ap...
- 2028 Views
- 3 replies
- 0 kudos
- 0 kudos
Hey! If the API is checking the job status too frequently, you might want to consider implementing a custom polling mechanism. Instead of relying on the default frequency, you can build a simple script or function that pauses for a longer interval b...
- 0 kudos
- 1814 Views
- 0 replies
- 0 kudos
Table Row Filter with a criteria on CURRENT_USER() belonging to a Unity Catalog group
HelloI defined a Row Filter to exclude some rows for a given user 'user@mail.com' in SQL.Instead of providing a list of users to exclude, I would like to define my criteria on Unity Catalog groups instead of users.Here is my current filter:-- apply f...
- 1814 Views
- 0 replies
- 0 kudos
- 4238 Views
- 1 replies
- 0 kudos
Error Sent message larger than max
Hello, I'm receiving a large amount of data in a dataframe, when trying to record or display it, I receive the error below. How can I fix it, or where do I change the setting?SparkConnectGrpcException: <_MultiThreadedRendezvous of RPC that terminated...
- 4238 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Retired_mod, I'm facing the same error. Can you provide the full name of the spark conf we have to change?Thank you
- 0 kudos
- 843 Views
- 0 replies
- 0 kudos
Critical/Important Spark Listener for performance tuning of spark code
Please suggest some key/important Listener which is most helpful in performance tuning of spark code.Kindly suggest how to use the listener in pyspark code to access the metrics as well for reference.
- 843 Views
- 0 replies
- 0 kudos
- 4900 Views
- 3 replies
- 2 kudos
Access Databricks API using IDP token
Hello,We have a databricks account & workspace, provided by AWS with SSO enabled. Is there any way to access databricks workspace API ( jobs/clusters, etc ) using a token retrieved from IdentityProvider ? We can access databricks workspace API with A...
- 4900 Views
- 3 replies
- 2 kudos
- 2 kudos
Hey - Costin and Anonymous user, have you managed to get this working, do you have examples by any chance?I'm also trying something similar but I haven't been able to make it work.> authenticate and access the Databricks REST API by setting the Autho...
- 2 kudos
- 2764 Views
- 2 replies
- 0 kudos
AWS NAT (Network Address Translation) Automated On-demand Destruct / Create
Hi folks, Our company typically uses Databrick during a 12 hour block, however the AWS NAT for elastic compute is up 24 hours, and I'd rather not pay for those hours.I gather AWS lambda and cloudwatch can be used to schedule / trigger NAT destruction...
- 2764 Views
- 2 replies
- 0 kudos
- 0 kudos
For interest, this is how I ended up solving the situation, with pointers from AWS support:<< CREATE NAT >>import boto3 import logging from datetime import datetime ec2 = boto3.client('ec2') cloudwatch = boto3.client('logs') def lambda_handler(even...
- 0 kudos
- 2662 Views
- 2 replies
- 1 kudos
How to update num_workers dynamically in a job cluster
I am setting up a workflows with the UI. In the first task, a dynamic value for the next task's num_workers is calculated based on actual data size. In the subsequent task, I'd like to use this calculated num_workers to update the job cluster's defau...
- 2662 Views
- 2 replies
- 1 kudos
- 1790 Views
- 1 replies
- 1 kudos
Discrepancy in Record Count in DLT Pipeline Data Quality Tab
Hello,I have set up a DLT pipeline where I ingest data from a Kafka topic into a table. Then, I create another table that filters records from the first table. However, I'm facing an issue:When I check the Data Quality tab for the second table, it sh...
- 1790 Views
- 1 replies
- 1 kudos
- 10074 Views
- 3 replies
- 5 kudos
Resolved! Driver: Out of Memory
Hi everyone,I have a streaming job with 29 notebooks that runs continuously. Initially, I allocated 28 GB of memory to the driver, but the job failed with a "Driver Out of Memory" error after 4 hours of execution.To address this, I increased the driv...
- 10074 Views
- 3 replies
- 5 kudos
- 5 kudos
It looks like _source_cdc_time is the timestamp for when the CDC transaction occurred in your source system. This would be a good choice for a timestamp column for your watermark, since you would be deduping values according to the time the transacti...
- 5 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
