Hi -We have mounted BigQuery, hosting Google Analytics data, as a foreign catalog.When querying the tables, objects are returned as strings, with all keys obfuscated by "f" or "v", likely to avoid replicating object keys across all records and hence ...
Hi everyone,I'm experiencing a significant delay when deploying a view in the Unity Catalog compared to the Hive Metastore. Specifically, the deployment on the Unity Catalog takes 20 to 30 minutes, whereas the same deployment on the Hive Metastore co...
We are using DABs for our deployment and to invoke any workflow. Behinds the scenes, it calls below API to get the status of workflow. Currently, it checks every few seconds. Is there a way to decrease this frequency from seconds to minutes. GET /ap...
Hey! If the API is checking the job status too frequently, you might want to consider implementing a custom polling mechanism. Instead of relying on the default frequency, you can build a simple script or function that pauses for a longer interval b...
HelloI defined a Row Filter to exclude some rows for a given user 'user@mail.com' in SQL.Instead of providing a list of users to exclude, I would like to define my criteria on Unity Catalog groups instead of users.Here is my current filter:-- apply f...
Hello, I'm receiving a large amount of data in a dataframe, when trying to record or display it, I receive the error below. How can I fix it, or where do I change the setting?SparkConnectGrpcException: <_MultiThreadedRendezvous of RPC that terminated...
Hi @Retired_mod, I'm facing the same error. Can you provide the full name of the spark conf we have to change?Thank you
Please suggest some key/important Listener which is most helpful in performance tuning of spark code.Kindly suggest how to use the listener in pyspark code to access the metrics as well for reference.
Hello,We have a databricks account & workspace, provided by AWS with SSO enabled. Is there any way to access databricks workspace API ( jobs/clusters, etc ) using a token retrieved from IdentityProvider ? We can access databricks workspace API with A...
Hey - Costin and Anonymous user, have you managed to get this working, do you have examples by any chance?I'm also trying something similar but I haven't been able to make it work.> authenticate and access the Databricks REST API by setting the Autho...
Hi folks, Our company typically uses Databrick during a 12 hour block, however the AWS NAT for elastic compute is up 24 hours, and I'd rather not pay for those hours.I gather AWS lambda and cloudwatch can be used to schedule / trigger NAT destruction...
For interest, this is how I ended up solving the situation, with pointers from AWS support:<< CREATE NAT >>import boto3 import logging from datetime import datetime ec2 = boto3.client('ec2') cloudwatch = boto3.client('logs') def lambda_handler(even...
I am setting up a workflows with the UI. In the first task, a dynamic value for the next task's num_workers is calculated based on actual data size. In the subsequent task, I'd like to use this calculated num_workers to update the job cluster's defau...
Hello,I have set up a DLT pipeline where I ingest data from a Kafka topic into a table. Then, I create another table that filters records from the first table. However, I'm facing an issue:When I check the Data Quality tab for the second table, it sh...
Hi everyone,I have a streaming job with 29 notebooks that runs continuously. Initially, I allocated 28 GB of memory to the driver, but the job failed with a "Driver Out of Memory" error after 4 hours of execution.To address this, I increased the driv...
It looks like _source_cdc_time is the timestamp for when the CDC transaction occurred in your source system. This would be a good choice for a timestamp column for your watermark, since you would be deduping values according to the time the transacti...
I am performing some tests with delta tables. For each test, I write a delta table to Azure Blob Storage. Then I manually delete the delta table. After deleting the table and running my code again, I get this error: AnalysisException: [PATH_NOT_FOUN...
yes it is weird , workaround for thisfiles = dbutils.fs.ls("s3 bucket or azure blob path")file_paths = [file.path for file in files]if target_path not in file_paths: dbutils.fs.mkdirs(target_path)
I am using a cluster in databricks to connect to a Tableau workbook through the JDBC connector. My Tableau workbook has been unable to load due to resources not being available through the data connection. I went to look at the driver log for my clus...
To cache/persist an action needs to be triggered. I'm just wondering, will it make any difference if, after persisting some df, I use, for instance, take(5) instead of count()?Will it be a bit more effective, because of sending results from 5 partiti...
Yes take (5) will be more efficient in some ways.When you cache or persist a DataFrame in Spark, you are instructing Spark to store the DataFrame's intermediate data in memory (or on disk, depending on the storage level). This can significantly speed...
Hi allI've used mounts based on service principals but users using shaed clusters or the new serverless they have problems with permissions to access resources on dbfs. Right now we have used clusters in single modeWhat should be the best approach to...
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 566 | |
| 349 | |
| 287 |
