- 13528 Views
- 6 replies
- 0 kudos
"Unable to upload to DBFS Query" Error running SQL warehouse query?
I have sql warehouse endpoints that work fine when querying from applications such as Tableau, but just running the included sample query against a running endpoint from the Query Editor from the workspace is returning "Unable to upload to DBFS Query...
- 13528 Views
- 6 replies
- 0 kudos
- 0 kudos
Hi @Marvin Ginns​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...
- 0 kudos
- 2418 Views
- 1 replies
- 1 kudos
For dummies: How to avoid 'bill shock' & control AWS charges while learning to use Databricks?
Hi, I'm an out-of-work data analyst wanting to re-skill as a 'citizen data engineer'. By following how-to guides I was able to set up my own Databricks account etc along with a personal VPC in AWS. After 2 weeks of problem-free training I checked my...
- 2418 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi Hantha Databricks needs VPCs to work, but there are the default ones and customer managed ones: https://docs.databricks.com/en/security/network/classic/customer-managed-vpc.htmlCustomer managed ones are optional, but many tutorials include them as...
- 1 kudos
- 2990 Views
- 2 replies
- 0 kudos
Resolved! Row Filter on Unity Catalog Tables based on Unity Catalog group appartenance
Hello,I would like to prevent users belonging to a given Unity Catalog group ('restricted_users_group') to access some rows of a Unity Catalog Table.For now, I was able to define a Row Filter function to prevent a list of users to access some rows, t...
- 2990 Views
- 2 replies
- 0 kudos
- 0 kudos
Ok, so this problem needs no tricks. All was in the documentationI did not know about the function IS_ACCOUNT_GROUP_MEMBER(). So this Row Filter function did the job:CREATE FUNCTION rd.my_schema.my_row_filter(filter_column INTEGER) RETURNS BOOLEANRET...
- 0 kudos
- 1909 Views
- 2 replies
- 3 kudos
Resolved! Applying Row Filters to a table removes the ability to DEEP CLONE or SHALLOW CLONE this table
Hello,In this documentation I see some limitations coming with using Row Filters, like "Deep and shallow clones are not supported"We plan to use these Row Filters to hide sensitive data to some users.But not having CLONE available for tables with Row...
- 1909 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi Antoine_B The document explicitly calls out these limitations but there is no explicit mention of the roadmap for support. I would request you to reachout to the Account Executive and SAs so that they can suggest the alternative approach for now. ...
- 3 kudos
- 3065 Views
- 1 replies
- 1 kudos
Resolved! Preferred way to read S3 - dbutils or Boto3 or better solution ?
We have a usecase where table has 15K rows , one of the column has S3 location. We need to read each row from table and fetch s3 location from one of the column,read its content from s3. To read the content from S3 , workflow is taking lot of time, ...
- 3065 Views
- 1 replies
- 1 kudos
- 1 kudos
Create IAM role in AWS S3 and use those credentials to connect to Databricks by using the below codeAWS_SECRET_ACCESS_KEY={{secrets/scope/aws_secret_access_key}}AWS_ACCESS_KEY_ID={{secrets/scope/aws_access_key_id}}aws_bucket_name = "my-s3-bucket"df =...
- 1 kudos
- 1747 Views
- 2 replies
- 2 kudos
How to remove specific data from bronze layer
Hello Team,At my end, we bring data from Kafka and through autoloader we ingest it in Bronze layer and further to silver and silver plus layer. Lately due to business changes, we need to delete specific data from the bronze and silver layer due to da...
- 1747 Views
- 2 replies
- 2 kudos
- 2 kudos
Hello,Thankyou for your reply on my query!My bronze layer has data in json format and currently, I need to remove 400 records from it. I also have job set up in streaming mode. Could you please suggest how I can go further with it?
- 2 kudos
- 2042 Views
- 2 replies
- 0 kudos
Query on using secret scope for dbt-core integration with databricks workflow
Hello all,I am currently configuring dbt-core with Azure Databricks Workflow and using Azure Databricks M2M (Machine-to-Machine) authentication for this setup. I have the cluster ID and cluster secret ID stored in Databricks secret scope.I am seeking...
- 2042 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Ram-Dev7 , Thank you for reaching out to our community! We're here to help you. To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedbac...
- 0 kudos
- 2090 Views
- 1 replies
- 2 kudos
DLT Serverless costs
I recently started checking out the serverless delta live tables. In my understanding serverless continuous jobs (with autoloader) would only do something when new files arrive. However for serverless 4 pipelines running continuously, I spend in two ...
- 2090 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @mbaas , That does not sound right. So, were you able to compare the jobs, stages and spot the difference. Are there more tasks added, or with compute perspective you do not find a difference at all but only the cost? Also, it may be required to ...
- 2 kudos
- 4112 Views
- 2 replies
- 2 kudos
Maximum Number of Parameters in Databricks SQL Queries
Hello Databricks Community,I'm working with Databricks SQL and encountered an issue when passing a large number of parameters in a query. Specifically, I attempted to pass 493 parameters, but I received the following error message:BAD_REQUEST : Too m...
- 4112 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @rumfox ,I would assume there is such a limit, the error is pretty clear. But the weird thing is, I cannot find any mention in documentation about it
- 2 kudos
- 1105 Views
- 1 replies
- 1 kudos
Delta Live Tables Unity Catalog Insufficient Permissions
I am receiving the following error when I try to run my DLT pipeline with unity catalog enabled.```raise Py4JJavaError( py4j.protocol.Py4JJavaError: An error occurred while calling o950.load. : org.apache.spark.SparkSecurityException: [INSUFFICIENT_P...
- 1105 Views
- 1 replies
- 1 kudos
- 1 kudos
I have also tried granting all permissions on the schema to myself and to all users and neither helped
- 1 kudos
- 744 Views
- 0 replies
- 0 kudos
BigQuery as foreign catalog - full object structs
Hi -We have mounted BigQuery, hosting Google Analytics data, as a foreign catalog.When querying the tables, objects are returned as strings, with all keys obfuscated by "f" or "v", likely to avoid replicating object keys across all records and hence ...
- 744 Views
- 0 replies
- 0 kudos
- 866 Views
- 0 replies
- 0 kudos
Significant Delay in Deploying View on Unity Catalog Compared to Hive Metastore
Hi everyone,I'm experiencing a significant delay when deploying a view in the Unity Catalog compared to the Hive Metastore. Specifically, the deployment on the Unity Catalog takes 20 to 30 minutes, whereas the same deployment on the Hive Metastore co...
- 866 Views
- 0 replies
- 0 kudos
- 1687 Views
- 3 replies
- 0 kudos
Decrease frequency of Databricks Asset Bundle API
We are using DABs for our deployment and to invoke any workflow. Behinds the scenes, it calls below API to get the status of workflow. Currently, it checks every few seconds. Is there a way to decrease this frequency from seconds to minutes. GET /ap...
- 1687 Views
- 3 replies
- 0 kudos
- 0 kudos
Hey! If the API is checking the job status too frequently, you might want to consider implementing a custom polling mechanism. Instead of relying on the default frequency, you can build a simple script or function that pauses for a longer interval b...
- 0 kudos
- 1458 Views
- 0 replies
- 0 kudos
Table Row Filter with a criteria on CURRENT_USER() belonging to a Unity Catalog group
HelloI defined a Row Filter to exclude some rows for a given user 'user@mail.com' in SQL.Instead of providing a list of users to exclude, I would like to define my criteria on Unity Catalog groups instead of users.Here is my current filter:-- apply f...
- 1458 Views
- 0 replies
- 0 kudos
- 3933 Views
- 1 replies
- 0 kudos
Error Sent message larger than max
Hello, I'm receiving a large amount of data in a dataframe, when trying to record or display it, I receive the error below. How can I fix it, or where do I change the setting?SparkConnectGrpcException: <_MultiThreadedRendezvous of RPC that terminated...
- 3933 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Retired_mod, I'm facing the same error. Can you provide the full name of the spark conf we have to change?Thank you
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 555 | |
| 349 | |
| 287 |
