- 4314 Views
- 1 replies
- 1 kudos
Resolved! Materilized view quite slower than table and lots of time on "Optimizing query & pruning files"
I have a query that calls different materialized views, anyway most of the time of the query is spent in "Optimizing query & pruning files" vs the execution.The difference is like 2-3 secs for the optimization and 300-400ms for the executionSimilar i...
- 4314 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi Roxio, How are you doing today?The difference in query times between materialized views and tables likely comes from the complexity of the views, as they often involve more steps in the background. To reduce the optimization time, you can try simp...
- 1 kudos
- 1171 Views
- 1 replies
- 0 kudos
Autloader error for assuming a role
Hi @Retired_mod I have seen numerous post by you. Thanks for continuously providing support. Can you or your colleagues help on this. We have a basic user which assumes a role with S3 policy to a specific bucket. When we try to read the bucket from D...
- 1171 Views
- 1 replies
- 0 kudos
- 0 kudos
Py4JJavaError: An error occurred while calling o503.json. : java.nio.file.AccessDeniedException: s3a://xxxxxx.json: shaded.databricks.org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by AwsCredentialContextTokenProvid...
- 0 kudos
- 1914 Views
- 1 replies
- 0 kudos
Databricks view error
Data source errorDataSource.Error: ODBC: ERROR [42000] [Microsoft][Hardy] (80) Syntax or semantic analysis error thrown in server while executing query.Error message from server: org.apache.hive.service.cli.HiveSQLException: Error running query: [DEL...
- 1914 Views
- 1 replies
- 0 kudos
- 0 kudos
This view used to run till last week and suddenly started giving this error. So need to check what can be the reason for this issue
- 0 kudos
- 6520 Views
- 4 replies
- 4 kudos
Incorrect secret value when loaded as environment variable
I recently faced an issue that took good hours to identify. I'm loading an environment variable with a secretENVVAR: {{secrets/scope/key}}The secret is loaded in my application, I could verify it's there, but its value is not correct. I realised tha...
- 6520 Views
- 4 replies
- 4 kudos
- 6681 Views
- 5 replies
- 5 kudos
Resolved! IDENTIFIER not working in UPDATE
The following code works perfectly fine: df = spark.createDataFrame([('A', 1), ('B', 2)]) df.createOrReplaceTempView('temp') spark.sql(""" SELECT IDENTIFIER(:col) FROM temp """, args={ "col": "_1" } ).display(...
- 6681 Views
- 5 replies
- 5 kudos
- 5 kudos
If it helps anyone else I found this article that described a few limitations:https://community.databricks.com/t5/technical-blog/how-not-to-build-an-execute-immediate-demo/ba-p/82167
- 5 kudos
- 3596 Views
- 3 replies
- 3 kudos
Resolved! RESOURCE_EXHAUSTED dbutils.jobs.taskValues.get
I've a job in Databricks running multiple tasks in parallel. Those tasks read parameters of the job using the utility of dbutils. I'm getting the following error when trying to read parameters in my different tasks:com.databricks.common.client.Databr...
- 3596 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi all,Our solution has been to use job parameters and dynamic value references. These are read using dbutils.widgets.get() instead of dbutils.jobs.taskValues.get(). Now, our ETL is working well again.Pass context about job runs into job tasks - Azur...
- 3 kudos
- 3485 Views
- 3 replies
- 0 kudos
Unity Catalog Lineage Not Working on GCP
Hello,We have set up a lakehouse in Databricks for one of our clients. One of the features our client would like to use is the Unity Catalog data lineage view. This is a handy feature that we have used with other clients (in both AWS and Azure) witho...
- 3485 Views
- 3 replies
- 0 kudos
- 0 kudos
Hello,It's been a few months since this exchange. The feature limitation is not documented anywhere - documents imply that this should be working in GCP:https://docs.gcp.databricks.com/en/data-governance/unity-catalog/data-lineage.htmlIs this feature...
- 0 kudos
- 9831 Views
- 5 replies
- 4 kudos
Import module never ends on random branches
Hello,Since a week ago, our notebook are stuck in running on the firsts cells which import python module from our github repository which is cloned in databricks.The cells stays in running state and when we try to manually cancel the jobs in databric...
- 9831 Views
- 5 replies
- 4 kudos
- 13757 Views
- 6 replies
- 10 kudos
Resolved! Spark - API Jira
Hello guys. I use pyspark in my daily life. A demand has arisen to collect information in Jira. I was able to do this via Talend ESB, but I wouldn't want to use different tools to get the job done. Do you have any example of how to extract data from ...
- 13757 Views
- 6 replies
- 10 kudos
- 10 kudos
Hi,There is also a new Databricks for Jira add-on on the Atlassian Marketplace. It is easy to setup and exports are directly created within Jira. They can be one-time, scheduled, or real-time. It can also export additional Jira data such as Assets, C...
- 10 kudos
- 11124 Views
- 1 replies
- 0 kudos
Resolved! [INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error
Hello,I am currently working through an issue I am seeing when querying a Lakehouse Federation UC table in a workflow. I am using pyspark to query a table through Lakehouse Federation which returns a result based on the query. When running it in a ce...
- 11124 Views
- 1 replies
- 0 kudos
- 0 kudos
I hit a similar error when trying to view FC data using a cluster with a lower Databricks Runtime.Needs to be DBR 13.1 or higher.
- 0 kudos
- 18502 Views
- 6 replies
- 2 kudos
How to transfer ownership of a database and/or table?
We created a new Service Principal (SP) on Azure and would like to transfer the ownership of the databases and tables created with the old SP. The issue is that these databases and tables are not visible to the users using the new SP.I am using a Hiv...
- 18502 Views
- 6 replies
- 2 kudos
- 2 kudos
Regarding the [PARSE_SYNTAX_ERROR] Syntax error at or near 'OWNER'.Remember to wrap the new owner name in the SQL statement with the Grave Accent (`) as the below sample. ALTER SCHEMA schema_name OWNER TO `new_oner_name`;
- 2 kudos
- 4441 Views
- 1 replies
- 0 kudos
Resolved! Hanging/frozen cancelling/running cells in notebook
I'm constantly getting frozen "Cancelling" or "Running" cells since a few days when prototyping some stuff in notebooks. Examples:And the same for instantiating functions (should be instant), neither finishing or able to cancelThis button doesn't wor...
- 4441 Views
- 1 replies
- 0 kudos
- 0 kudos
Seems to have resolved itself today. Can't reproduce it anymore
- 0 kudos
- 4181 Views
- 1 replies
- 0 kudos
How to create a single CSV file with specified file name Spark in Databricks?
I know how to use Spark in Databricks to create a CSV, but it always has lots of side effects.For example, here is my code:file_path = “dbfs:/mnt/target_folder/file.csv”df.write.mode("overwrite").csv(file_path, header=True)Then what I got isA folder ...
- 4181 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @guangyi ,To disable _commited_xxx, _started_xxx and _SUCCSSS you must set below spark options: spark.conf.set("spark.databricks.io.directoryCommit.createSuccessFile","false") spark.conf.set("mapreduce.fileoutputcommitter.marksuccessfuljobs", "f...
- 0 kudos
- 32232 Views
- 5 replies
- 2 kudos
org.apache.spark.sql.AnalysisException: Undefined function: 'MAX'
I am trying to create a JAR for a Azure Databricks job but some code that works when using the notebook interface does not work when calling the library through a job. The weird part is that the job will complete the first run successfully but on an...
- 32232 Views
- 5 replies
- 2 kudos
- 2 kudos
I am facing similar issue when trying to use from_utc_timestamp function. I am able to call the function from databricks notebook but when I use the same function inside my java jar and running as a job in databricks, it is giving below error. Analys...
- 2 kudos
- 6483 Views
- 2 replies
- 0 kudos
- 6483 Views
- 2 replies
- 0 kudos
- 0 kudos
Yes! Cluster's owner/creator can be changed with the REST API - POST /api/2.1/clusters/change-ownerRequest Body sample:{ "cluster_id": "string", "owner_username": "string" }Ref: Clusters API | Change cluster ownerHope this helps!
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 579 | |
| 349 | |
| 287 |
