- 1960 Views
- 3 replies
- 0 kudos
Unity Catalog Lineage Not Working on GCP
Hello,We have set up a lakehouse in Databricks for one of our clients. One of the features our client would like to use is the Unity Catalog data lineage view. This is a handy feature that we have used with other clients (in both AWS and Azure) witho...
- 1960 Views
- 3 replies
- 0 kudos
- 0 kudos
Hello,It's been a few months since this exchange. The feature limitation is not documented anywhere - documents imply that this should be working in GCP:https://docs.gcp.databricks.com/en/data-governance/unity-catalog/data-lineage.htmlIs this feature...
- 0 kudos
- 8101 Views
- 5 replies
- 4 kudos
Import module never ends on random branches
Hello,Since a week ago, our notebook are stuck in running on the firsts cells which import python module from our github repository which is cloned in databricks.The cells stays in running state and when we try to manually cancel the jobs in databric...
- 8101 Views
- 5 replies
- 4 kudos
- 1547 Views
- 0 replies
- 0 kudos
How to implement Lifecycle of Data When Use ADLS
Hello everyone, nice to greet you. I have a question about the data lifecycle in ADLS. I know ADLS has its own rules, but they aren't working properly because I have two ADLS accounts: one for hot data and another for cool storage where the informati...
- 1547 Views
- 0 replies
- 0 kudos
- 8962 Views
- 6 replies
- 10 kudos
Resolved! Spark - API Jira
Hello guys. I use pyspark in my daily life. A demand has arisen to collect information in Jira. I was able to do this via Talend ESB, but I wouldn't want to use different tools to get the job done. Do you have any example of how to extract data from ...
- 8962 Views
- 6 replies
- 10 kudos
- 10 kudos
Hi,There is also a new Databricks for Jira add-on on the Atlassian Marketplace. It is easy to setup and exports are directly created within Jira. They can be one-time, scheduled, or real-time. It can also export additional Jira data such as Assets, C...
- 10 kudos
- 7192 Views
- 1 replies
- 0 kudos
Resolved! [INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error
Hello,I am currently working through an issue I am seeing when querying a Lakehouse Federation UC table in a workflow. I am using pyspark to query a table through Lakehouse Federation which returns a result based on the query. When running it in a ce...
- 7192 Views
- 1 replies
- 0 kudos
- 0 kudos
I hit a similar error when trying to view FC data using a cluster with a lower Databricks Runtime.Needs to be DBR 13.1 or higher.
- 0 kudos
- 15016 Views
- 6 replies
- 2 kudos
How to transfer ownership of a database and/or table?
We created a new Service Principal (SP) on Azure and would like to transfer the ownership of the databases and tables created with the old SP. The issue is that these databases and tables are not visible to the users using the new SP.I am using a Hiv...
- 15016 Views
- 6 replies
- 2 kudos
- 2 kudos
Regarding the [PARSE_SYNTAX_ERROR] Syntax error at or near 'OWNER'.Remember to wrap the new owner name in the SQL statement with the Grave Accent (`) as the below sample. ALTER SCHEMA schema_name OWNER TO `new_oner_name`;
- 2 kudos
- 2633 Views
- 1 replies
- 0 kudos
Resolved! Hanging/frozen cancelling/running cells in notebook
I'm constantly getting frozen "Cancelling" or "Running" cells since a few days when prototyping some stuff in notebooks. Examples:And the same for instantiating functions (should be instant), neither finishing or able to cancelThis button doesn't wor...
- 2633 Views
- 1 replies
- 0 kudos
- 0 kudos
Seems to have resolved itself today. Can't reproduce it anymore
- 0 kudos
- 2313 Views
- 1 replies
- 0 kudos
How to create a single CSV file with specified file name Spark in Databricks?
I know how to use Spark in Databricks to create a CSV, but it always has lots of side effects.For example, here is my code:file_path = “dbfs:/mnt/target_folder/file.csv”df.write.mode("overwrite").csv(file_path, header=True)Then what I got isA folder ...
- 2313 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @guangyi ,To disable _commited_xxx, _started_xxx and _SUCCSSS you must set below spark options: spark.conf.set("spark.databricks.io.directoryCommit.createSuccessFile","false") spark.conf.set("mapreduce.fileoutputcommitter.marksuccessfuljobs", "f...
- 0 kudos
- 29815 Views
- 5 replies
- 2 kudos
org.apache.spark.sql.AnalysisException: Undefined function: 'MAX'
I am trying to create a JAR for a Azure Databricks job but some code that works when using the notebook interface does not work when calling the library through a job. The weird part is that the job will complete the first run successfully but on an...
- 29815 Views
- 5 replies
- 2 kudos
- 2 kudos
I am facing similar issue when trying to use from_utc_timestamp function. I am able to call the function from databricks notebook but when I use the same function inside my java jar and running as a job in databricks, it is giving below error. Analys...
- 2 kudos
- 4396 Views
- 2 replies
- 0 kudos
- 4396 Views
- 2 replies
- 0 kudos
- 0 kudos
Yes! Cluster's owner/creator can be changed with the REST API - POST /api/2.1/clusters/change-ownerRequest Body sample:{ "cluster_id": "string", "owner_username": "string" }Ref: Clusters API | Change cluster ownerHope this helps!
- 0 kudos
- 2061 Views
- 4 replies
- 0 kudos
How to solve udf performance issue with databricks sql function?
Hi,I am dealing with a situation where i need to secure data at rest on storage (azure data lake), hence saving the data as encrypted text into the delta table. While serving, i want to create dynamic view which will be created from the delta table a...
- 2061 Views
- 4 replies
- 0 kudos
- 0 kudos
I don't clearly understand your full problem. But I do know the following regarding UDFs:1. PySpark UDFs are extremely slow, because it needs to deserialize the Java Object (DataFrame), transform it with the Python UDF, then Serialize it back. This h...
- 0 kudos
- 839 Views
- 3 replies
- 0 kudos
Unity Catalog external tables
What are the consistency guarantees that DataBricks supply for multi writers, given that the written table is an external table?Are they different from the consistency guarantees given for managed tables? Thanks!
- 839 Views
- 3 replies
- 0 kudos
- 0 kudos
Thank you @Ajay-Pandey, that is helpful.One thing that I'm not sure about is how does DataBricks can use the same ACID mechanism that external tools use with the external tables? For example, if an external Spark cluster write Delta Logs with a LogSt...
- 0 kudos
- 2201 Views
- 1 replies
- 0 kudos
Resolved! Authentication Issue while connecting to Databricks using Looker Studio
So previously I created source connections from looker with Databricks using my personal access token.I followed this databricks docs. https://docs.databricks.com/en/partners/bi/looker-studio.htmlBut from 10 July, I think basic authentication has bee...
- 2201 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi,You would still connect using OAuth tokens. It is just that Databricks recommends using personal access tokens belonging to service principals instead of workspace users. To create tokens for service principals, see Manage tokens for a service pri...
- 0 kudos
- 2808 Views
- 8 replies
- 1 kudos
Resolved! Question: Issue with Overwatch Deployment on Databricks (on AWS) - Missing Tables in Gold Schema
Hi all,I'm working on setting up Overwatch in our Databricks workspace to monitor resources, and I've encountered an issue during overwatch deployment. I am able to deploy overwatch, but the validation for the `Gold_jobRunCostPotentialFact` module fa...
- 2808 Views
- 8 replies
- 1 kudos
- 1 kudos
Hi @jiteshraut20 , 1) Storage_prefix: It is updated in the documents please reffer config . 2)If the system table is in use, the recommended Databricks runtime version is 13.3 LTS. For other cases, 11.3 LTS should work seamlessly. Please see the docu...
- 1 kudos
- 2947 Views
- 2 replies
- 6 kudos
Getting error popup in databricks
when i migrated to new databricks workspace, I am getting error popup message continuously and also indentation what I changed it is getting changed to other value every with new login .
- 2947 Views
- 2 replies
- 6 kudos
- 6 kudos
Hi I am facing this issue session expired pop-ups frequently every 3 minutes while I start working on databricks.
- 6 kudos
| User | Count |
|---|---|
| 1613 | |
| 770 | |
| 349 | |
| 286 | |
| 253 |
