Join discussions on data governance practices, compliance, and security within the Databricks Community. Exchange strategies and insights to ensure data integrity and regulatory compliance.
Hello,Im using Unity Catalog managed tables and Unity Catalog DLTs. I urgently need to use a UDF in DBSQL because pySpark UDFs dont work in UC. I found this way to create a SQL function and use python language inside:CREATE FUNCTION redact(a STRING)
...
It will be available soon in public preview. You will need to have a Pro SQL Warehouse or a Serverless SQL Warehouse to use Python UDF in Databricks SQL.
I have databricks premium and I want to enable unity catalog on it, I am not able to do so. I followed the following steps which are attached in the attachments. I am also attaching the screenshot of my cluster configuration. When I run this command:...
Hi @Prerna Munjal Help us build a vibrant and resourceful community by recognizing and highlighting insightful contributions. Mark the best answers and show your appreciation!Regards
maybe this is a bug in the unity catalog as the error happens multiple times when a table becomes inaccessibleerror: Request failed with status code 404try to drop the table error: NoSuchTableException: [TABLE_OR_VIEW_NOT_FOUND] try to replace the ta...
the error disappeared now after a few months.It could behappens when there is a lot of schema change on db during the initial project stage.there may be some temp security issuesmay be databricks fixed in the back end appears less issue when schema r...
HiWe have faced an issue recently (everything worked well 1 week before)We are using exnernal hive metastore:"spark.hadoop.datanucleus.connectionPool.maxPoolSize" : "10"
"spark.sql.hive.metastore.jars" : "/dbfs/ivyjars/*"
"spark.sql.hive.metastor...
Hey guys,I am trying to create a delta live table in Unity Catalog as follows:CREATE OR REFRESH STREAMING LIVE TABLE <catalog>.<db>.<table_name> AS SELECT ... However, I get the error: org.apache.spark.sql.AnalysisException: Unsupported SQL statemen...
Hello, I am trying to test the new federation for Snowflake feature in Databricks SQL, following the docs (https://docs.databricks.com/query-federation/snowflake.html).Environment:Subscription: PremiumCompute: DB SQL (serverless)Metastore: Unity Cata...
Hi @Landan George, thanks for the quick response. I get the following error if I try to run the SQL code from a Notebook:SQLException: No suitable driver found for jdbc:snowflake://app.snowflake.com/region/foo/Any suggestions?
Error in SQL statement: AnalysisException: SHOW PARTITIONS is not supported in Unity Catalog;What happened with the SHOW PARTITIONS command?Is this information available anywhere in the documentation?If I go to Databricks SQL documentation there is n...
I got answer from the support that there is no 'SHOW PARTITIONS' in Unity Catalog as "UC doesn't manage the partitions for tables".They suggested for work around `LIST` command, this is what I was able to came up with.The problem I see here, is for t...
Hi All, does anyone know the best way to provided feedback to the databricks team in regards to feature request or enhancements?This is a general question but I have immediate items I'm wishing to have improved: In the new data explorer for unity cat...
Hi @Ashley Betts You can open a feature request (Aha Idea) at https://databricks.aha.io/ Ideas registered here are reviewed by our product management team and will be picked up for implementation in the roadmap.The ideas here are generally listed fo...
I am trying to setup a Databricks SQL endpoint to connect to an external hive metastore. I have replicated my metastore spark config in the SQL endpoint, with the addition of adding the below to the configuration:spark.sql.hive.metastore.jars maven I...
Hi @SETH JENSEN , Just checking if you were able to solve this issue? If no, Can you try to modify the connection string and make sslMode=Require and see if it works?
I have created a premium cluster on Azure site. There were no problem Data Science and Engineering (DSAE) while i was binding Postgre SQL hive metastore. I have done all settings via Global init Scripts from Admin Console.However when I try to adjust...
Find soluttion :spark.hadoop.javax.jdo.option.ConnectionURL jdbc:postgresql://#########.postgres.database.azure.com:5432/databricks_metastore?ssl=trueneeds to be spark.hadoop.javax.jdo.option.ConnectionURL jdbc:postgresql://#########.postgres.databas...
