Here's your Data + AI Summit 2024 - Data Governance recap as you navigate the explosion of AI, data and tools in efforts to build a flexible and scalable governance framework that spans your entire data and AI estate. Keynote: Evolving Data Governan...
HiWe have faced an issue recently (everything worked well 1 week before)We are using exnernal hive metastore:"spark.hadoop.datanucleus.connectionPool.maxPoolSize" : "10" "spark.sql.hive.metastore.jars" : "/dbfs/ivyjars/*" "spark.sql.hive.metastor...
Generally, this should not be a case. Please raise a support request to databricks
I dont know what is TAClbut in the unity catalog you can have governance like catalog.database.table featurewhere you have to explicitly give access to the databricks data user so you have to give him access separately for catalog and database and ta...
We're trying to use Notebook widgets in DBR LTS 10.4 (Shared Mode) with Unity Catalogbut we're running into following error.Because of this our notebook dashboard are not working after migrating our data to Unity catalog.Following is the queryCREATE...
unity catalog is different and Data science panel is different , in the above notebook you have to use formatted or this widget will work there and in the unity catalog you have to use get argument formatted in the query
Zipped csv files are receiving to s3 raw layer.
@Jog Giri​ I also recently encountered a similar scenario, the below code solved my purpose without any issues.import zipfile for i in dbutils.fs.ls('/mnt/zipfilespath/'): with zipfile.ZipFile(i.path.replace('dbfs:','/dbfs'), mode="r") as zip_ref:...
I was trying to understand Unity Catalog, if metastore is at the account level or the each workspace in the account will have there own metastore? My understanding is the metastore cant be shared between two workspace.
@Sanjay Janardhan​ , no i have done this, you can share it in multiple workspaces
How prevent users from being able to create NEW databases? Possible without Unity Catalog?I'm working on access control and I know how to remove select and modify privileges on schemas and tables but that's it.
Hi @Christian Seberino​,Hope everything is going great.Thank you for reaching out to the community with your question. Would you be happy to mark an answer as best so that other members can find the solution more quickly?Cheers!
Hey guys,I am trying to create a delta live table in Unity Catalog as follows:CREATE OR REFRESH STREAMING LIVE TABLE <catalog>.<db>.<table_name> AS SELECT ... However, I get the error: org.apache.spark.sql.AnalysisException: Unsupported SQL statemen...
@Anant Pingle​ As of right now, DLT doesn't support Unity Catalog yet. We can expect this feature in upcomig days.
HiI have just created a metastore to allow unity catalog to be enabled for my workspace. However I am getting the following error when trying to create a table as per the unity catalog demo notbook:Error in SQL statement: UnauthorizedAccessException:...
Thanks for your answer to the second part of my question, super helpful! Can you confirm the first part of my question as well please?
Hello, I am trying to test the new federation for Snowflake feature in Databricks SQL, following the docs (https://docs.databricks.com/query-federation/snowflake.html).Environment:Subscription: PremiumCompute: DB SQL (serverless)Metastore: Unity Cata...
Hi @Landan George​, thanks for the quick response. I get the following error if I try to run the SQL code from a Notebook:SQLException: No suitable driver found for jdbc:snowflake://app.snowflake.com/region/foo/Any suggestions?
I could able to save data using pyspark into S3 but not sure on how to save a file stream object into S3 bucket using pyspark. I could achieve this with help of python but when Unity catalog was enabled on Databrciks it always ends up with an access ...
I got to know that there is a change required at Unity-catalog to make it work with Python and got a recommendation to use pyspark to store file into S3.I do not see much information about storing a file stream object in an S3 bucket anywhere. Can an...
How can we connect to unity catalog tables to create REST API's using Python/.NET?
We are setting up Databricks on our AWS account using customer managed VPC and enabling private link. I could see the private endpoints for frontend and backend with which we could spin up our EC2 instances. But the problem we are facing is to connec...
Este triángulo amoroso se puso complicado... ¡Faltan solo dÃas para el estreno de la segunda temporada de "Wildrose" en flixlatino!
Can we use existing AWS Glue catalog with Unity Catalog?
Hi @Rahul Mishra​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...
I have discovered some strange behavior when adding new columns to a table associated with Unity Catalog. I overwrote the data specifying overwriteSchema = True and passing in the abfss path as it is an external table. When quering against it, it is ...
Hi @Oscar Dyremyhr​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...
