Join discussions on data governance practices, compliance, and security within the Databricks Community. Exchange strategies and insights to ensure data integrity and regulatory compliance.
Hi Team,I have setup UC and have a Metastore for AUE. Assigned 2 workspaces to the AUE Metastore and both with different set of users and group . 1 workspace is the data provider and the other is a data consumerNow since 2 workspaces are in one Meta...
In my Databricks regular Clusters, Our clusters are configured to an External Hive Meta Store (Azure SQL Server). I am able to set the External Hive configuration in the Advanced options Spark config setting within the cluster setting. In our setting...
@RafaelGomez61 I have the some problem and after some exploring, I've configured it and also wrote a blog for the whole steps. You can have a look here: https://medium.com/@judy3.yang/how-to-configure-external-hive-meta-store-for-databricks-sql-wareh...
Hi guys, how are you ?How can I access tables outside the Databricks ? (hive metastore)I have a python script in my local structure but I need to access are tables stored in Databricks (hive metastore) , how ???? Have any idea ?Thank you guys
I would like to publish data from from delta live table (DLT) to an Azure ADLS Gen2 storage. Checked the documentation but cannot get it to work. Any insights on how to do so?I've tried to specify the "Storage location" with many combinations of abfs...
I am trying to load from ADLS with DLT, after add the spark configurations I got below errors:org.apache.spark.sql.streaming.StreamingQueryException: [STREAM_FAILED] Query [id = 818323fc-80d5-4833-9f46-7d1afc9c5bf7, runId = 722e9aac-0fdd-4206-9d49-68...
Hi @Janga Reddy Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers y...
Hi all,I am trying to write data from external s3 bucket to delta live tables at unity catalog( not dbfs ) from a workflow. I am seeing the following error ( saying catalog namespace is not supported. Please check full error below ) Error Detailspy4...
I dont know what is TAClbut in the unity catalog you can have governance like catalog.database.table featurewhere you have to explicitly give access to the databricks data user so you have to give him access separately for catalog and database and ta...
Hi,Does the Unity catalog support the Delta Live Tables? If yes, please help me with the documentation link. If not, any update on the feature readiness timeline would help. Thank you!
Hi @Venkadeshwaran K ,No, UC does not support DLT as of today. We have this feature on the timeline. We will be having a private preview by end of this year for DLT and UC integration for SQL. There is no ETA yet for GA but you can definitely expect...
Hi,I'm working with unity catalog for the last week. I'm refering to delta table by path, as follwing: path='s3://<my_bucket_name>/silver/data/<table_name>DeltaTable.forPath(spark, path)I get an exception that "is not a Delta table"using the table na...
Your guide to data and AI governance sessions at Data + AI Summit 2022We are so excited to welcome you to Data + AI Summit this year. This year’s hybrid conference is jam packed with exciting product announcements, live keynotes, breakout sessions wi...
Databricks Runtime 10.5 (Beta) Auto Loader: new SQL function CLOUD_FILES_STATEYou can use the new CLOUD_FILES_STATE function to query the internal state of an Auto Loader stream. Delta Lake: new maxRecordsPerFile option for maximum records written t...
anyone that got CLOUD_FILES_STATE to work on 10.5 Beta/Azure. Syntax:%sqlSELECT * FROM cloud_files_state('abfss://test@test.dfs.core.windows.net/test/checkpoint/'); i get this error: com.databricks.backend.common.rpc.DatabricksExceptions$SQLExecu...
Hi, I would like to use Delta Sharing but I have not understood how to configure the Delta Sharing Server. Is that fully managed by Databricks ? If so I tried to share a table :%sql
create share first_shareand I got the following error message with t...
Hi @Bertrand BURCKER ,This feature is not available yet. If you would like to test it or explore it, then you will need to sign up to get more details once the feature is available. In addition, you could reach out to your account team (in case you...
You could use delta sharing without unity catalog. It is an open protocol for secure real-time exchange of large datasets. There repo at https://github.com/delta-io/delta-sharing also has a reference implementation of the delta sharing server which c...
