Here's your Data + AI Summit 2024 - Data Governance recap as you navigate the explosion of AI, data and tools in efforts to build a flexible and scalable governance framework that spans your entire data and AI estate. Keynote: Evolving Data Governan...
hi Team ,We are using a shared hive metastore for connecting to dev and prod as we need cross DB access for certain ML use-cases.Currently we are having issues as "DROP TABLE COMMAND" directly drops the table without checking the underlying locationH...
Hi All,I am facing the GC metadata issue while performing distributed computing on Spark. 2022-01-13T22:02:28.467+0000: [GC (Metadata GC Threshold) [PSYoungGen: 458969K->18934K(594944K)] 458969K->18958K(1954816K), 0.0144028 secs] [Times: user=0.05 sy...
Hi @Chandan Angadi​ Hope you are doing well now.Thanks for getting back to us and sending in your solution. Would you like to mark an answer as best?Thanks!
Databricks Runtime 10.5 (Beta) Auto Loader: new SQL function CLOUD_FILES_STATEYou can use the new CLOUD_FILES_STATE function to query the internal state of an Auto Loader stream. Delta Lake: new maxRecordsPerFile option for maximum records written t...
anyone that got CLOUD_FILES_STATE to work on 10.5 Beta/Azure. Syntax:%sqlSELECT * FROM cloud_files_state('abfss://test@test.dfs.core.windows.net/test/checkpoint/'); i get this error: com.databricks.backend.common.rpc.DatabricksExceptions$SQLExecu...
I am adding a credential to access MongoDB through the Python MongoDB Spark connector. It is stored as a secret and the `users` group has `MANAGE` ACL privileges. I turned on table ACLs and created a `High Concurrency` compute cluster. I then run a s...
@Erik Louie​ looks the issue is resolved. Please let me know if you need further help on this. Thanks!
Hi, I would like to use Delta Sharing but I have not understood how to configure the Delta Sharing Server. Is that fully managed by Databricks ? If so I tried to share a table :%sql create share first_shareand I got the following error message with t...
Hi @Bertrand BURCKER​ ,This feature is not available yet. If you would like to test it or explore it, then you will need to sign up to get more details once the feature is available. In addition, you could reach out to your account team (in case you...
I found a page that tells the compatibility between Databricks and Hive metastore versions: https://docs.microsoft.com/en-us/azure/databricks/data/metastores/external-hive-metastoreBut the information is only up to Databricks 7.1. Anywhere I can find...
it still should work with 7.x +. version. Are you seeing any issue on your cluster ? @Jeffrey Mak​
What industry security controls and practices do Databricks follow?
Databricks recently also updated the Security Addendum that is incorporated into our typical contracts. It's now written in plain language, and provides detailed content around security controls and practices. https://databricks.com/security-addendum...
Yes I believe that is in pipeline
You could use delta sharing without unity catalog. It is an open protocol for secure real-time exchange of large datasets. There repo at https://github.com/delta-io/delta-sharing also has a reference implementation of the delta sharing server which c...
Yes, you'll have an option to use the existing metastore, an external metastore, and Unity at the same time.
Unity Catalog is planned to support multi-cloud architectures from one catalog instance.
Yes, UC will have a suite of REST APIs allowing external tools to query it.UC Data Dictionary tables will be available via SQL as well. It is possible to query the UC Data Dictionary via JDBC/ODBC connection, but this requires a running endpoint or c...
Yes, this is the vision. The first phase of Unity Catalog is focused on the operational aspects of accessing data i.e. data access controls (Table ACL model, data sharing etc) and auditing. Lineage will be introduced in future phases.
Initially UC will support Delta and Parquet formats only. We plan to support more formats with time
The delta sharing protocol is a mechanism to share live data in your Delta Lake without copying it to another system.Currently it supports delta as the source format. For more details on the advantages of using delta as a table format for data lakes ...
