cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Governance
Join discussions on data governance practices, compliance, and security within the Databricks Community. Exchange strategies and insights to ensure data integrity and regulatory compliance.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
NOOR_BASHASHAIK
by • Contributor
  • 4852 Views
  • 6 replies
  • 4 kudos

Resolved! Exploiting Hive Metastore of Databricks for lineage

Hi allWanted to check if anyone has made an attempt to exploit the Hive Metastore of Databricks for lineage?For example, I loaded metadata of 2 databricks databases using the Collibra Marketplace provided Databricks driver. Here is the scenario -Data...

Data Governance
Reply
  • 4852 Views
  • 6 replies
  • 4 kudos
View Replies
Latest Reply
NOOR_BASHASHAIK
Contributor
  • 4 kudos

@Chetan Kardekar​ @Kaniz Fatma​ yes, I still need a standard way (through SQL) to access Hive Metastore.

  • 4 kudos
5 More Replies
Soma
by • Valued Contributor
  • 1495 Views
  • 0 replies
  • 1 kudos

Shared Hive External Metastore Drop table issue

hi Team ,We are using a shared hive metastore for connecting to dev and prod as we need cross DB access for certain ML use-cases.Currently we are having issues as "DROP TABLE COMMAND" directly drops the table without checking the underlying locationH...

Data Governance
Reply
  • 1495 Views
  • 0 replies
  • 1 kudos
chandan_a_v
by • Valued Contributor
  • 15123 Views
  • 10 replies
  • 6 kudos

Resolved! GC (Metadata GC Threshold) issue

Hi All,I am facing the GC metadata issue while performing distributed computing on Spark. 2022-01-13T22:02:28.467+0000: [GC (Metadata GC Threshold) [PSYoungGen: 458969K->18934K(594944K)] 458969K->18958K(1954816K), 0.0144028 secs] [Times: user=0.05 sy...

Data Governance
Reply
  • 15123 Views
  • 10 replies
  • 6 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 6 kudos

Hi @Chandan Angadi​ Hope you are doing well now.Thanks for getting back to us and sending in your solution. Would you like to mark an answer as best?Thanks!

  • 6 kudos
9 More Replies
Hubert-Dudek
by • Esteemed Contributor III
  • 3350 Views
  • 2 replies
  • 24 kudos

Databricks Runtime 10.5 (Beta)  �� Auto Loader: new SQL function CLOUD_FILES_STATE You can use the new CLOUD_FILES_STATE function to query the ...

Databricks Runtime 10.5 (Beta) Auto Loader: new SQL function CLOUD_FILES_STATEYou can use the new CLOUD_FILES_STATE function to query the internal state of an Auto Loader stream. Delta Lake: new maxRecordsPerFile option for maximum records written t...

Data Governance
Reply
  • 3350 Views
  • 2 replies
  • 24 kudos
View Replies
Latest Reply
SørenRavn
New Contributor II
  • 24 kudos

anyone that got CLOUD_FILES_STATE to work on 10.5 Beta/Azure. Syntax:%sqlSELECT * FROM cloud_files_state('abfss://test@test.dfs.core.windows.net/test/checkpoint/'); i get this error: com.databricks.backend.common.rpc.DatabricksExceptions$SQLExecu...

  • 24 kudos
1 More Replies
Mr__E
by • Contributor II
  • 4144 Views
  • 3 replies
  • 1 kudos

Resolved! Table ACLs, secrets, and compute clusters

I am adding a credential to access MongoDB through the Python MongoDB Spark connector. It is stored as a secret and the `users` group has `MANAGE` ACL privileges. I turned on table ACLs and created a `High Concurrency` compute cluster. I then run a s...

Data Governance
Reply
  • 4144 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Atanu
Esteemed Contributor
  • 1 kudos

@Erik Louie​  looks the issue is resolved. Please let me know if you need further help on this. Thanks!

  • 1 kudos
2 More Replies
idmserialkey
by • New Contributor
  • 20720 Views
  • 0 replies
  • 0 kudos

assadpc.com

IDM Serial Key with Internet Download Manager (IDM) is a tool to increase download speeds, resume, and schedule downloads. https://assadpc.com/idm-serial-number/

Data Governance
Reply
  • 20720 Views
  • 0 replies
  • 0 kudos
RantoB
by • Valued Contributor
  • 12291 Views
  • 8 replies
  • 6 kudos

Resolved! Enable Unity Catalog and pre-requisite for Delta Sharing

Hi, I would like to use Delta Sharing but I have not understood how to configure the Delta Sharing Server. Is that fully managed by Databricks ? If so I tried to share a table :%sql create share first_shareand I got the following error message with t...

Data Governance
Reply
  • 12291 Views
  • 8 replies
  • 6 kudos
View Replies
Latest Reply
jose_gonzalez
Moderator
  • 6 kudos

Hi @Bertrand BURCKER​ ,This feature is not available yet. If you would like to test it or explore it, then you will need to sign up to get more details once the feature is available. In addition, you could reach out to your account team (in case you...

  • 6 kudos
7 More Replies
jeffreym9
by • New Contributor III
  • 5275 Views
  • 3 replies
  • 4 kudos

Resolved! Compatibility of Hive metastore schema version and Databricks 9.1 LTS

I found a page that tells the compatibility between Databricks and Hive metastore versions: https://docs.microsoft.com/en-us/azure/databricks/data/metastores/external-hive-metastoreBut the information is only up to Databricks 7.1. Anywhere I can find...

Data Governance
Reply
  • 5275 Views
  • 3 replies
  • 4 kudos
View Replies
Latest Reply
Atanu
Esteemed Contributor
  • 4 kudos

it still should work with 7.x +. version. Are you seeing any issue on your cluster ? @Jeffrey Mak​ 

  • 4 kudos
2 More Replies
prakash_raj
by • New Contributor II
  • 4840 Views
  • 2 replies
  • 3 kudos

Resolved! How to Create Cluster that should have access to AWS Glue Catalog in different account

How to Create Cluster that should have access to AWS Glue Catalog in different account

Data Governance
Reply
  • 4840 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Prabakar
Esteemed Contributor III
  • 3 kudos

Hi @Prakash Rajendran​ , If the Glue Data Catalog is in a different AWS account from where Databricks is deployed, a cross-account access policy must allow access to the catalog from the AWS account where Databricks is deployed.Please refer to the be...

  • 3 kudos
1 More Replies
User16826994223
by • Honored Contributor III
  • 3640 Views
  • 2 replies
  • 1 kudos

What industry security controls and practices do Databricks follow?

What industry security controls and practices do Databricks follow?

Data Governance
Reply
  • 3640 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
davidveuve
New Contributor II
  • 1 kudos

Databricks recently also updated the Security Addendum that is incorporated into our typical contracts. It's now written in plain language, and provides detailed content around security controls and practices. https://databricks.com/security-addendum...

  • 1 kudos
1 More Replies
Anonymous
by • Not applicable
  • 4730 Views
  • 2 replies
  • 0 kudos

Can I use Delta Sharing without Unity Catalog?

Data Governance
Reply
  • 4730 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
sajith_appukutt
Honored Contributor II
  • 0 kudos

You could use delta sharing without unity catalog. It is an open protocol for secure real-time exchange of large datasets. There repo at https://github.com/delta-io/delta-sharing also has a reference implementation of the delta sharing server which c...

  • 0 kudos
1 More Replies
Anonymous
by • Not applicable
  • 3255 Views
  • 1 replies
  • 1 kudos

Can other systems/compute engines connect to the Unity Catalog?

Data Governance
Reply
  • 3255 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Yes, UC will have a suite of REST APIs allowing external tools to query it.UC Data Dictionary tables will be available via SQL as well. It is possible to query the UC Data Dictionary via JDBC/ODBC connection, but this requires a running endpoint or c...

  • 1 kudos
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels