cancel
Showing results for 
Search instead for 
Did you mean: 
Data Governance
Join discussions on data governance practices, compliance, and security within the Databricks Community. Exchange strategies and insights to ensure data integrity and regulatory compliance.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Governance

Forum Posts

Erik
by Valued Contributor II
  • 1508 Views
  • 3 replies
  • 2 kudos

Resolved! Will Unity catalog come to Europe?

It is our understanding that enabling unity catalog means that some metadata (for example usernames) will be sent to the USA for processing/storage. This is unfortunately a deal-breaker for us, and we need the data to reside solely in Europe for comp...

  • 1508 Views
  • 3 replies
  • 2 kudos
Latest Reply
youssefmrini
Honored Contributor III
  • 2 kudos

Usernames have always been stored centrally in the US. That's similar to most global services for the IAM Layer. The only thing additional that gets stored in the US as part of UC is the metastore names. you can check this link ( https://azure.micros...

  • 2 kudos
2 More Replies
Seth_J
by New Contributor III
  • 5117 Views
  • 9 replies
  • 6 kudos

SQL Endpoint with External Hive Metastore

I am trying to setup a Databricks SQL endpoint to connect to an external hive metastore. I have replicated my metastore spark config in the SQL endpoint, with the addition of adding the below to the configuration:spark.sql.hive.metastore.jars maven I...

  • 5117 Views
  • 9 replies
  • 6 kudos
Latest Reply
Vivian_Wilfred
Honored Contributor
  • 6 kudos

Hi @SETH JENSEN​ , Just checking if you were able to solve this issue? If no, Can you try to modify the connection string and make sslMode=Require and see if it works?

  • 6 kudos
8 More Replies
626371
by New Contributor III
  • 2043 Views
  • 4 replies
  • 3 kudos

Union Data from two separate Databricks Workspaces

Hi Everyone,This may sound bit odd as I haven't used Databricks much.I have a use case where I want to union data from two separate Databricks workspaces (under the same account) into a single SQL view. (Basically 2-2 tables from each workspaces and ...

  • 2043 Views
  • 4 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hey there @Bindi P​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? It would be really helpful for the other members too.Cheers!

  • 3 kudos
3 More Replies
129876
by New Contributor III
  • 1724 Views
  • 2 replies
  • 2 kudos

Unable to run spark sql commands from ipywidget button click event

I'm unable to run any command that queries data from the unity catalog within a function that executes in the event of an ipywidget button click. Code block below. I cannot do queries such as spark.sql(f"SHOW SCHEMAS;") or spark.sql(f"select * from d...

  • 1724 Views
  • 2 replies
  • 2 kudos
Latest Reply
Atanu
Esteemed Contributor
  • 2 kudos

can you try to println out? val databricksApiTokenKey = CredentialContext.INHERITED_PROPERTY_DATABRICKS_API_TOKEN val databricksApiCredentialOpt = CredentialContext.getCredential(databricksApiTokenKey) val rawUrlProp = spark.sparkContext.get...

  • 2 kudos
1 More Replies
Alexey
by Contributor
  • 1199 Views
  • 0 replies
  • 4 kudos

Table Access Control without High-Concurrency and with R

As we are starting to build our Lakehouse solution on Databricks, we need ACLs to be active. So far I have found two options:via UI or terraform: create a high-concurrency cluster and enable table access control for python and SQL. In terraform this ...

  • 1199 Views
  • 0 replies
  • 4 kudos
aladda
by Honored Contributor II
  • 1047 Views
  • 1 replies
  • 2 kudos
  • 1047 Views
  • 1 replies
  • 2 kudos
Latest Reply
zpappa
New Contributor III
  • 2 kudos

Privileges are additive by design.Let's say I as a user belong to 2 groups, foo1, foo2.The admin has run the following SQL:GRANT SELECT on table bar to `foo1`This means that I'll be able to select from the table bar because there's a grant.Let's say ...

  • 2 kudos
saltuk
by Contributor
  • 6048 Views
  • 7 replies
  • 5 kudos

Resolved! Databricks SQL cannot Communicate With External Hive Metastore which runs on Postgre SQL

I have created a premium cluster on Azure site. There were no problem Data Science and Engineering (DSAE) while i was binding Postgre SQL hive metastore. I have done all settings via Global init Scripts from Admin Console.However when I try to adjust...

  • 6048 Views
  • 7 replies
  • 5 kudos
Latest Reply
saltuk
Contributor
  • 5 kudos

Find soluttion :spark.hadoop.javax.jdo.option.ConnectionURL jdbc:postgresql://#########.postgres.database.azure.com:5432/databricks_metastore?ssl=trueneeds to be spark.hadoop.javax.jdo.option.ConnectionURL jdbc:postgresql://#########.postgres.databas...

  • 5 kudos
6 More Replies
Anonymous
by Not applicable
  • 864 Views
  • 0 replies
  • 6 kudos

Your guide to data and AI governance sessions at Data + AI Summit 2022 We are so excited to welcome you to Data + AI Summit this year. This year’s hyb...

Your guide to data and AI governance sessions at Data + AI Summit 2022We are so excited to welcome you to Data + AI Summit this year. This year’s hybrid conference is jam packed with exciting product announcements, live keynotes, breakout sessions wi...

  • 864 Views
  • 0 replies
  • 6 kudos
NOOR_BASHASHAIK
by Contributor
  • 5390 Views
  • 6 replies
  • 4 kudos

Resolved! Exploiting Hive Metastore of Databricks for lineage

Hi allWanted to check if anyone has made an attempt to exploit the Hive Metastore of Databricks for lineage?For example, I loaded metadata of 2 databricks databases using the Collibra Marketplace provided Databricks driver. Here is the scenario -Data...

  • 5390 Views
  • 6 replies
  • 4 kudos
Latest Reply
NOOR_BASHASHAIK
Contributor
  • 4 kudos

@Chetan Kardekar​ @Kaniz Fatma​ yes, I still need a standard way (through SQL) to access Hive Metastore.

  • 4 kudos
5 More Replies
Soma
by Valued Contributor
  • 1534 Views
  • 0 replies
  • 1 kudos

Shared Hive External Metastore Drop table issue

hi Team ,We are using a shared hive metastore for connecting to dev and prod as we need cross DB access for certain ML use-cases.Currently we are having issues as "DROP TABLE COMMAND" directly drops the table without checking the underlying locationH...

  • 1534 Views
  • 0 replies
  • 1 kudos
chandan_a_v
by Valued Contributor
  • 15487 Views
  • 10 replies
  • 6 kudos

Resolved! GC (Metadata GC Threshold) issue

Hi All,I am facing the GC metadata issue while performing distributed computing on Spark. 2022-01-13T22:02:28.467+0000: [GC (Metadata GC Threshold) [PSYoungGen: 458969K->18934K(594944K)] 458969K->18958K(1954816K), 0.0144028 secs] [Times: user=0.05 sy...

  • 15487 Views
  • 10 replies
  • 6 kudos
Latest Reply
Anonymous
Not applicable
  • 6 kudos

Hi @Chandan Angadi​ Hope you are doing well now.Thanks for getting back to us and sending in your solution. Would you like to mark an answer as best?Thanks!

  • 6 kudos
9 More Replies
Hubert-Dudek
by Esteemed Contributor III
  • 3419 Views
  • 2 replies
  • 24 kudos

Databricks Runtime 10.5 (Beta)  �� Auto Loader: new SQL function CLOUD_FILES_STATE You can use the new CLOUD_FILES_STATE function to query the ...

Databricks Runtime 10.5 (Beta) Auto Loader: new SQL function CLOUD_FILES_STATEYou can use the new CLOUD_FILES_STATE function to query the internal state of an Auto Loader stream. Delta Lake: new maxRecordsPerFile option for maximum records written t...

  • 3419 Views
  • 2 replies
  • 24 kudos
Latest Reply
SørenRavn
New Contributor II
  • 24 kudos

anyone that got CLOUD_FILES_STATE to work on 10.5 Beta/Azure. Syntax:%sqlSELECT * FROM cloud_files_state('abfss://test@test.dfs.core.windows.net/test/checkpoint/'); i get this error: com.databricks.backend.common.rpc.DatabricksExceptions$SQLExecu...

  • 24 kudos
1 More Replies
Mr__E
by Contributor II
  • 4242 Views
  • 3 replies
  • 1 kudos

Resolved! Table ACLs, secrets, and compute clusters

I am adding a credential to access MongoDB through the Python MongoDB Spark connector. It is stored as a secret and the `users` group has `MANAGE` ACL privileges. I turned on table ACLs and created a `High Concurrency` compute cluster. I then run a s...

  • 4242 Views
  • 3 replies
  • 1 kudos
Latest Reply
Atanu
Esteemed Contributor
  • 1 kudos

@Erik Louie​  looks the issue is resolved. Please let me know if you need further help on this. Thanks!

  • 1 kudos
2 More Replies
idmserialkey
by New Contributor
  • 20934 Views
  • 0 replies
  • 0 kudos

assadpc.com

IDM Serial Key with Internet Download Manager (IDM) is a tool to increase download speeds, resume, and schedule downloads. https://assadpc.com/idm-serial-number/

  • 20934 Views
  • 0 replies
  • 0 kudos
RantoB
by Valued Contributor
  • 12705 Views
  • 8 replies
  • 6 kudos

Resolved! Enable Unity Catalog and pre-requisite for Delta Sharing

Hi, I would like to use Delta Sharing but I have not understood how to configure the Delta Sharing Server. Is that fully managed by Databricks ? If so I tried to share a table :%sql create share first_shareand I got the following error message with t...

  • 12705 Views
  • 8 replies
  • 6 kudos
Latest Reply
jose_gonzalez
Moderator
  • 6 kudos

Hi @Bertrand BURCKER​ ,This feature is not available yet. If you would like to test it or explore it, then you will need to sign up to get more details once the feature is available. In addition, you could reach out to your account team (in case you...

  • 6 kudos
7 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels