cancel
Showing results for 
Search instead for 
Did you mean: 
Data Governance
Join discussions on data governance practices, compliance, and security within the Databricks Community. Exchange strategies and insights to ensure data integrity and regulatory compliance.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

MadelynM
by Databricks Employee
  • 1098 Views
  • 0 replies
  • 0 kudos

[Recap] Data + AI Summit 2024 - Data Governance | Navigate the explosion of AI, data and tools

Here's your Data + AI Summit 2024 - Data Governance recap as you navigate the explosion of AI, data and tools in efforts to build a flexible and scalable governance framework that spans your entire data and AI estate. Keynote: Evolving Data Governan...

Screenshot 2024-07-03 at 9.27.29 AM.png
  • 1098 Views
  • 0 replies
  • 0 kudos
Kanib
by New Contributor III
  • 17624 Views
  • 6 replies
  • 4 kudos

No parent external location found for <S3-location> | External Table in UnityCatalog

Hi,We are trying to adopt the Unity Catalog and are testing out creating an External table with some data from the samples catalog that comes along with creating a meta store. The command we are using is:CREATE TABLE IF NOT EXISTS my_catalog.quicksta...

  • 17624 Views
  • 6 replies
  • 4 kudos
Latest Reply
Debayan
Databricks Employee
  • 4 kudos

In Unity catalog you have to pre-define the EXTERNAL LOCATION (and a STORAGE CREDENTIAL) before you use it this way. Please refer below:https://docs.databricks.com/data-governance/unity-catalog/manage-external-locations-and-credentials.html#manage-ex...

  • 4 kudos
5 More Replies
cax
by New Contributor II
  • 1596 Views
  • 3 replies
  • 1 kudos

Where can I find more info about the api of unity catalog?

Where can I find more info about the api of unity catalog?

  • 1596 Views
  • 3 replies
  • 1 kudos
Latest Reply
Vartika
Databricks Employee
  • 1 kudos

Hey there @Carlos Queiroz​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please te...

  • 1 kudos
2 More Replies
RicksDB
by Contributor III
  • 1413 Views
  • 0 replies
  • 2 kudos

Renaming spark files using External Locations

Hello,Before Unity catalog, we were able to rename spark files to a single file (ex: accounts.csv instead of part001) using mounts with pandas or dbutils.fs. How can we achieve the same today without mounts? Is it possible to use external locations w...

  • 1413 Views
  • 0 replies
  • 2 kudos
Erik
by Valued Contributor III
  • 2068 Views
  • 3 replies
  • 2 kudos

Resolved! Will Unity catalog come to Europe?

It is our understanding that enabling unity catalog means that some metadata (for example usernames) will be sent to the USA for processing/storage. This is unfortunately a deal-breaker for us, and we need the data to reside solely in Europe for comp...

  • 2068 Views
  • 3 replies
  • 2 kudos
Latest Reply
youssefmrini
Databricks Employee
  • 2 kudos

Usernames have always been stored centrally in the US. That's similar to most global services for the IAM Layer. The only thing additional that gets stored in the US as part of UC is the metastore names. you can check this link ( https://azure.micros...

  • 2 kudos
2 More Replies
Seth_J
by New Contributor III
  • 7233 Views
  • 8 replies
  • 6 kudos

SQL Endpoint with External Hive Metastore

I am trying to setup a Databricks SQL endpoint to connect to an external hive metastore. I have replicated my metastore spark config in the SQL endpoint, with the addition of adding the below to the configuration:spark.sql.hive.metastore.jars maven I...

  • 7233 Views
  • 8 replies
  • 6 kudos
Latest Reply
Vivian_Wilfred
Databricks Employee
  • 6 kudos

Hi @SETH JENSEN​ , Just checking if you were able to solve this issue? If no, Can you try to modify the connection string and make sslMode=Require and see if it works?

  • 6 kudos
7 More Replies
626371
by New Contributor III
  • 3245 Views
  • 3 replies
  • 3 kudos

Union Data from two separate Databricks Workspaces

Hi Everyone,This may sound bit odd as I haven't used Databricks much.I have a use case where I want to union data from two separate Databricks workspaces (under the same account) into a single SQL view. (Basically 2-2 tables from each workspaces and ...

  • 3245 Views
  • 3 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hey there @Bindi P​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? It would be really helpful for the other members too.Cheers!

  • 3 kudos
2 More Replies
129876
by New Contributor III
  • 2233 Views
  • 2 replies
  • 2 kudos

Unable to run spark sql commands from ipywidget button click event

I'm unable to run any command that queries data from the unity catalog within a function that executes in the event of an ipywidget button click. Code block below. I cannot do queries such as spark.sql(f"SHOW SCHEMAS;") or spark.sql(f"select * from d...

  • 2233 Views
  • 2 replies
  • 2 kudos
Latest Reply
Atanu
Databricks Employee
  • 2 kudos

can you try to println out? val databricksApiTokenKey = CredentialContext.INHERITED_PROPERTY_DATABRICKS_API_TOKEN val databricksApiCredentialOpt = CredentialContext.getCredential(databricksApiTokenKey) val rawUrlProp = spark.sparkContext.get...

  • 2 kudos
1 More Replies
Alexey
by Contributor
  • 1750 Views
  • 0 replies
  • 4 kudos

Table Access Control without High-Concurrency and with R

As we are starting to build our Lakehouse solution on Databricks, we need ACLs to be active. So far I have found two options:via UI or terraform: create a high-concurrency cluster and enable table access control for python and SQL. In terraform this ...

  • 1750 Views
  • 0 replies
  • 4 kudos
aladda
by Databricks Employee
  • 1488 Views
  • 1 replies
  • 2 kudos
  • 1488 Views
  • 1 replies
  • 2 kudos
Latest Reply
zpappa
Databricks Employee
  • 2 kudos

Privileges are additive by design.Let's say I as a user belong to 2 groups, foo1, foo2.The admin has run the following SQL:GRANT SELECT on table bar to `foo1`This means that I'll be able to select from the table bar because there's a grant.Let's say ...

  • 2 kudos
saltuk
by Contributor
  • 7846 Views
  • 7 replies
  • 5 kudos

Resolved! Databricks SQL cannot Communicate With External Hive Metastore which runs on Postgre SQL

I have created a premium cluster on Azure site. There were no problem Data Science and Engineering (DSAE) while i was binding Postgre SQL hive metastore. I have done all settings via Global init Scripts from Admin Console.However when I try to adjust...

  • 7846 Views
  • 7 replies
  • 5 kudos
Latest Reply
saltuk
Contributor
  • 5 kudos

Find soluttion :spark.hadoop.javax.jdo.option.ConnectionURL jdbc:postgresql://#########.postgres.database.azure.com:5432/databricks_metastore?ssl=trueneeds to be spark.hadoop.javax.jdo.option.ConnectionURL jdbc:postgresql://#########.postgres.databas...

  • 5 kudos
6 More Replies
Anonymous
by Not applicable
  • 1483 Views
  • 0 replies
  • 6 kudos

Your guide to data and AI governance sessions at Data + AI Summit 2022 We are so excited to welcome you to Data + AI Summit this year. This year’s hyb...

Your guide to data and AI governance sessions at Data + AI Summit 2022We are so excited to welcome you to Data + AI Summit this year. This year’s hybrid conference is jam packed with exciting product announcements, live keynotes, breakout sessions wi...

  • 1483 Views
  • 0 replies
  • 6 kudos
NOOR_BASHASHAIK
by Contributor
  • 7466 Views
  • 3 replies
  • 1 kudos

Exploiting Hive Metastore of Databricks for lineage

Hi allWanted to check if anyone has made an attempt to exploit the Hive Metastore of Databricks for lineage?For example, I loaded metadata of 2 databricks databases using the Collibra Marketplace provided Databricks driver. Here is the scenario -Data...

  • 7466 Views
  • 3 replies
  • 1 kudos
Latest Reply
NOOR_BASHASHAIK
Contributor
  • 1 kudos

@Chetan Kardekar​ @Kaniz Fatma​ yes, I still need a standard way (through SQL) to access Hive Metastore.

  • 1 kudos
2 More Replies
Soma
by Valued Contributor
  • 2394 Views
  • 0 replies
  • 1 kudos

Shared Hive External Metastore Drop table issue

hi Team ,We are using a shared hive metastore for connecting to dev and prod as we need cross DB access for certain ML use-cases.Currently we are having issues as "DROP TABLE COMMAND" directly drops the table without checking the underlying locationH...

  • 2394 Views
  • 0 replies
  • 1 kudos
chandan_a_v
by Valued Contributor
  • 21745 Views
  • 9 replies
  • 6 kudos

Resolved! GC (Metadata GC Threshold) issue

Hi All,I am facing the GC metadata issue while performing distributed computing on Spark. 2022-01-13T22:02:28.467+0000: [GC (Metadata GC Threshold) [PSYoungGen: 458969K->18934K(594944K)] 458969K->18958K(1954816K), 0.0144028 secs] [Times: user=0.05 sy...

  • 21745 Views
  • 9 replies
  • 6 kudos
Latest Reply
Anonymous
Not applicable
  • 6 kudos

Hi @Chandan Angadi​ Hope you are doing well now.Thanks for getting back to us and sending in your solution. Would you like to mark an answer as best?Thanks!

  • 6 kudos
8 More Replies
Hubert-Dudek
by Esteemed Contributor III
  • 4056 Views
  • 1 replies
  • 24 kudos

Databricks Runtime 10.5 (Beta)  &#xd83d;&#xdc49; Auto Loader: new SQL function CLOUD_FILES_STATE You can use the new CLOUD_FILES_STATE function to query the ...

Databricks Runtime 10.5 (Beta) Auto Loader: new SQL function CLOUD_FILES_STATEYou can use the new CLOUD_FILES_STATE function to query the internal state of an Auto Loader stream. Delta Lake: new maxRecordsPerFile option for maximum records written t...

  • 4056 Views
  • 1 replies
  • 24 kudos
Latest Reply
SørenRavn
New Contributor II
  • 24 kudos

anyone that got CLOUD_FILES_STATE to work on 10.5 Beta/Azure. Syntax:%sqlSELECT * FROM cloud_files_state('abfss://test@test.dfs.core.windows.net/test/checkpoint/'); i get this error: com.databricks.backend.common.rpc.DatabricksExceptions$SQLExecu...

  • 24 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels