cancel
Showing results for 
Search instead for 
Did you mean: 
Data Governance
Join discussions on data governance practices, compliance, and security within the Databricks Community. Exchange strategies and insights to ensure data integrity and regulatory compliance.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Governance

Forum Posts

MadelynM
by Databricks Employee
  • 652 Views
  • 0 replies
  • 0 kudos

[Recap] Data + AI Summit 2024 - Data Governance | Navigate the explosion of AI, data and tools

Here's your Data + AI Summit 2024 - Data Governance recap as you navigate the explosion of AI, data and tools in efforts to build a flexible and scalable governance framework that spans your entire data and AI estate. Keynote: Evolving Data Governan...

Screenshot 2024-07-03 at 9.27.29 AM.png
  • 652 Views
  • 0 replies
  • 0 kudos
amitca71
by Contributor II
  • 3424 Views
  • 2 replies
  • 4 kudos

Resolved! unity catalog databricks_metastore terraform - not authorized

i created unity catalog using terraform on AWS, and able to create manualy a meta store with same user.when trying to create metsastore from terraform, i get unauthorized error:Error: cannot create metastore: Unauthorized│ â”‚  with module.unity_catalo...

  • 3424 Views
  • 2 replies
  • 4 kudos
Latest Reply
amitca71
Contributor II
  • 4 kudos

thanks

  • 4 kudos
1 More Replies
yashpalshah
by New Contributor
  • 964 Views
  • 1 replies
  • 0 kudos

Is Unity Catalog Enterprise ready?

I am new to Databricks and currently trying to setup Unity Catalog on AWS at Enterprise level. I would like to know if Unity Catalog is enterprise ready and if there are any pointers that could help me set is up on AWS at enterprise level.Appreciate ...

  • 964 Views
  • 1 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Yashpal Shah​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...

  • 0 kudos
ossinova
by Contributor II
  • 4596 Views
  • 8 replies
  • 2 kudos

UC Enabled cluster for ADF ingestion

I am migrating my Data Lake to use Unity Catalog. However, this comes with changes to the clusters. I have tried a few options, but it seems rather complex than it should be. I need to create a cluster used by ADF that is Unity Enabled that can insta...

  • 4596 Views
  • 8 replies
  • 2 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 2 kudos

I exclusively use job clusters. They are cheaper.Especially when you create a pool with spot instances. I'd go for that because that is what it's made for: batch jobs/

  • 2 kudos
7 More Replies
amitca71
by Contributor II
  • 25553 Views
  • 4 replies
  • 2 kudos

DeltaTable.forPath(spark, path) doesnt recognize table

Hi,I'm working with unity catalog for the last week. I'm refering to delta table by path, as follwing: path='s3://<my_bucket_name>/silver/data/<table_name>DeltaTable.forPath(spark, path)I get an exception that "is not a Delta table"using the table na...

image
  • 25553 Views
  • 4 replies
  • 2 kudos
Latest Reply
amitca71
Contributor II
  • 2 kudos

its even more weired. on one next cells it doesnt... see below older version even by name doesnt work 

  • 2 kudos
3 More Replies
data_boy_2022
by New Contributor III
  • 3188 Views
  • 2 replies
  • 3 kudos

Resolved! Unity Catalog + Streaming error: method public X is not whitelisted on class DataStreamReader

I want to import data using the autoloader from a S3 bucket into a table which is managed inside a Unity Catalog.Right now, I run the code on an interactive cluster inside a notebook. In the future the code should run in a job cluster. The error I ge...

Screenshot 2022-09-09 at 6.15.00 PM
  • 3188 Views
  • 2 replies
  • 3 kudos
Latest Reply
data_boy_2022
New Contributor III
  • 3 kudos

Works with single user mode. Thank you!

  • 3 kudos
1 More Replies
parthsalvi
by Contributor
  • 5963 Views
  • 4 replies
  • 3 kudos

Resolved! Python Code not working in DBR 10.4 LTS - Shared Mode

Hello,We're trying to use DBR 10.4 LTS in Shared Mode. But our python code is not working in that cluster.We're running into following messageYour administrator has only allowed sql commands on this cluster. This execution contained at least one disa...

image
  • 5963 Views
  • 4 replies
  • 3 kudos
Latest Reply
Cedric
Databricks Employee
  • 3 kudos

Hi Parthsalvi,Could you please try using 11.1+ and see if that works?

  • 3 kudos
3 More Replies
data_boy_2022
by New Contributor III
  • 5889 Views
  • 2 replies
  • 3 kudos

Are Python UDFs supported in Unity Catalog?

I am trying to execute a UDF written in Python and receive the following error message:Error message: [UC_COMMAND_NOT_SUPPORTED] UDF/UDAF functions are not supported in Unity Catalog.;The limitation is not mentioned in the docs( https://docs.databric...

  • 5889 Views
  • 2 replies
  • 3 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 3 kudos

Indeed it is not mentioned in the limitations.This blog mentions python UDFs in UC, but in databricks SQL, not the classic DE workspace.So it seems that it is not (yet) supported, as the docs only mention SQL functions.

  • 3 kudos
1 More Replies
User16826994223
by Honored Contributor III
  • 4055 Views
  • 4 replies
  • 1 kudos

Where the data is stored in processing

Do I have to transfer my data into Databricks' AWS account?,Candatabricks access my data and see all my compnay information

  • 4055 Views
  • 4 replies
  • 1 kudos
Latest Reply
PriyaAnanthram
Contributor III
  • 1 kudos

No all the data is stored within your own AWS.Some data to enhance your databricks experience maybe stored in DBFS

  • 1 kudos
3 More Replies
zbir84
by New Contributor II
  • 569 Views
  • 0 replies
  • 1 kudos

Can't read multiline json from a unity catalog S3 external location

Hi,​Is there a know limitation in Unity Catalog workspaces ​with reading multiline json files from an S3 external location using pyspark? The file works perfectly fine if it's on dbfs, and reading json files in general works from that s3 location if ...

  • 569 Views
  • 0 replies
  • 1 kudos
Kanib
by New Contributor III
  • 15490 Views
  • 6 replies
  • 4 kudos

No parent external location found for <S3-location> | External Table in UnityCatalog

Hi,We are trying to adopt the Unity Catalog and are testing out creating an External table with some data from the samples catalog that comes along with creating a meta store. The command we are using is:CREATE TABLE IF NOT EXISTS my_catalog.quicksta...

  • 15490 Views
  • 6 replies
  • 4 kudos
Latest Reply
Debayan
Databricks Employee
  • 4 kudos

In Unity catalog you have to pre-define the EXTERNAL LOCATION (and a STORAGE CREDENTIAL) before you use it this way. Please refer below:https://docs.databricks.com/data-governance/unity-catalog/manage-external-locations-and-credentials.html#manage-ex...

  • 4 kudos
5 More Replies
cax
by New Contributor II
  • 1357 Views
  • 3 replies
  • 1 kudos

Where can I find more info about the api of unity catalog?

Where can I find more info about the api of unity catalog?

  • 1357 Views
  • 3 replies
  • 1 kudos
Latest Reply
Vartika
Databricks Employee
  • 1 kudos

Hey there @Carlos Queiroz​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please te...

  • 1 kudos
2 More Replies
RicksDB
by Contributor II
  • 1263 Views
  • 0 replies
  • 2 kudos

Renaming spark files using External Locations

Hello,Before Unity catalog, we were able to rename spark files to a single file (ex: accounts.csv instead of part001) using mounts with pandas or dbutils.fs. How can we achieve the same today without mounts? Is it possible to use external locations w...

  • 1263 Views
  • 0 replies
  • 2 kudos
Erik
by Valued Contributor II
  • 1808 Views
  • 3 replies
  • 2 kudos

Resolved! Will Unity catalog come to Europe?

It is our understanding that enabling unity catalog means that some metadata (for example usernames) will be sent to the USA for processing/storage. This is unfortunately a deal-breaker for us, and we need the data to reside solely in Europe for comp...

  • 1808 Views
  • 3 replies
  • 2 kudos
Latest Reply
youssefmrini
Databricks Employee
  • 2 kudos

Usernames have always been stored centrally in the US. That's similar to most global services for the IAM Layer. The only thing additional that gets stored in the US as part of UC is the metastore names. you can check this link ( https://azure.micros...

  • 2 kudos
2 More Replies
Seth_J
by New Contributor III
  • 6469 Views
  • 8 replies
  • 6 kudos

SQL Endpoint with External Hive Metastore

I am trying to setup a Databricks SQL endpoint to connect to an external hive metastore. I have replicated my metastore spark config in the SQL endpoint, with the addition of adding the below to the configuration:spark.sql.hive.metastore.jars maven I...

  • 6469 Views
  • 8 replies
  • 6 kudos
Latest Reply
Vivian_Wilfred
Databricks Employee
  • 6 kudos

Hi @SETH JENSEN​ , Just checking if you were able to solve this issue? If no, Can you try to modify the connection string and make sslMode=Require and see if it works?

  • 6 kudos
7 More Replies
626371
by New Contributor III
  • 2897 Views
  • 3 replies
  • 3 kudos

Union Data from two separate Databricks Workspaces

Hi Everyone,This may sound bit odd as I haven't used Databricks much.I have a use case where I want to union data from two separate Databricks workspaces (under the same account) into a single SQL view. (Basically 2-2 tables from each workspaces and ...

  • 2897 Views
  • 3 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hey there @Bindi P​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? It would be really helpful for the other members too.Cheers!

  • 3 kudos
2 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels