cancel
Showing results for 
Search instead for 
Did you mean: 
Data Governance
Join discussions on data governance practices, compliance, and security within the Databricks Community. Exchange strategies and insights to ensure data integrity and regulatory compliance.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Governance

Forum Posts

Chris_Shehu
by Valued Contributor III
  • 8705 Views
  • 5 replies
  • 2 kudos

Resolved! Are their plans to support init scripts within shared compute resources?

Currently we have an Init script that specifies the location of different network resources by updating the hosts file. When we moved to the Unity catalog all of our clusters had to be upgraded to support it. Now our scripts have stopped working due ...

  • 8705 Views
  • 5 replies
  • 2 kudos
Latest Reply
youssefmrini
Honored Contributor III
  • 2 kudos

We are bringing soon init script and Scala to shared Cluster

  • 2 kudos
4 More Replies
Avvar2022
by Contributor
  • 8415 Views
  • 4 replies
  • 1 kudos

Resolved! I am new to Data bricks. Setting up Data bricks Unity Catalog, in terms of best practice i have few questions.

Is it best practice to separate unity catalog meta store ADLS Gen2 separate from ADLS Gen 2 to store data ?Since per region only one meta store can be created, will there be a separate meta store for PROD, and NON-PROD(QA and DEV)? If yes they need t...

  • 8415 Views
  • 4 replies
  • 1 kudos
Latest Reply
karthik_p
Esteemed Contributor
  • 1 kudos

@Ashok Zubrewar​  coming to your 3 rd question, if you are using any external tables then non uc ADLS GEN 2 is mandatory, you can not use UC ADLS GEN2. as it hosts metadata and managed table data. there is no restriction in terms of your external buc...

  • 1 kudos
3 More Replies
ossinova
by Contributor II
  • 12318 Views
  • 4 replies
  • 3 kudos

Resolved! Best practice for organizing data objects using Unity Catalog

I was wondering if there is a best practice or recommended way to organize data objects (tables) in Unity Catalog.Following the medallion architecture, how should one organize and name these three namespaces:Catalog(s)Schema(s)Table(s)Example:Catalog...

  • 12318 Views
  • 4 replies
  • 3 kudos
Latest Reply
Avvar2022
Contributor
  • 3 kudos

In Real time do we have 1 meta store for PROD and NON-PROD ? if Yes in the backend there would be a one storage account (ADLS Gen2) > Container for all Environments.

  • 3 kudos
3 More Replies
User16752244127
by Contributor
  • 8833 Views
  • 3 replies
  • 7 kudos

Free e-book: Rise of the Data Lakehouse Exciting news! You can now get a free copy of Bill Inmon's new book, Rise of the Data Lakehouse. The book ...

Free e-book: Rise of the Data LakehouseExciting news! You can now get a free copy of Bill Inmon's new book, Rise of the Data Lakehouse. The book is a must-read for anyone looking to implement the data lakehouse approach in their organization. The dat...

bibook
  • 8833 Views
  • 3 replies
  • 7 kudos
Latest Reply
Anonymous
Not applicable
  • 7 kudos

Thanks for sharing the valuable information!

  • 7 kudos
2 More Replies
sqlshep
by New Contributor III
  • 6613 Views
  • 12 replies
  • 2 kudos

Create Schema with Unity enabled

i have unit setup, and am trying to create a delta schema and table, i am getting an error on the schema creation, i am able to list the files and folders in the ADLS Gen2 storage account, i am able to write a parquet file to ADLS, but i cannot crea...

Screenshot 2023-03-11 at 12.11.33 PM
  • 6613 Views
  • 12 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Shep Sheppard​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...

  • 2 kudos
11 More Replies
laksh
by New Contributor II
  • 1236 Views
  • 3 replies
  • 0 kudos

Real time data quality validation (Streaming data ingestion)

I was wondering how the Unity Catalog would help in data quality validations for real time (streaming data) data ingestion. 

  • 1236 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @arun laksh​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 0 kudos
2 More Replies
Hubert-Dudek
by Esteemed Contributor III
  • 2351 Views
  • 4 replies
  • 5 kudos

SET MASK and SET ROW FILTER are coming to the databricks unity catalog  Inside, you use is_account_group_member() to check for account-level groups or...

SET MASK and SET ROW FILTER are coming to the databricks unity catalog Inside, you use is_account_group_member() to check for account-level groups or is_member() for workplace-level groups. is_account_group_member() is recommended with Unity Catalog.

Screenshot 2023-01-30 123545
  • 2351 Views
  • 4 replies
  • 5 kudos
Latest Reply
Jerry01
New Contributor III
  • 5 kudos

Do we have this feature SET MASK and ROW FILTER enabled now?

  • 5 kudos
3 More Replies
Mado
by Valued Contributor II
  • 8744 Views
  • 1 replies
  • 1 kudos

Resolved! Databricks Cluster Logs, Where can I find table usage information or queries?

Hi,I am trying to find queries I run in a notebook (running on a cluster) in Cluster Logs.I set the cluster to deliver logs to a folder on DBFS and I can read log files from there. I created a Databricks workspace on the premium pricing tier and it i...

d580c18e-59f2-4291-8b51-9a4c8cb0ea0a Screenshot
  • 8744 Views
  • 1 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

@Mohammad Saber​ :If you are not seeing the expected logs in the log files, it's possible that either logging was not properly configured or the logs have been rotated out of the active log file and into an archive file. Here are some suggestions you...

  • 1 kudos
Elon
by New Contributor III
  • 7562 Views
  • 1 replies
  • 5 kudos

Create Metastore. Missing permissions: The associated credential does not grant permission to perform all necessary operations.

Cloud: AWSRegion: eu-west-1S3 location: s3://databricks-dev-bucketIAM role ARN: arn:aws:iam::18XXXXXXXX29:role/databricks-s3-metastoreGuide followed: ref: https://docs.databricks.com/data-governance/unity-catalog/get-started.html#cloud-tenant-setup-a...

image.png Polic simulator
  • 7562 Views
  • 1 replies
  • 5 kudos
Latest Reply
Elon
New Contributor III
  • 5 kudos

Bump. @Yeshaswini P V​ @Gokul Kumar P​

  • 5 kudos
Christine
by Contributor II
  • 6432 Views
  • 4 replies
  • 0 kudos

"spark.databricks.pyspark.enablePy4JSecurity is not allowed when choosing an access mode"

Due to some problems, I need to set Py4JSecurity to false, however I get the following message: "spark.databricks.pyspark.enablePy4JSecurity is not allowed when choosing an access mode"My cluster is in Shared access mode and a runtime version 12.1 (i...

  • 6432 Views
  • 4 replies
  • 0 kudos
Latest Reply
Debayan
Esteemed Contributor III
  • 0 kudos

Hi, Could you please test it on a standard standalone cluster/single user cluster and confirm? The error is occurring because enabling the security may break user isolation. Please let us know if this helps. Also please tag @Debayan​ with your next r...

  • 0 kudos
3 More Replies
Lizzz
by New Contributor II
  • 1628 Views
  • 2 replies
  • 2 kudos

Resolved! Sync delta tables across workspaces

What is the most cost-effective way to sync delta tables across workspaces? I'd like to sync a production table from the production workspace to a staging workspace for testing. It doesn't have to be in real-time, on a daily basis is acceptable. We a...

  • 1628 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Liz Zhang​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we ...

  • 2 kudos
1 More Replies
lizou
by Contributor II
  • 3112 Views
  • 3 replies
  • 1 kudos

unity catalog Request failed with status code 404

maybe this is a bug in the unity catalog as the error happens multiple times when a table becomes inaccessibleerror: Request failed with status code 404try to drop the table error: NoSuchTableException: [TABLE_OR_VIEW_NOT_FOUND] try to replace the ta...

image
  • 3112 Views
  • 3 replies
  • 1 kudos
Latest Reply
lizou
Contributor II
  • 1 kudos

the error disappeared now after a few months.It could behappens when there is a lot of schema change on db during the initial project stage.there may be some temp security issuesmay be databricks fixed in the back end appears less issue when schema r...

  • 1 kudos
2 More Replies
Mado
by Valued Contributor II
  • 3887 Views
  • 4 replies
  • 1 kudos

Resolved! How to create a new metastore?

Hi,I want to configure Unity Catalog and one step is creating a metastore in the region where I create databricks workspace (I am on Azure). I created a workspace with a premium pricing tier and I am the admin. Following the documentation, I should g...

image.png image
  • 3887 Views
  • 4 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Mohammad Saber​ Glad to hear that!Please mark the answer as best. It will be highly grateful.Thanks

  • 1 kudos
3 More Replies
Vivek_Singh
by New Contributor III
  • 2306 Views
  • 6 replies
  • 6 kudos

Resolved! enabled the unity catalog having Premium subscription

Hello All, I am using Premium subscription and have admin access in workspace, however I am not able to get the option to create metastore for unity catalog

  • 2306 Views
  • 6 replies
  • 6 kudos
Latest Reply
Anonymous
Not applicable
  • 6 kudos

Hi @Vivek Singh Banafar​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your f...

  • 6 kudos
5 More Replies
knawara
by Contributor
  • 3704 Views
  • 7 replies
  • 5 kudos

READ FILES and WRITE FILES when using Hive Metastore

Hello,I'm confused about documentation on privilege types when using HMS.The following page is supposed to talk about HMShttps://docs.databricks.com/sql/language-manual/sql-ref-privileges-hms.htmlbut it also mentions READ FILESQuery files directly us...

  • 3704 Views
  • 7 replies
  • 5 kudos
Latest Reply
Anonymous
Not applicable
  • 5 kudos

Hi @Chris Nawara​  I'm sorry you could not find a solution to your problem in the answers provided.Our community strives to provide helpful and accurate information, but sometimes an immediate solution may only be available for some issues.I suggest ...

  • 5 kudos
6 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels