Here's your Data + AI Summit 2024 - Data Governance recap as you navigate the explosion of AI, data and tools in efforts to build a flexible and scalable governance framework that spans your entire data and AI estate. Keynote: Evolving Data Governan...
After implementing Unity catalog, assuming a user has access to more than one DB. Can that user access all the DB in a single workspace? Can we restrict any specific DB in a workspace?
Hi @Prashant Lalaiya​,Hope you are well. Just wanted to see if you were able to find an answer to your question and would you like to mark an answer as best? It would be really helpful for the other members too.Cheers!
Hi all,I see a number of metastore down events in databricks workspace: do any one know what's the reason why this happens?Thanks,Barney
Hi @Ron Barneveld​ Could you kindly help with more details? Is this is a production metastore that you are mentioning or any metastore created by you for test environment. Also where it is hosted?
Hi,dbutils.fs.ls() is a most useful utility to list the files in remote cloud storage. I cannot use this function on top of the location managed by the Unity catalog.I have a managed external location created on `'abfss://xyz@abcd.dfs.core.windows.ne...
Hi @Venkadeshwaran K​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answ...
I have databricks premium and I want to enable unity catalog on it, I am not able to do so. I followed the following steps which are attached in the attachments. I am also attaching the screenshot of my cluster configuration. When I run this command:...
Hi @Prerna Munjal​ Help us build a vibrant and resourceful community by recognizing and highlighting insightful contributions. Mark the best answers and show your appreciation!Regards
Currently we have an Init script that specifies the location of different network resources by updating the hosts file. When we moved to the Unity catalog all of our clusters had to be upgraded to support it. Now our scripts have stopped working due ...
Is it best practice to separate unity catalog meta store ADLS Gen2 separate from ADLS Gen 2 to store data ?Since per region only one meta store can be created, will there be a separate meta store for PROD, and NON-PROD(QA and DEV)? If yes they need t...
@Ashok Zubrewar​ coming to your 3 rd question, if you are using any external tables then non uc ADLS GEN 2 is mandatory, you can not use UC ADLS GEN2. as it hosts metadata and managed table data. there is no restriction in terms of your external buc...
I was wondering if there is a best practice or recommended way to organize data objects (tables) in Unity Catalog.Following the medallion architecture, how should one organize and name these three namespaces:Catalog(s)Schema(s)Table(s)Example:Catalog...
In Real time do we have 1 meta store for PROD and NON-PROD ? if Yes in the backend there would be a one storage account (ADLS Gen2) > Container for all Environments.
Free e-book: Rise of the Data LakehouseExciting news! You can now get a free copy of Bill Inmon's new book, Rise of the Data Lakehouse. The book is a must-read for anyone looking to implement the data lakehouse approach in their organization. The dat...
i have unit setup, and am trying to create a delta schema and table, i am getting an error on the schema creation, i am able to list the files and folders in the ADLS Gen2 storage account, i am able to write a parquet file to ADLS, but i cannot crea...
Hi @Shep Sheppard​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...
I was wondering how the Unity Catalog would help in data quality validations for real time (streaming data) data ingestion.
Hi @arun laksh​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
SET MASK and SET ROW FILTER are coming to the databricks unity catalog Inside, you use is_account_group_member() to check for account-level groups or is_member() for workplace-level groups. is_account_group_member() is recommended with Unity Catalog.
Hi,I am trying to find queries I run in a notebook (running on a cluster) in Cluster Logs.I set the cluster to deliver logs to a folder on DBFS and I can read log files from there. I created a Databricks workspace on the premium pricing tier and it i...
@Mohammad Saber​ :If you are not seeing the expected logs in the log files, it's possible that either logging was not properly configured or the logs have been rotated out of the active log file and into an archive file. Here are some suggestions you...
Cloud: AWSRegion: eu-west-1S3 location: s3://databricks-dev-bucketIAM role ARN: arn:aws:iam::18XXXXXXXX29:role/databricks-s3-metastoreGuide followed: ref: https://docs.databricks.com/data-governance/unity-catalog/get-started.html#cloud-tenant-setup-a...
Bump. @Yeshaswini P V​ @Gokul Kumar P​
What is the most cost-effective way to sync delta tables across workspaces? I'd like to sync a production table from the production workspace to a staging workspace for testing. It doesn't have to be in real-time, on a daily basis is acceptable. We a...
Hi @Liz Zhang​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we ...
maybe this is a bug in the unity catalog as the error happens multiple times when a table becomes inaccessibleerror: Request failed with status code 404try to drop the table error: NoSuchTableException: [TABLE_OR_VIEW_NOT_FOUND] try to replace the ta...
the error disappeared now after a few months.It could behappens when there is a lot of schema change on db during the initial project stage.there may be some temp security issuesmay be databricks fixed in the back end appears less issue when schema r...
