Here's your Data + AI Summit 2024 - Data Governance recap as you navigate the explosion of AI, data and tools in efforts to build a flexible and scalable governance framework that spans your entire data and AI estate. Keynote: Evolving Data Governan...
I would like to create an external location on a publicly available S3 bucket, for which I don't have credentials. I get a syntax error unless I include credentials. Is there a way to do this? CREATE EXTERNAL LOCATION public_bucket URL 's3://public_b...
Hi @jerickson ,I have tested this on Databricks Runtime 14.3 LTS:Install on the cluster the below Maven packages:com.amazonaws:aws-java-sdk-bundle:1.12.262org.apache.hadoop:hadoop-aws:3.3.4Run the below code to read your csv file into dataframe: spar...
Hi everyone,I need to transfer tables present in the Hive metastore from one workspace to another. My ultimate goal is to migrate these tables to Unity Catalog. I already have a code that will handle the migration to Unity Catalog, but I need help wi...
Are these tables managed tables or external tables? This is very important step for your migration.
Hi Team,I have setup UC and have a Metastore for AUE. Assigned 2 workspaces to the AUE Metastore and both with different set of users and group . 1 workspace is the data provider and the other is a data consumerNow since 2 workspaces are in one Meta...
Do we have any updates on this in terms of how we can share the data between different workspaces part of the same metastore?
Hello Everyone,I have an AWS Databricks premium trial account and I’m trying to enable the configuration "Enable external data access on the metastore" for a metastore. I followed the steps in the official documentation (https://docs.databricks.com/e...
Hi @Praveen Thank you for sharing your concern here. You are looking into the Catalog on the account console, but according to the document, you have to do this in the Workspace Catalog UI. Please follow the below steps, Log into any of the workspace...
Scenario Under Hive Metastore four diff schema exist S1,S2,S3 and S4 User1 want access to schema S1 but not allowed to see other 3 Schema which can be done using grant permission simple and easy case Same User1 has access to workspace. Now same user ...
The best option for Governance is to upgrade to Unity Catalog. This is alternate, not the best. The cluster must be enabled for table access control. https://docs.databricks.com/en/data-governance/table-acls/object-privileges.html#requirements
Hi Databricks Community,I’m encountering some challenges when working with external tables in Databricks. I’m using both Unity Catalog and Hive Meta store and have noticed differences in how external tables are handled.Unity Catalog: I can successful...
hi @jv_v , that's possible to put a path to external storage i.e.:documentationCREATE OR REPLACE TABLE hive_metastore.schema.table LOCATION 's3://depts/finance/sec_filings'Can you share which issue do you have cause it's not clear from your questio...
Hi,Could you please give me example how to search by both tag key and tag value?This article doesn't show such example: https://docs.databricks.com/en/search/index.html#tagsThanks,Karol
If you want to search by tag key alone, use the syntax: tag:<tag_key>. To search by both tag key and tag value, omit tag:. https://docs.databricks.com/en/search/index.html#use-tags-to-search-for-tables
I need to know how much money each user spend in my workspace, i already have the system billing table, but there is not much information in it. How can i achive that?
You can tag the cluster and use this dashboard https://learn.microsoft.com/en-us/azure/databricks/admin/account-settings/usage
Hi @dataguy555 ,You will still need to have your external table in delta format. JSON or parquet are not supported:What is Delta Sharing? - Azure Databricks | Microsoft Learn
When attempting to write a dataframe to SQL Server using the provided "sqlserver" connector or the old "jdbc" one, it seems to be breaking due to old values used to build the dataframe. The code I'm using to write is:```python( df.write.format("sq...
Hi @MarthinusB, How are you doing today?To overcome this, Give a try explicitly defining the schema for your DataFrame to ensure no unwanted metadata is retained. You can also recreate the DataFrame using rdd.toDF() or checkpoint it to drop unnecessa...
I have a delta table with a column mask (azure databricks in eastus2 metastore) and another with no mask. I shared both tables with westeurope unity catalog metastore azure region via delta sharing (created share object, added the table as asset, ...
@prasadvaze That's correct, Delta Sharing does not work with RLS or column mask. It's well described in the documentation: https://docs.databricks.com/en/tables/row-and-column-filters.html#limitations
Currently, Databricks allows us to grant access to catalogs, schemas, and objects individually, but this process can be repetitive and time-consuming when managing access for multiple applications across different schemas. A feature that enables the...
Hi @sdheepak, Thanks for reaching out! Please review the response and let us know which best addresses your question. Your feedback is valuable to us and the community. If the response resolves your issue, kindly mark it as the accepted solution. Thi...
Hello,Is there any plan to support the equivalent of fabric semantic-link on Databricks? Essentially, having the ability to query PowerBi Dataset using SQL on interactive/jobs cluster and SQL warehouse (i.e a Unity Catalog federated source being Powe...
I think partners connect give option to connect BI tools and their sementic models to DBX not DBX to SM,in case you want to use models used for reports for AI/ML processing in dbx, you need to use something like semantic link to get data from SM dire...
We have an existing metastore in a region. We would like to delete it (Create a Unity Catalog metastore - Azure Databricks | Microsoft Learn) and create a new metastore in that same region. Are there any issues that we should be aware of with creatin...
I have exactly the same query.But before that, is it at all a good practice to delete the default metastore ? I am asking this because Databricks creates a lot of things on the Azure side like a storage account, access connector, managed identity, li...
Hi all,There is no official doc with a step-by-step process to enable Unity Catalog in azure databricks.If anyone has created doc or has the process, please share it here.Thanks, in advance.
Hi everyone!I have an Azure for Students subscription. Since I am unable to be an Azure Active Directory Global Administrator, I tried but it is restricted by the organization to add my user as a Global Administrator, I guess with an Azure for Studen...
