Here's your Data + AI Summit 2024 - Data Governance recap as you navigate the explosion of AI, data and tools in efforts to build a flexible and scalable governance framework that spans your entire data and AI estate. Keynote: Evolving Data Governan...
I would like to set up GitLab and Grafana servers using Databricks Container Services (DCS). The reason is that our development team is small, and the management costs of using EKS are not justifiable. We want to make GitLab and Grafana accessible in...
Yes, it’s possible to set up GitLab and Grafana servers using Databricks Container Services (DCS). Here’s a high-level overview of how you can achieve this:Prepare Your Docker Images:Create Docker images for GitLab and Grafana. You can use official ...
I would like to create an external location on a publicly available S3 bucket, for which I don't have credentials. I get a syntax error unless I include credentials. Is there a way to do this? CREATE EXTERNAL LOCATION public_bucket URL 's3://public_b...
Hi @jerickson ,I have tested this on Databricks Runtime 14.3 LTS:Install on the cluster the below Maven packages:com.amazonaws:aws-java-sdk-bundle:1.12.262org.apache.hadoop:hadoop-aws:3.3.4Run the below code to read your csv file into dataframe: spar...
Hi everyone!I would love to know whether Databricks has the following capabilities and what are the core functionalities of the tool:Data glossary: provides clear definitions of business terms, ensuring that everyone in the organization uses the same...
Hi everyone,I need to transfer tables present in the Hive metastore from one workspace to another. My ultimate goal is to migrate these tables to Unity Catalog. I already have a code that will handle the migration to Unity Catalog, but I need help wi...
Are these tables managed tables or external tables? This is very important step for your migration.
Hi Team,I have setup UC and have a Metastore for AUE. Assigned 2 workspaces to the AUE Metastore and both with different set of users and group . 1 workspace is the data provider and the other is a data consumerNow since 2 workspaces are in one Meta...
Do we have any updates on this in terms of how we can share the data between different workspaces part of the same metastore?
Hello Everyone,I have an AWS Databricks premium trial account and I’m trying to enable the configuration "Enable external data access on the metastore" for a metastore. I followed the steps in the official documentation (https://docs.databricks.com/e...
Hi @Praveen Thank you for sharing your concern here. You are looking into the Catalog on the account console, but according to the document, you have to do this in the Workspace Catalog UI. Please follow the below steps, Log into any of the workspace...
Scenario Under Hive Metastore four diff schema exist S1,S2,S3 and S4 User1 want access to schema S1 but not allowed to see other 3 Schema which can be done using grant permission simple and easy case Same User1 has access to workspace. Now same user ...
The best option for Governance is to upgrade to Unity Catalog. This is alternate, not the best. The cluster must be enabled for table access control. https://docs.databricks.com/en/data-governance/table-acls/object-privileges.html#requirements
Hi Databricks Community,I’m encountering some challenges when working with external tables in Databricks. I’m using both Unity Catalog and Hive Meta store and have noticed differences in how external tables are handled.Unity Catalog: I can successful...
hi @jv_v , that's possible to put a path to external storage i.e.:documentationCREATE OR REPLACE TABLE hive_metastore.schema.table LOCATION 's3://depts/finance/sec_filings'Can you share which issue do you have cause it's not clear from your questio...
Hi,Could you please give me example how to search by both tag key and tag value?This article doesn't show such example: https://docs.databricks.com/en/search/index.html#tagsThanks,Karol
If you want to search by tag key alone, use the syntax: tag:<tag_key>. To search by both tag key and tag value, omit tag:. https://docs.databricks.com/en/search/index.html#use-tags-to-search-for-tables
I need to know how much money each user spend in my workspace, i already have the system billing table, but there is not much information in it. How can i achive that?
You can tag the cluster and use this dashboard https://learn.microsoft.com/en-us/azure/databricks/admin/account-settings/usage
Hi @dataguy555 ,You will still need to have your external table in delta format. JSON or parquet are not supported:What is Delta Sharing? - Azure Databricks | Microsoft Learn
hi..I want to use 1 all purpose cluster and use dynamic taging in each notebook.for example tag project and department.I want to be able to change the tag from the notebook so I will be able to understand the costs for each project and departme...
Is it possible to update the schema/change the type of individual feature in a feature store?
When attempting to write a dataframe to SQL Server using the provided "sqlserver" connector or the old "jdbc" one, it seems to be breaking due to old values used to build the dataframe. The code I'm using to write is:```python( df.write.format("sq...
Hi @MarthinusB, How are you doing today?To overcome this, Give a try explicitly defining the schema for your DataFrame to ensure no unwanted metadata is retained. You can also recreate the DataFrame using rdd.toDF() or checkpoint it to drop unnecessa...
I have a delta table with a column mask (azure databricks in eastus2 metastore) and another with no mask. I shared both tables with westeurope unity catalog metastore azure region via delta sharing (created share object, added the table as asset, ...
@prasadvaze That's correct, Delta Sharing does not work with RLS or column mask. It's well described in the documentation: https://docs.databricks.com/en/tables/row-and-column-filters.html#limitations
