Databricks has introduced Duration Thresholds for workflows. This new addition allows users to set time limits for workflow execution, significantly improving monitoring of job performance. When a job exceeds the preset duration, the system triggers ...
data coming in from multiple sources belonging to multiple customers.Should I create Single storage account name it bronze and then separate containers for each client's data to be put in.And then merge the data in the silver layer. What's the best p...
I want to set the account admin for a service principal in order to create the Unity Catalog metastore. The Terraform code looks like this: data "databricks_service_principal" "application" { count = var.environment == "dev" ? 1 : 0 application_...
We're trying to read from an S3 bucket using unity catalog and are selectively getting "INVALID_PARAMETER_VALUE.LOCATION_OVERLAP: overlaps with managed storage error" errors within the same bucket. This works: "dbutils.fs.ls("s3://BUCKETNAME/dev/he...
Hi,Could you please elaborate on the issue here? Running the list command on a managed directory is not supported in Unity Catalog. Catalog/schema storage locations are reserved for managed storage.Please tag @Debayan with your next comment which wi...
I am trying to add a Bitbucket cloud repo to my workspace, but keep getting a credentials invalid error. I made sure to set a username and generate an app password with repo read write permissions. What am i doing wrong?
Hello,Did you also set the password in User Settings > Git integration ? If so can you paste the full error?
Hi all, I have a user who is enrolled in multiple groups within databricks but she is only able to query one of the query tables based on the original group assigned. For example:Group 1 will only be able to view data where [group] (column) = 'group1...
Hey folks! I am seeking clarity for this passage in documentation: "This means that SSO for your account and SSO for each workspace must be managed separately. You should configure SSO to the the same identity provider at the account and on your work...
If both are pointing to the same IDP then it should not be a problem.
In my Account Console and workspace selector I'm seeing a list of 10 workspaces, but to half of them I don't have access anymore (as intended). How can I remove these workspaces which have become irrelevant for me from this view?
Hi @FrankBarten Hope you are well. Just wanted to see if you were able to find an answer to your question and would you like to mark an answer as best? It would be really helpful for the other members too. Cheers!
Metrics are the lifeblood of every business. They help measure performance, track progress, and optimize decision-making. Unfortunately, however, most projects' metrics are buried in code and scattered across different tools and platforms. This makes...
Thanks for sharing your knowledge with Community.
I am getting this type of erro "com.databricks.spark.sqldw.SqlDWConnectorException: Exception encountered in Azure Synapse Analytics connector code" while loading the data into Synapse. I am able to load other tables with the same connection. Not sur...
The error message you provided indicates that an exception occurred in the Azure Synapse Analytics connector code while loading data into Synapse. This can happen due to various reasons, such as issues with the table structure, data type compatibilit...
Where can I learn more about system tables? Looking for docs or usage stories! Thanks
Please refer to this - https://docs.databricks.com/administration-guide/system-tables/index.html
Hi, I am new to Databricks and I am exploring its CE. I have data science background and I am building an analytics webapp, instead of connecting to Supabase or other back-end services, is it possible for me to connect to Databricks directly? I could...
How about the Databricks REST API? DB offers a comprehensive REST API that allows you to programmatically interact with Databricks resources. You can use this API to submit jobs, run queries, manage clusters, and perform various other operations.
Hi,Has anybody tried using Duo for SSO (Single Sign On)?We used OIDC on Duo, and getting this error when we test. Basically, after we get authenticated by Duo, the redirect page shows up like this:Any tips would be greatly appreciated. Thanks!
Hi @yubin-apollo We are reaching out to inform you that Databricks Community has migrated to a new service provider. As a result, all users and published content are migrated on June 24, 2023.This change will enhance your overall Databricks Community...
Hi, I am looking for a way to better migrate our workspace to Unity Catalog. We have several tables created as external tables and created using mount points, but I am worried that migrating to unity catalog will break those tables if we migrate usin...
@Rodrigo_Mohr you need to mainly consider below things1. Number of managed tables2. Number of external tablesin case external tables upgrade tool will help which is built in with UC, where as for managed you need to either go CTAS/Deep clone. but we ...
The company is getting a high expensive cost in data traffic after setting up the VPC, now we notice that we have to set up a s3 gateway, I wonder what is the best practices
@Ayun In terms of s3 gateway wrt to aws that looks promising in terms of price, but i have not seen anything related to s3 gateway mentioned in databricks articles. for most of implementations we went with standard approach that databricks mentioned....
