cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Lakehouse Architecture


Forum Posts

Start a conversation
hattier
by • New Contributor II
  • 2566 Views
  • 1 replies
  • 1 kudos

Resolved! OIDC account-level SSO and SAML workspace-level SSO

Hey folks! I am seeking clarity for this passage in documentation: "This means that SSO for your account and SSO for each workspace must be managed separately. You should configure SSO to the the same identity provider at the account and on your work...

Administration & Architecture
Reply
  • 2566 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Prabakar
Esteemed Contributor III
  • 1 kudos

If both are pointing to the same IDP then it should not be a problem.

  • 1 kudos
FrankBarten
by • New Contributor II
  • 2351 Views
  • 5 replies
  • 0 kudos

How to hide workspace to which I don't have access anymore?

In my Account Console and workspace selector I'm seeing a list of 10 workspaces, but to half of them I don't have access anymore (as intended). How can I remove these workspaces which have become irrelevant for me from this view? 

FrankBarten_0-1688374156442.png
Administration & Architecture
Reply
  • 2351 Views
  • 5 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @FrankBarten  Hope you are well. Just wanted to see if you were able to find an answer to your question and would you like to mark an answer as best? It would be really helpful for the other members too. Cheers!

  • 0 kudos
4 More Replies
Hubert-Dudek
by • Esteemed Contributor III
  • 1911 Views
  • 1 replies
  • 4 kudos

Metrics are the lifeblood of every business. They help measure performance, track progress, and optimize decision-making. Unfortunately, however, most...

Metrics are the lifeblood of every business. They help measure performance, track progress, and optimize decision-making. Unfortunately, however, most projects' metrics are buried in code and scattered across different tools and platforms. This makes...

1684244491813
Administration & Architecture
Reply
  • 1911 Views
  • 1 replies
  • 4 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Thanks for sharing your knowledge with Community.

  • 4 kudos
Dharinip
by • New Contributor
  • 2016 Views
  • 1 replies
  • 0 kudos

Connection Error

I am getting this type of erro "com.databricks.spark.sqldw.SqlDWConnectorException: Exception encountered in Azure Synapse Analytics connector code" while loading the data into Synapse. I am able to load other tables with the same connection. Not sur...

Administration & Architecture
Reply
  • 2016 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

The error message you provided indicates that an exception occurred in the Azure Synapse Analytics connector code while loading data into Synapse. This can happen due to various reasons, such as issues with the table structure, data type compatibilit...

  • 0 kudos
Lwin
by • New Contributor
  • 613 Views
  • 1 replies
  • 0 kudos

Want to learn more about system tables

Where can I learn more about system tables? Looking for docs or usage stories! Thanks

Administration & Architecture
Metadata
Reply
  • 613 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Please refer to this - https://docs.databricks.com/administration-guide/system-tables/index.html

  • 0 kudos
Ting
by • New Contributor
  • 867 Views
  • 1 replies
  • 0 kudos

Can I use Databricks as a Backend?

Hi, I am new to Databricks and I am exploring its CE. I have data science background and I am building an analytics webapp, instead of connecting to Supabase or other back-end services, is it possible for me to connect to Databricks directly? I could...

Administration & Architecture
Reply
  • 867 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

How about the Databricks REST API? DB offers a comprehensive REST API that allows you to programmatically interact with Databricks resources. You can use this API to submit jobs, run queries, manage clusters, and perform various other operations. 

  • 0 kudos
yubin-apollo
by • New Contributor II
  • 516 Views
  • 1 replies
  • 0 kudos

Single Sign On with Duo - Redirect Error

Hi,Has anybody tried using Duo for SSO (Single Sign On)?We used OIDC on Duo, and getting this error when we test. Basically, after we get authenticated by Duo, the redirect page shows up like this:Any tips would be greatly appreciated. Thanks!

screencapture-accounts-cloud-databricks-login-2023-06-30-19_18_44.png
Administration & Architecture
single-sign-on
Reply
  • 516 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @yubin-apollo We are reaching out to inform you that Databricks Community has migrated to a new service provider. As a result, all users and published content are migrated on June 24, 2023.This change will enhance your overall Databricks Community...

  • 0 kudos
Rodrigo_Mohr
by • New Contributor II
  • 7913 Views
  • 2 replies
  • 0 kudos

Strategy for Unity Catalog migration

Hi, I am looking for a way to better migrate our workspace to Unity Catalog. We have several tables created as external tables and created using mount points, but I am worried that migrating to unity catalog will break those tables if we migrate usin...

Administration & Architecture
Summit23
Reply
  • 7913 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
karthik_p
Esteemed Contributor
  • 0 kudos

@Rodrigo_Mohr you need to mainly consider below things1. Number of managed tables2. Number of external tablesin case external tables upgrade tool will help which is built in with UC, where as for managed you need to either go CTAS/Deep clone. but we ...

  • 0 kudos
1 More Replies
Ayun
by • New Contributor II
  • 1084 Views
  • 2 replies
  • 0 kudos

Best practice to use s3 gateway with VPC

The company is getting a high expensive cost in data traffic after setting up the VPC, now we notice that we have to set up a s3 gateway, I wonder what is the best practices

Administration & Architecture
Reply
  • 1084 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
karthik_p
Esteemed Contributor
  • 0 kudos

@Ayun In terms of s3 gateway wrt to aws that looks promising in terms of price, but i have not seen anything related to s3 gateway mentioned in databricks articles. for most of implementations we went with standard approach that databricks mentioned....

  • 0 kudos
1 More Replies
msserpa
by • New Contributor II
  • 3081 Views
  • 1 replies
  • 0 kudos

Resolved! Issue with Enabling Unity Catalog in Workspace

I hope this message finds you all well.I followed the necessary steps to enable the Unity Catalog in our workspace.However, upon saving the configuration, I noticed that the Unity Catalog feature is automatically being disabled.I have double-checked ...

169300_0-1688056840335.png
Administration & Architecture
AWS
S3
Unity Catalog
Reply
  • 3081 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
karthik_p
Esteemed Contributor
  • 0 kudos

@msserpa where you are enabling this config please, usually if you add workspace in data--> metastore--> workspace , then you should not get any issue

  • 0 kudos
JohannaAK
by • New Contributor
  • 703 Views
  • 1 replies
  • 1 kudos

Restrict access to Hive Metastore in environments

 Hi, We have issues with users creating tables and storing data in hive metastore in environments with UC. How to you restrict this access? 

Administration & Architecture
Reply
  • 703 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
karthik_p
Esteemed Contributor
  • 1 kudos

@JohannaAK please restrict users in catalog level, by only providing select operation, looks all permissions are enabled for users 

  • 1 kudos
rmennes
by • New Contributor II
  • 315 Views
  • 0 replies
  • 0 kudos

API to access query data plan

I'm working on a tool to visualise who accessed which (unity) catalogs, schemas and tables. To do that, I would like to access the query plan of the queries in the query history. Unfortunately, It seems like the REST api doesn't support accessing tho...

Administration & Architecture
Reply
  • 315 Views
  • 0 replies
  • 0 kudos
spallinti
by • New Contributor
  • 798 Views
  • 1 replies
  • 0 kudos

Bronze layer tables

What type of tables do we use for Bronze layer. Managed or external when our raw data is in csv file

Administration & Architecture
Reply
  • 798 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Vinay_M_R
Valued Contributor II
  • 0 kudos

You can create either a managed or an unmanaged (external) table in the bronze layer, depending on your preference and use case. If you choose to create a managed table, Databricks will manage both the metadata and the data for the table. If you choo...

  • 0 kudos
Anonymous
by • Not applicable
  • 2104 Views
  • 2 replies
  • 0 kudos

Are there any best practices on using Redis with structured streaming (for data enrichment) ?

Administration & Architecture
Reply
  • 2104 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Joe_Suarez
New Contributor III
  • 0 kudos

Redis offers various data structures such as strings, lists, sets, and hashes. Depending on your use case, select the appropriate data structure for storing the crm enrich data. For example, if you need to store key-value pairs, Redis hashes (HSET, H...

  • 0 kudos
1 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels