cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

yubin-apollo
by New Contributor II
  • 420 Views
  • 1 replies
  • 0 kudos

Single Sign On with Duo - Redirect Error

Hi,Has anybody tried using Duo for SSO (Single Sign On)?We used OIDC on Duo, and getting this error when we test. Basically, after we get authenticated by Duo, the redirect page shows up like this:Any tips would be greatly appreciated. Thanks!

screencapture-accounts-cloud-databricks-login-2023-06-30-19_18_44.png
  • 420 Views
  • 1 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @yubin-apollo We are reaching out to inform you that Databricks Community has migrated to a new service provider. As a result, all users and published content are migrated on June 24, 2023.This change will enhance your overall Databricks Community...

  • 0 kudos
Rodrigo_Mohr
by New Contributor II
  • 3326 Views
  • 2 replies
  • 0 kudos

Strategy for Unity Catalog migration

Hi, I am looking for a way to better migrate our workspace to Unity Catalog. We have several tables created as external tables and created using mount points, but I am worried that migrating to unity catalog will break those tables if we migrate usin...

  • 3326 Views
  • 2 replies
  • 0 kudos
Latest Reply
karthik_p
Esteemed Contributor
  • 0 kudos

@Rodrigo_Mohr you need to mainly consider below things1. Number of managed tables2. Number of external tablesin case external tables upgrade tool will help which is built in with UC, where as for managed you need to either go CTAS/Deep clone. but we ...

  • 0 kudos
1 More Replies
Ayun
by New Contributor II
  • 765 Views
  • 2 replies
  • 0 kudos

Best practice to use s3 gateway with VPC

The company is getting a high expensive cost in data traffic after setting up the VPC, now we notice that we have to set up a s3 gateway, I wonder what is the best practices

  • 765 Views
  • 2 replies
  • 0 kudos
Latest Reply
karthik_p
Esteemed Contributor
  • 0 kudos

@Ayun In terms of s3 gateway wrt to aws that looks promising in terms of price, but i have not seen anything related to s3 gateway mentioned in databricks articles. for most of implementations we went with standard approach that databricks mentioned....

  • 0 kudos
1 More Replies
msserpa
by New Contributor II
  • 2219 Views
  • 1 replies
  • 0 kudos

Resolved! Issue with Enabling Unity Catalog in Workspace

I hope this message finds you all well.I followed the necessary steps to enable the Unity Catalog in our workspace.However, upon saving the configuration, I noticed that the Unity Catalog feature is automatically being disabled.I have double-checked ...

169300_0-1688056840335.png
Administration & Architecture
AWS
S3
Unity Catalog
  • 2219 Views
  • 1 replies
  • 0 kudos
Latest Reply
karthik_p
Esteemed Contributor
  • 0 kudos

@msserpa where you are enabling this config please, usually if you add workspace in data--> metastore--> workspace , then you should not get any issue

  • 0 kudos
JohannaAK
by New Contributor
  • 553 Views
  • 1 replies
  • 1 kudos

Restrict access to Hive Metastore in environments

 Hi, We have issues with users creating tables and storing data in hive metastore in environments with UC. How to you restrict this access? 

  • 553 Views
  • 1 replies
  • 1 kudos
Latest Reply
karthik_p
Esteemed Contributor
  • 1 kudos

@JohannaAK please restrict users in catalog level, by only providing select operation, looks all permissions are enabled for users 

  • 1 kudos
rmennes
by New Contributor II
  • 262 Views
  • 0 replies
  • 0 kudos

API to access query data plan

I'm working on a tool to visualise who accessed which (unity) catalogs, schemas and tables. To do that, I would like to access the query plan of the queries in the query history. Unfortunately, It seems like the REST api doesn't support accessing tho...

  • 262 Views
  • 0 replies
  • 0 kudos
spallinti
by New Contributor
  • 597 Views
  • 1 replies
  • 0 kudos

Bronze layer tables

What type of tables do we use for Bronze layer. Managed or external when our raw data is in csv file

  • 597 Views
  • 1 replies
  • 0 kudos
Latest Reply
Vinay_M_R
Valued Contributor II
  • 0 kudos

You can create either a managed or an unmanaged (external) table in the bronze layer, depending on your preference and use case. If you choose to create a managed table, Databricks will manage both the metadata and the data for the table. If you choo...

  • 0 kudos
Anonymous
by Not applicable
  • 1612 Views
  • 2 replies
  • 0 kudos
  • 1612 Views
  • 2 replies
  • 0 kudos
Latest Reply
Joe_Suarez
New Contributor III
  • 0 kudos

Redis offers various data structures such as strings, lists, sets, and hashes. Depending on your use case, select the appropriate data structure for storing the crm enrich data. For example, if you need to store key-value pairs, Redis hashes (HSET, H...

  • 0 kudos
1 More Replies
yuyang
by New Contributor
  • 1063 Views
  • 1 replies
  • 0 kudos

How can I use Delta Lake with AWS Athena and AWS Glue catalog?

Currently we use AWS Athena and aws glue catalog for our data lake. We would like to evaluate delta lake for data management. How shall we try this with the existing setup?  

  • 1063 Views
  • 1 replies
  • 0 kudos
Latest Reply
OlivierAllovon
New Contributor II
  • 0 kudos

Actually the Glue Hive Metastore integration with Unity Catalog as been announced today at the Databricks Summit.Give it a try herehttps://docs.databricks.com/archive/external-metastores/aws-glue-metastore.html

  • 0 kudos
carroll_q2
by New Contributor III
  • 826 Views
  • 2 replies
  • 0 kudos

Resolved! Connect Spark Cluster to SQL Endpoint

Hello! Is it possible to retrieve data from a SQL Endpoint in the Databricks SQL persona using the Data Science and Engineering persona?  In other words, I would like to use pyspark in DS&E to query a table in Databricks SQL.#DatabricksSQL#Databricks...

  • 826 Views
  • 2 replies
  • 0 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

you do not need the sql warehouse itself for that.  for ds & e you need a classic cluster (not a sql endpoint) anyway so you can easily read the tables from the metastore using spark.read.table().Connecting the sql endpoint to the ds cluster seems od...

  • 0 kudos
1 More Replies
User16826992666
by Valued Contributor
  • 2454 Views
  • 5 replies
  • 1 kudos

Resolved! Use different instance types in pools

I am wondering if it's possible to create a pool that has a mix of instance types in it?

  • 2454 Views
  • 5 replies
  • 1 kudos
Latest Reply
abagshaw
New Contributor III
  • 1 kudos

AWS Fleet instance types are now GA and available for clusters and instance pools. You can see more details here: https://docs.databricks.com/compute/aws-fleet-instances.html

  • 1 kudos
4 More Replies
User16826992666
by Valued Contributor
  • 1074 Views
  • 3 replies
  • 0 kudos

Resolved! Are there any scenarios where it doesn't make sense to use Spot Instances?

It seems like using spot instances make a lot of sense for cost savings. But are there any risks to using them? Or things to consider before enabling them?

  • 1074 Views
  • 3 replies
  • 0 kudos
Latest Reply
abagshaw
New Contributor III
  • 0 kudos

On AWS, to further improve the chance of acquiring spot instances, you can use the newly GA'd feature Fleet instance types: https://docs.databricks.com/compute/aws-fleet-instances.html

  • 0 kudos
2 More Replies
Labels