Databricks

Nino · 09-11-2023

Concerning job cluster configuration, I'm trying to figure out what happens if AWS node type availability is smaller than the minimum number of workers specified in the configuration json (either availabilty<num_workers or, for autoscaling, availabil...

Nino · 08-09-2023

I have a few Databricks clusters, some share a single Hive Metastore (HMS), call them PROD_CLUSTERS, and an additional cluster, ADHOC_CLUSTER, which has its own HMS. All my data is stored in S3, as Databricks delta tables: PROD_CLUSTERS have read-wri...

Nino · 09-12-2023

thanks, @Kaniz , useful info!My specific scenario is running a notebook task with Job Clusters, and I've noticed that I get the best overall notebook run time by going without Autoscaling, setting the cluster configuration with a fixed `num_workers` ...

Nino · 09-01-2023

Hi @Kaniz , do you mean this ? couldn't see there's a GET_TABLE_FILES, not sure how to make it work, if you have a working example or some pointer it would be great. Thanks!

Nino · 08-24-2023

Hi @Mo , Sorry for the gap, just back from vacation...So yes, what you describe is possible, and quite close to what I'm doing now. The thing is, assuming this is not done manually, this requires:Setting up a process running all those "describe detai...

Nino · 08-11-2023

Something went wrong there, here's the last sentence: I expected "location" will be the s3 path but it's not always so (elaborated in the original posting). Thanks!

Nino · 08-11-2023

here's an example query I run from ADHOC_CLUSTER: df_sds = (spark.read .format("jdbc") .option("driver", driver) .option("url", "jdbc:mysql://..../prod_hms") .option("dbtable", "(select SD_ID, CD_ID, location from SDS) a") .option("user", "prod_hms...

Databricks

User Stats

User Activity

cluster nodes unavailable scenarios

Where in Hive Metastore can the s3 locations of Databricks tables be found?

Re: cluster nodes unavailable scenarios

Re: Where in Hive Metastore can the s3 locations of Databricks tables be found?

Re: Where in Hive Metastore can the s3 locations of Databricks tables be found?

Re: Where in Hive Metastore can the s3 locations of Databricks tables be found?

Re: Where in Hive Metastore can the s3 locations of Databricks tables be found?