Databricks Community

giohappy · 07-13-2023

Managed tables are stored under the /user/hive/warehouse, which is also mentioned in the documentation. In our workflow, we use that path to read the parquet files from outside (through databricks connector). Can we assume this path is reliable, or i...

giohappy · 02-06-2023

The usual way to use Apache Sedona inside pySpark is by first registering Sedona types and functions withSedonaRegistrator.registerAll(spark)We need to have these autoregistered when the cluster start (to be able, for example, to perform geospatial q...

giohappy · 07-14-2023

In our case we haven't configured or created the metastore directly. We're relying on the default metastore, which is where the tables are written when we do:df.write.format("delta").mode("overwrite").saveAsTable(output_table_name)I haven't found any...

giohappy · 07-13-2023

Yes, that link was also mentioned in my question. The point is if our pipeline can always assume that the path is where the parquet files for the managed tables are expected to be, or it's just an internal detail that could change at any time.

Databricks Community

User Stats

User Activity

Can we assume the path to the managed tables in the hive_metastore is reliable?

SedonaSqlExtensions is not autoregistering types and functions

Re: Can we assume the path to the managed tables in the hive_metastore is reliable?

Re: Can we assume the path to the managed tables in the hive_metastore is reliable?