topic Re: Global or environment parameters. in Data Engineering

Global or environment parameters.

Data_Engineeri7 — Tue, 23 Jan 2024 16:57:37 GMT

Hi All,

Need a help on creating utility file that can be use in pyspark notebook.

Utility file contain variables like database and schema names. So I need to pass this variables in other notebook wherever I am using database and schema.

Thanks

Re: Global or environment parameters.

ilarsen — Tue, 23 Jan 2024 21:24:04 GMT

Hi. Have you looked at using cluster spark configurations? We're using it for default catalog for example and it works quite well. I don't know for certain, but there might be an option to pre-set databases as well.

Here's a topic about setting catalog: Re: Set default database thru Cluster Spark Config... - Databricks - 47645

Re: Global or environment parameters.

KSI — Tue, 23 Jan 2024 22:39:52 GMT

You can use:

${param_catalog}.schema.tablename.
Pass actual value in the notebook through a job param "param_catalog" or widget utils through text called "param_catalog"

Re: Global or environment parameters.

ilarsen — Tue, 23 Jan 2024 22:46:15 GMT

Oh yeah, using widgets, of course!