topic Is there a way to validate the values of spark configs? in Data Engineering

Is there a way to validate the values of spark configs?

Anonymous — Thu, 10 Jun 2021 21:54:43 GMT

We can set for example:

spark.conf.set('aaa.test.junk.config', 99999) , and then run spark.conf.get("aaa.test.junk.config”) which will return a value.

The problem occurs when incorrectly setting to a similar matching property.

spark.conf.set('spark.sql.shuffle.partition', 999) ==> without the trailing ’s'

Where the actual property is: ‘spark.sql.shuffle.partitions' ==> has a training ’s’

Running spark.conf.get('spark.sql.shuffle.partition’) will return a value ==> without the trailing ’s'

I thought I could run the getAll() as a validation, but the getAll() may not return properties that are explicitly defined in a Notebook session.

Is there a way to check if what I have used as config parameter is actually valid or not? I don't see any error message either

Re: Is there a way to validate the values of spark configs?

sajith_appukutt — Thu, 17 Jun 2021 18:19:20 GMT

You could check the list of valid configurations here and ensure that there are no typos

Re: Is there a way to validate the values of spark configs?

User16857281974 — Fri, 30 Jul 2021 21:38:33 GMT

You would solve this just like we solve this problem for all lose string references. Namely, that is to create a constant that represents the key-value you want to ensure doesn't get mistyped.

Naturally, if you type it wrong the first time, it will be wrong everywhere, but that is true for all software development. Beyond that, a simple assert will avoid regressions where someone might change your value.

The good news is that this has already been done for you and you could simply include it in your code if you wanted to as seen here:

If you are using Python, I would go ahead and create your own constants and assertions give that integrating with the underlying Scala code just wouldn't be worth it (in my opinion)