topic Encapsulate Databricks Pyspark/SparkSql code in Data Engineering

Encapsulate Databricks Pyspark/SparkSql code

Databricks_7045 — Mon, 07 Feb 2022 12:53:18 GMT

Hi All ,

I have Custom code ( Pyspark & SparkSQL) (notebooks) which I want to deploy at customer location and encapsulate so that end customers don't see the actual code. Currently we have all code in Notebooks (Pyspark/spark sql). Could you please let me know

1) if there is any way to run the code as executables/jars.

2) How to convert the code as executables.

Thanks a lot for the help.

Re: Encapsulate Databricks Pyspark/SparkSql code

-werners- — Tue, 08 Feb 2022 10:33:56 GMT

With notebooks that is not possible.

You can write your code in scala/java and build a jar, which you then run with spark-submit.

(example)

Or use python and deploy a wheel.

(example)

This can become quite complex when you have dependencies.

Also: a jar etc can be decompiled rendering your usecase useless.

Re: Encapsulate Databricks Pyspark/SparkSql code

Databricks_7045 — Tue, 08 Feb 2022 10:48:08 GMT

Hi , Thanks for the reply . Is it possible to restrict the view access by some premium plan ? please let me know . Thank You

Re: Encapsulate Databricks Pyspark/SparkSql code

-werners- — Tue, 08 Feb 2022 10:50:11 GMT

yes certainly.

You can put permissions on notebooks and folders.

But probably the customer will be admin of the databricks workspace.