topic How do you create a Sandbox in your data environment ? in Data Engineering

How do you create a Sandbox in your data environment ?

William_Scardua — Mon, 29 Nov 2021 15:15:15 GMT

Hi guys,

How do you create a Sandbox in your data environment ? have any idea ?

Azzure/AWS + Data Lake + Databricks

Re: How do you create a Sandbox in your data environment ?

Hubert-Dudek — Mon, 29 Nov 2021 15:25:44 GMT

maybe use Azure DataFactory to periodically copy your data to sandbox storage.

If you need to configure mounts and sql databases/tables the best is to have notebook to do that which is run in both environments with widget which specify deployment (so it will replace mount and table location accordingly).

1) copy in dataFactory

2) when completed run deployment notebook (also can be with dataFactory and param available though widget can be set there)

Re: How do you create a Sandbox in your data environment ?

-werners- — Mon, 29 Nov 2021 17:49:27 GMT

Depends on how much you want the sandbox to be disconnected from the rest.

The ideal scenario is a complete separate setup, as in DEV-QA-PRD-SANDBOX.

But to be honest, I think that is overkill.

If you have a separate storage account or you have a separate BLOB or even a subdirectory which is protected with permissions, you have already quite a lot.

Because the data is the most important part.

Then the notebooks: you can opt for a separate databricks account but again, you can do without.

f.e. use Repos for your 'official' notebooks, and the workspace/user folder for playing around.

You only have to make sure that you use the correct mount and that can be set with a widget as Hubert mentioned.

It also depends on the amount of people working on databricks. If you are only a small team you do not have to be too strict. But with lots of people and frequent personnel changes (consultants f.e.) it is a good idea to have strict permissions/procedures etc.

Re: How do you create a Sandbox in your data environment ?

missyT — Tue, 30 Nov 2021 04:25:40 GMT

In a sandbox environment, you will find the Designer enabled. You can activate Designer by selecting the design icon Designer. on a page, or by choosing the Design menu item in the Settings Settings menu.

Re: How do you create a Sandbox in your data environment ?

William_Scardua — Thu, 02 Dec 2021 00:14:45 GMT

thank you @Hubert Dudek

Re: How do you create a Sandbox in your data environment ?

William_Scardua — Thu, 02 Dec 2021 00:15:11 GMT

I agree @Werner Stinckens , thank you

Re: How do you create a Sandbox in your data environment ?

William_Scardua — Thu, 02 Dec 2021 00:15:25 GMT

Thank you @Missy Trussell