topic Re: Copying files from dev environment to prod environment in Data Engineering

Copying files from dev environment to prod environment

pshuk — Wed, 20 Mar 2024 19:18:20 GMT

Hi,

Is there a quick and easy way to copy files between different environments? I have copied a large number of files on my dev environment (unity catalog) and want to copy them over to production environment. Instead of doing it from scratch, can I just copy them from one to another?

TIA

Re: Copying files from dev environment to prod environment

Hubert-Dudek — Wed, 20 Mar 2024 19:53:20 GMT

If you want to copy files in Azure, ADF is usually the fastest option (for example TB of csvs, parquets). If you want to copy tables, just use CLONE. If it is files with code just use Repos and branches.

Re: Copying files from dev environment to prod environment

pshuk — Thu, 21 Mar 2024 15:05:41 GMT

Thanks Hubert. It's not tables. It's the raw file that I put in Volumes on my dev environment on databricks. How can I copy all the files to different environment on databricks.