topic Re: dbutils.fs.mv - 1 folder and 1 file with the same name and only move the folder in Data Engineering

dbutils.fs.mv - 1 folder and 1 file with the same name and only move the folder

SaraCorralLou — Sun, 09 Jul 2023 17:52:13 GMT

Hello!

I am contacting you because of the following problem I am having:

In an ADLS folder I have two items, a folder and an automatically generated Block blob file with the same name as the folder.

I want to use the dbutils.fs.mv command to move the folder to another folder but as the Block blob file has the same name it is also being moved together with the folder and this is causing me a problem.

Would it be possible to somehow just specify that I want to move the folder?

Thank you very much in advance!

Re: dbutils.fs.mv - 1 folder and 1 file with the same name and only move the folder

Hemant — Mon, 10 Jul 2023 02:23:25 GMT

Hi, @SaraCorralLou can you share a snapshot for more clarity?

Re: dbutils.fs.mv - 1 folder and 1 file with the same name and only move the folder

SaraCorralLou — Mon, 10 Jul 2023 08:13:02 GMT

Yes, sure.

This is the situation. The second file is this Block Blob file that does not have extension. The folder is the one that we want to move but when we do dbutils.fs.mv is moving both.

Re: dbutils.fs.mv - 1 folder and 1 file with the same name and only move the folder

Hemant — Mon, 10 Jul 2023 09:12:27 GMT

Thanks @SaraCorralLou for sharing snapshot, for this you have to handle the code itself while moving:

def move_folders(filepath, destination_path): """ This function moves all folders from the specified filepath to the specified destination_path. Args: filepath (str): The path to the source folders. destination_path (str): The path to the destination folders. """ # Iterate over all files in the source directory. for file in dbutils.fs.ls(filepath): # Check if the file is a directory. if file.isDir(): # Move the directory to the destination directory. dbutils.fs.mv(file.path, destination_path,True) # Print a message to indicate that all folders have been moved. print("All folders have been moved successfully.")

Re: dbutils.fs.mv - 1 folder and 1 file with the same name and only move the folder

SaraCorralLou — Mon, 10 Jul 2023 12:08:11 GMT

Great! Thank you so much.

Re: dbutils.fs.mv - 1 folder and 1 file with the same name and only move the folder

Hemant — Mon, 10 Jul 2023 13:53:18 GMT

Thanks @SaraCorralLou, can you select it as the answer if it's work for you?

Re: dbutils.fs.mv - 1 folder and 1 file with the same name and only move the folder

Anonymous — Tue, 11 Jul 2023 10:34:53 GMT

Hi @SaraCorralLou

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!

Re: dbutils.fs.mv - 1 folder and 1 file with the same name and only move the folder

SaraCorralLou — Wed, 12 Jul 2023 09:14:41 GMT

Thank you for your answer.

At the end I had to add one condition more because when you have these to elements (the folder and the block blob file) Databricks "does not understand" that is a directory so I added this condition:
if "ckd_tvhdep_parquet" in file.name and file.isDir() == False to remove first the file and after that I could move the folder. Because at that time, when the file was no longer there, Databricks did identify it as a folder.

Thank you very much anyway, your answer was very useful!

Re: dbutils.fs.mv - 1 folder and 1 file with the same name and only move the folder

deep_coder16 — Wed, 09 Oct 2024 15:03:04 GMT

What are the possible reasons for the generation of those extra files with same name with zero bytes of data?