cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

dbutils.fs.mv - 1 folder and 1 file with the same name and only move the folder

SaraCorralLou
New Contributor III

Hello!

I am contacting you because of the following problem I am having:

In an ADLS folder I have two items, a folder and an automatically generated Block blob file with the same name as the folder.

I want to use the dbutils.fs.mv command to move the folder to another folder but as the Block blob file has the same name it is also being moved together with the folder and this is causing me a problem.

Would it be possible to somehow just specify that I want to move the folder?

Thank you very much in advance!

1 ACCEPTED SOLUTION

Accepted Solutions

Hemant
Valued Contributor II

Thanks @SaraCorralLou  for sharing snapshot, for this you have to handle the code itself while moving:

def move_folders(filepath, destination_path):
    """
    This function moves all folders from the specified filepath to the specified destination_path.

    Args:
     filepath (str): The path to the source folders.
     destination_path (str): The path to the destination folders.
    """

    # Iterate over all files in the source directory.
    for file in dbutils.fs.ls(filepath):
        # Check if the file is a directory.
        if file.isDir():
           # Move the directory to the destination directory.
           dbutils.fs.mv(file.path, destination_path,True)

    # Print a message to indicate that all folders have been moved.
    print("All folders have been moved successfully.")

 

Hemant Soni

View solution in original post

8 REPLIES 8

Hemant
Valued Contributor II

Hi, @SaraCorralLou  can you share a snapshot for more clarity?

Hemant Soni

SaraCorralLou
New Contributor III

Yes, sure. 

SaraCorralLou_0-1688976633463.png

This is the situation. The second file is this Block Blob file that does not have extension. The folder is the one that we want to move but when we do dbutils.fs.mv is moving both.

Hemant
Valued Contributor II

Thanks @SaraCorralLou  for sharing snapshot, for this you have to handle the code itself while moving:

def move_folders(filepath, destination_path):
    """
    This function moves all folders from the specified filepath to the specified destination_path.

    Args:
     filepath (str): The path to the source folders.
     destination_path (str): The path to the destination folders.
    """

    # Iterate over all files in the source directory.
    for file in dbutils.fs.ls(filepath):
        # Check if the file is a directory.
        if file.isDir():
           # Move the directory to the destination directory.
           dbutils.fs.mv(file.path, destination_path,True)

    # Print a message to indicate that all folders have been moved.
    print("All folders have been moved successfully.")

 

Hemant Soni

SaraCorralLou
New Contributor III

Great! Thank you so much.

Hemant
Valued Contributor II

Thanks @SaraCorralLou, can you select it as the answer if it's work for you?

 

Hemant Soni

Thank you for your answer.

At the end I had to add one condition more because when you have these to elements (the folder and the block blob file) Databricks "does not understand" that is a directory so I added this condition:
if "ckd_tvhdep_parquet" in file.name and file.isDir() == False to remove first the file and after that I could move the folder. Because at that time, when the file was no longer there, Databricks did identify it as a folder. 

Thank you very much anyway, your answer was very useful!

Anonymous
Not applicable

Hi @SaraCorralLou 

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance! 

deep_coder16
New Contributor II

What are the possible reasons for the generation of those extra files with same name with zero bytes of data? 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group