@Artem Sachuk​ :

One way to achieve landing zone cleansing is to use the Azure Storage SDK in a script or job after the successful load of the file via Autoloader.

First, you can use the Databricks dbutils.fs.ls() command to get the list of files in the landing zone directory. Then, using the Azure Storage SDK, you can delete the files that have already been loaded.

Here is an example Python code snippet that demonstrates this approach:

import os
from azure.storage.blob import BlobServiceClient
 
# Set your Azure Storage account details
account_name = "your_account_name"
account_key = "your_account_key"
 
# Set your landing zone directory path
landing_zone_path = "/mnt/landing_zone"
 
# Create a BlobServiceClient object to connect to your storage account
connect_str = "DefaultEndpointsProtocol=https;AccountName={};AccountKey={};EndpointSuffix=core.windows.net".format(account_name, account_key)
blob_service_client = BlobServiceClient.from_connection_string(connect_str)
 
# Get the list of files in the landing zone directory
landing_zone_files = dbutils.fs.ls(landing_zone_path)
 
# Loop through each file in the landing zone
for file in landing_zone_files:
    file_path = file.path
    file_name = os.path.basename(file_path)
    
    # Check if the file has already been loaded (you may need to modify this logic based on your specific use case)
    if file_name.startswith("loaded_"):
        
        # Delete the file from Azure Storage
        container_name = "your_container_name"
        blob_client = blob_service_client.get_blob_client(container=container_name, blob=file_name)
        blob_client.delete_blob()
        
        # Delete the file from the landing zone directory
        dbutils.fs.rm(file_path)

Note that in this example, the landing zone files that have already been loaded are assumed to have a prefix of "loaded_". You may need to modify the logic to suit your specific use case. Also, make sure to replace the account_name, account_key, landing_zone_path, and container_name variables with your own values.