topic Re: How to check file exists in databricks in Data Engineering

How to check file exists in databricks

SimhadriRaju — Thu, 25 Jul 2019 07:34:46 GMT

I Have a while loop there i have to check a file exists or not if exists read the file in a data frame else go to another file

Re: How to check file exists in databricks

User16857282152 — Thu, 31 Oct 2019 04:02:18 GMT

There might be other ways to do this, but this works.

@Simhadri Raju

Basically,

I use dbutils.fs.head here, but anything that throws an exception if it fails to find the file would work.

So I go to read the first byte of the file with

dbutils.fs.head(arg1,1)

If that throws an exception I return False

If that succeeds I return True.

Put that in a function, call the function with your filename and you are good to go.

Full code here

## Function to check to see if a file exists

def fileExists (arg1): try: dbutils.fs.head(arg1,1) except: return False; else: return True;

Calling that function with your filename

ilename = <pathtoyourfile>

if(fileExists(filename)): print("Yes it exists");

Re: How to check file exists in databricks

User16857282152 — Thu, 31 Oct 2019 04:03:20 GMT

Oh that code did not render well,

Re: How to check file exists in databricks

zerogjoe — Tue, 10 Dec 2019 09:49:29 GMT

Here's a Python solution:

def file_exists(path):
  try:
    dbutils.fs.ls(path)
    return True
  except Exception as e:
    if 'java.io.FileNotFoundException' in str(e):
      return False
    else:
      raise

Re: How to check file exists in databricks

dughub — Mon, 04 May 2020 16:32:56 GMT

Very many thanks to @zerogjoe for his elegant answer, which works perfectly for Databricks formatted file paths.

To make this a little more robust and allow for filesystem api paths (that can be used with os, glob etc and start with "/dbfs") I've added a few lines of code.

def exists(path): """ Check for existence of path within Databricks file system. """

if path[:5] == "/dbfs":
     import os
     return os.path.exists(path)
 else:
     try:
         dbutils.fs.ls(path)
         return True
     except Exception as e:
         if 'java.io.FileNotFoundException' in str(e):
             return False
         else:
             raise

Re: How to check file exists in databricks

Hari_Gopinath — Wed, 01 Dec 2021 20:26:49 GMT

If you are looking for a scala solution, here it is:

def pathExists(tablePath: String): Boolean = {
  try{
    dbutils.fs.ls(tablePath)
    return true
  } catch {
    case e: java.io.FileNotFoundException => println("Given path cannot be located")
    return false
  }
}

Re: How to check file exists in databricks

Mustious — Mon, 06 Nov 2023 11:15:43 GMT

You can do so by running the snippet below, which uses the new [databricks python SDK](https://github.com/databricks/databricks-sdk-py/😞

Install the package:
`pip install databricks-sdk`

Python snippet:
```python
from databricks.sdk import WorkspaceClient

# Remember to change the arguments below
w_client = WorkspaceClient(host="my_host", token="my_db_tokens")

# add an absolute path
dbfs_path_exist = w_client.dbfs.exists('/dbfs_my_path')
```

Hope it helps 🙂

Re: How to check file exists in databricks

Amit_Dass — Fri, 12 Jan 2024 00:31:01 GMT

How to check if a file exists in DBFS?

Let's write a Python function to check if the file exists or not

-------------------------------------------------------------

def file_exists(path):

try:

dbutils.fs.ls(path)

return True

except Exception as e:

if 'java.io.FileNotFoundException' in str(e):

return False

else:

raise

Result = file_exists("dbfs:/Repos/")

print(Result)

"True" represents that path exists