you can create recursive function in python inside the databricks.

something like this.

def filedetails(path):

lists = dbutils.fs.ls(path)

global num

for i in lists:

if (i[1][-1] == "/"):

num += 1

lenfiles = dbutils.fs.ls(i[0])

modifiedlist.append((i[0],i[1],i[2],len(lenfiles)))

filedetails(i[0])