topic Re: Programatic access to Files in Repos in Data Engineering

Programatic access to Files in Repos

Jiri_Koutny — Mon, 18 Oct 2021 14:17:16 GMT

Hi, we are testing the new Files support in Databricks repos. Is there a way how to programmatically read notebooks?

Thanks

Re: Programatic access to Files in Repos

Jiri_Koutny — Mon, 18 Oct 2021 14:24:42 GMT

Thanks @Kaniz Fatma for quick reply. Looking forward to the answers 🙂

Re: Programatic access to Files in Repos

Jiri_Koutny — Wed, 20 Oct 2021 09:25:26 GMT

We definitely work with Python files in Databricks Repos, not with local files.

Re: Programatic access to Files in Repos

Sandeep — Wed, 10 Nov 2021 16:06:00 GMT

@Jiri Koutny Alternatively, using export API, you can download it and use

curl -n -o example.scala \

'https://<databricks-instance>/api/2.0/workspace/export?path=/Users/user@example.com/ScalaExampleNotebook&direct_download=true'

Documentation https://docs.databricks.com/dev-tools/api/latest/workspace.html#export

Re: Programatic access to Files in Repos

Jiri_Koutny — Thu, 11 Nov 2021 08:21:19 GMT

Hi Sandeep, is it really possible to download files (not notebooks) from Repos using the export API? Last time I tried, only notebooks were exported and all files were skipped...

Re: Programatic access to Files in Repos

User16871418122 — Wed, 24 Nov 2021 03:37:48 GMT

Hi @Jiri Koutny these files anyway should be synced to your remote repository (git, bitbucket, GitLab etc). The APIs from version control tools Git API for example might help you achieve what you want.

https://stackoverflow.com/questions/38491722/reading-a-github-file-using-python-returns-html-tags

#!/usr/bin/env python3
import base64
import requests
 
 
url = 'https://api.github.com/repos/{user}/{repo_name}/contents/{path_to_file}'
req = requests.get(url)
if req.status_code == requests.codes.ok:
    req = req.json()  # the response is a JSON
    # req is now a dict with keys: name, encoding, url, size ...
    # and content. But it is encoded with base64.
    content = base64.decodestring(req['content'])
else:
    print('Content was not found.')