- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-29-2021 04:00 AM
Hello. I'm trying to connect to Databricks from my IDE (PyCharm) and then run delta table queries from there. However, the cluster I'm trying to access has to give me permission. In this case, I'd go to my cluster, run the cell which gives me permission and then go back to my PyCharm so I can query the tables. The question I'm trying to ask here is, is there some way to do this kind of configuration from PyCharm directly? This is the code I use to give permission
if StorageAccountName is not None and StorageAccountAccessKey is not None:
print('Configuring the spark context...')
spark_context._jsc.hadoopConfiguration().set(
f"fs.azure.account.key.{StorageAccountName}.blob.core.windows.net",
StorageAccountAccessKey)I use it this way, and not directly as a Spark configuration, because my StorageAccountAccessKey is stored in key-vault and a cluster might have info from various storage accounts which means I'd have various access keys, etc.