- 12321 Views
- 10 replies
- 4 kudos
How to pass configuration values to a Delta Live Tables job through the Delta Live Tables API
Hi Community,I have successfully run a job through the API but would need to be able to pass parameters (configuration) to the DLT workflow via the APII have tried passing JSON in this format:{ "full_refresh": "true", "configuration": [ ...
- 12321 Views
- 10 replies
- 4 kudos
- 4 kudos
You cannot pass parameters from a Databricks job to a DLT pipeline. Atleast not yet. You can see from the DLT rest API that there is no option for it to accept any parameters.But there is a workaround.But there is a workaround.With the assumption tha...
- 4 kudos
- 17659 Views
- 3 replies
- 3 kudos
How can I make Databricks API calls from notebook?
Access to Databricks APIs require the user to authenticate. This usually means creating a PAT (Personal Access Token) token. Conveniently, a token is readily available to you when you are using a Databricks notebook.databricksURL = dbutils.notebook....
- 17659 Views
- 3 replies
- 3 kudos
- 3 kudos
@User16752245312 You can use Databricks Secret Scope to manage sensitive data such as personal access tokens (PATs) securely. Storing your token in a secret scope ensures you don’t hard-code credentials in your notebook, making it more secure.For mo...
- 3 kudos
- 3818 Views
- 6 replies
- 2 kudos
Resolved! What would happen it my dynamic IP changed in IP Access list?
On this feature IP access lists IP access lists - Azure Databricks | Microsoft Docs, what we observe is that if your IP is not on the access list, you cannot modify the list via API since you are not on trusted location. What if I specify only 1 IP s...
- 3818 Views
- 6 replies
- 2 kudos
- 2 kudos
Curious to learn if somebody also figured out a way to solve for the above as we've encountered this situation and are now locked out...
- 2 kudos
- 67393 Views
- 5 replies
- 5 kudos
Asynchronous API calls from Databricks
Hi, I have to send thousands of API calls from a Databricks notebook to an API to retrieve some data. Right now, I am using a sequential approach using the python request package. As the performance is not acceptable anymore, I need to send my API c...
- 67393 Views
- 5 replies
- 5 kudos
- 5 kudos
Hey @Paul_Poco what about using the processpoolexecutor or threadypoolexecutor from the concurrent.futures module ? have u tried them or not . ?
- 5 kudos
- 3106 Views
- 5 replies
- 2 kudos
Job cluster support in jobs/runs/submit API
We are using jobs/runs/submit API of databricks to create and trigger a one-time run with new_cluster and existing_cluster configuration. We would like to check if there is provision to pass "job_clusters" in this API to reuse the same cluster across...
- 3106 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi, Any update on the above mentioned issue? Unable to submit a one time new job run (api/2.0 or 21/jobs/runs/submit) with shared job cluster or one new cluster has to be used for all TASKs in the job
- 2 kudos
- 9210 Views
- 4 replies
- 0 kudos
Unable to authenticate against https://accounts.cloud.databricks.com as an account admin.
ProblemI'm unable to authenticate against the https://accounts.cloud.databricks.com endpoint even though I'm an account admin. I need it to assign account level groups to workspaces via the workspace assignment api (https://api-docs.databricks.com/re...
- 9210 Views
- 4 replies
- 0 kudos
- 0 kudos
From this doc: To automate Databricks account-level functionality, you cannot use Databricks personal access tokens. Instead, you must use either OAuth tokens for Databricks account admin users or service principals. For more information, see:Use a s...
- 0 kudos
- 8116 Views
- 4 replies
- 1 kudos
Accidentally removing the service principal that owns the view seems to put the Unity Catalog in an illegal state. Can you fix this?
I renamed our service principal in Terraform, which forces a replacement where the old service principal is removed and a new principal with the same permission is recreated. The Terraform succeeds to apply, but when I try to run dbt that creates tab...
- 8116 Views
- 4 replies
- 1 kudos
- 1 kudos
This is also true for removing groups before unassigning them (removing and unassigning in Terraform)│ Error: cannot update grants: Could not find principal with name <My Group Name>
- 1 kudos
- 2416 Views
- 1 replies
- 0 kudos
CURL API - Error while parsing token: io.jsonwebtoken.ExpiredJwtException: JWT expired
I am running this code:curl -X --request GET -H "Authorization: Bearer <databricks token>" "https://adb-1817728758721967.7.azuredatabricks.net/api/2.0/clusters/list"And I am getting this error:2024-01-17T13:21:41.4245092Z </head>2024-01-17T13:21:41.4...
- 2416 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, Could you please renew the token and confirm?
- 0 kudos
- 4370 Views
- 4 replies
- 0 kudos
Resolved! Programmatically updating the “run_as_user_name” parameter for jobs
I am trying to write a process that will programmatically update the “run_as_user_name” parameter for all jobs in an Azure Databricks workspace, using powershell to interact with the Jobs API. I have been trying to do this with a test job without suc...
- 4370 Views
- 4 replies
- 0 kudos
- 0 kudos
Solution you've submitted is a solution for different topic (permission to run job, the job still runs as the user in run_as_user_name field). Here is an example of changing "run_as_user_name"Docs:https://docs.databricks.com/api/azure/workspace/job...
- 0 kudos
- 1775 Views
- 1 replies
- 0 kudos
Can Kaggle Kernels be used in Databricks notebooks?
suggestions).This one looks perfect kaggle kernels output rsrishav/starter-youtube-trending-video-dataset -p /path/to/dest but I'm not using CLI, I'm using a databricks notebook.I tried using this code but it doesn't work. data_path = 'rsrishav/youtu...
- 1775 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @jch Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 0 kudos
- 3384 Views
- 5 replies
- 2 kudos
Resolved! Is there a way to list the dlt maintenance jobs through the API?
After creating the delta pipeline, I would like to get details from the dlt maintenance job automatically created by Databricks, like the scheduled time when the dlt maintenance tasks will be executed. However, it seems the Job API 2.1 doesn't cover ...
- 3384 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi @Debayan Mukherjee ,Actually the Databricks Jobs API documentation has not been fixed yet. The parameter `job_type` should be included in the list endpoint request documentation. Please do this in order to avoid unnecessary questions here in the ...
- 2 kudos
- 4011 Views
- 4 replies
- 1 kudos
Is it possible to create "Notification destinations"/"Alert destinations" through API or Pulumi/Terraform?
Hello, I'm looking for a way of defining notification destination using API or Pulumi/Terraform providers. However I cannot find it anywhere. Could you please help and advice if i'm missing something or it's not available at the moment?And If it's no...
- 4011 Views
- 4 replies
- 1 kudos
- 1 kudos
This issue seems to point to the lack of a public API being the culprit behind the lack of a resource for Terraform.
- 1 kudos
- 938 Views
- 1 replies
- 1 kudos
API to get Databricks Status AWS.
Hi, Do you have an api endpoint to call to get the databricks status for AWS?Thanks,
- 938 Views
- 1 replies
- 1 kudos
- 1 kudos
@Chinu Lee you have webhook/slack that can be used to fetch status https://docs.databricks.com/resources/status.html#webhookare you specifically looking for your account workspace/above one
- 1 kudos
- 34965 Views
- 7 replies
- 13 kudos
Resolved! How to read data from a table into a dataframe outside of Databricks environment?
Hi, I am working on an ML project and I need to access the data in tables hosted in my Databricks cluster through a notebook that I am running locally. This has been very easy while I run the notebooks in Databricks but I cannot figure out how to do ...
- 34965 Views
- 7 replies
- 13 kudos
- 13 kudos
We can use Apis and pyodbc to achieve this. Once go through the official documentation of databricks that might be helpful to access outside of the databricks environment.
- 13 kudos
- 2906 Views
- 1 replies
- 1 kudos
Resolved! What is the best way to use credentials for API calls from databricks notebook?
Hello, I have an Databricks account on Azure, and the goal is to compare different image tagging services from Azure, GCP, AWS via corresponding API calls, with Python notebook. I have problems with GCP vision API calls, specifically with credentials...
- 2906 Views
- 1 replies
- 1 kudos
- 1 kudos
Ok, here is a trick: in my case, the file with GCP credentials is stored in notebook workspace storage, which is not visible to os.environ() command. So solution is to read a content of this file, and save it to the cluster storage attached to the no...
- 1 kudos