cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Jamie_209389
by New Contributor III
  • 3546 Views
  • 7 replies
  • 3 kudos

Resolved! In Azure Databricks CLI, how to pass in the parameter notebook_params? Error: Got unexpected extra argument

I am trying to call run-now with notebook_params in Azure Databricks CLI, following https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/jobs-cliandescapse the quotes as stated in the documentationhttps://learn.microsoft.com/en-us/azure/d...

  • 3546 Views
  • 7 replies
  • 3 kudos
Latest Reply
Vaitheesh
New Contributor II
  • 3 kudos

I have the latest Databricks CLI setup and configured in my Ubuntu VM. When I tried to run a job using the json template I generated using databricks jobs get 'xxxjob_idxxx' > orig.json it throws an unknown error.Databricks CLI v0.216.0databricks job...

  • 3 kudos
6 More Replies
Yoni
by New Contributor
  • 10006 Views
  • 5 replies
  • 3 kudos

Resolved! MLFlow failed: You haven't configured the CLI yet

I'm getting an errorYou haven’t configured the CLI yet! Please configure by entering `/databricks/python_shell/scripts/db_ipykernel_launcher.py configure`My cluster is running Databricks Runtime Version 10.1I've also installed mlflow to the cluster l...

  • 10006 Views
  • 5 replies
  • 3 kudos
Latest Reply
HemantKumar
New Contributor II
  • 3 kudos

dbutils.library.restartPython()Add that after you run the pip install mlflow, it worked for me in a non-ML cluster

  • 3 kudos
4 More Replies
Oliver_Angelil
by Valued Contributor II
  • 4378 Views
  • 8 replies
  • 1 kudos

How to use the git CLI in databricks?

After making some changes in my feature branch, I have committed and pushed (to Azure Devops) some work (note I have not yet raised a PR or merge to any other branch). Many of the files I committed are data files and so I would like to reverse the co...

  • 4378 Views
  • 8 replies
  • 1 kudos
Latest Reply
Kayla
Contributor
  • 1 kudos

I'm also curious about this question - does anyone have an answer? Being able to use the full repertoire of git commands inside Databricks would be quite useful.

  • 1 kudos
7 More Replies
houstonamoeba
by New Contributor III
  • 1897 Views
  • 7 replies
  • 1 kudos

Resolved! examples on python sdk for install libraries

Hi Everyone,I'm planning to use databricks python cli "install_libraries"can some one pls post examples on function install_libraries https://github.com/databricks/databricks-cli/blob/main/databricks_cli/libraries/api.py

  • 1897 Views
  • 7 replies
  • 1 kudos
Latest Reply
Loop-Insist
New Contributor II
  • 1 kudos

Here you go using Python SDKfrom databricks.sdk import WorkspaceClientfrom databricks.sdk.service import computew = WorkspaceClient(host="yourhost", token="yourtoken")# Create an array of Library objects to be installedlibraries_to_install = [compute...

  • 1 kudos
6 More Replies
User16790091296
by Contributor II
  • 1581 Views
  • 1 replies
  • 0 kudos

How to create a databricks job with parameters via CLI?

I'm creating a new job in databricks using the databricks-cli:databricks jobs create --json-file ./deploy/databricks/config/job.config.jsonWith the following json:{ "name": "Job Name", "new_cluster": { "spark_version": "4.1.x-scala2.1...

  • 1581 Views
  • 1 replies
  • 0 kudos
Latest Reply
matthew_m
New Contributor III
  • 0 kudos

This is an old post but still relevant for future readers, so will answer how it is done. You need to add base_parameters flag in the notebook_task config, like the following.   "notebook_task": { "notebook_path": "...", "base_parameters": { ...

  • 0 kudos
mshettar
by New Contributor II
  • 1083 Views
  • 2 replies
  • 0 kudos

Databricks CLI's workspace export_dir command adds unnecessary edits despite not making any change in the workspace

databricks workspace export_dir / export command with overwrite option enabled adds non-existent changes in the target directory. 1. It introduces new line deletion and 2. add/deletion of MAGIC comments despite not making any meaningful changes in th...

Screenshot 2023-06-06 at 2.44.48 PM
  • 1083 Views
  • 2 replies
  • 0 kudos
Latest Reply
RyanHager
Contributor
  • 0 kudos

I am encountering this issue as well and it did not happen previously.  Additionally, you see this pattern if you are using repos internally and make a change to a notebook in another section.

  • 0 kudos
1 More Replies
Paul_Seattle
by New Contributor
  • 1085 Views
  • 1 replies
  • 0 kudos

A Quick Question on Running a job from CLI

Could anyone tell me what could be wrong with my command to submit a spark job with params( If I don’t have --spark-submit-params, it’s fine). Please see the attached snapshot.

image
  • 1085 Views
  • 1 replies
  • 0 kudos
Latest Reply
User16539034020
Contributor II
  • 0 kudos

yes, there is no need for spark-submit-params. databricks jobs run-now --job-id ***reference: https://docs.databricks.com/dev-tools/cli/jobs-cli.html

  • 0 kudos
Suresh_AWS
by New Contributor III
  • 2876 Views
  • 4 replies
  • 2 kudos

Resolved! Databricks fs ls does not work

I have configured the databricks CLI using token generated in Azure databricks configure --aad-tokenConfigure is success. When I used the command "databricks fs ls" I am running into an error as shown belowError: AttributeError: type object 'Retry' ...

  • 2876 Views
  • 4 replies
  • 2 kudos
Latest Reply
abos
New Contributor II
  • 2 kudos

It seems the Databricks cli needs and update. This issue is discussed on StackOverflow. For the time being, installing an older version of `urllib3` should solve the issue. The method was removed since 2.X, the latest 1.X version is 1.26.15. You can ...

  • 2 kudos
3 More Replies
ducng
by New Contributor II
  • 2401 Views
  • 1 replies
  • 0 kudos

VScode extension - certificate signature failure

Hi everyone,I'm trying to use the new Databricks extension (v0.3.10) for VS code (v1.77.3).I face this problem when connecting to our workspace:This problem persists when I tried to login through az CLI with our SSO, or through local config using PAT...

image.png
  • 2401 Views
  • 1 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

@Minh Duc Nguyen​ :It seems like the error you are facing is due to a failure in verifying the SSL certificate of your Databricks workspace. To resolve this, you need to add the custom CA certificate to your VS Code settings. Here's how you can do it...

  • 0 kudos
k9
by New Contributor II
  • 1495 Views
  • 4 replies
  • 1 kudos

Resolved! Databricks CLI v0.17.6 issue

I do have multiple groups created in my databricks account and I have databricks cli installed on my mac. Some of the cli functions return errors that i cannot find solution for. databricks groups listReturns:Error: b'{"error_code":"INTERNAL_ERROR","...

  • 1495 Views
  • 4 replies
  • 1 kudos
Latest Reply
karthik_p
Esteemed Contributor
  • 1 kudos

@kenan hasanov​ which version python you have installed on your machine please, you need to have 3-3.6 or 2-2.7.9 above , please try to go with latest one as you are only seeing issues with few functions. please raise issue in case if you are still f...

  • 1 kudos
3 More Replies
Praveen
by New Contributor II
  • 5753 Views
  • 10 replies
  • 1 kudos

Resolved! Pass Typesafe config file to the Spark Submit Job

Hello everyone ! I am trying to pass a Typesafe config file to the spark submit task and print the details in the config file. Code: import org.slf4j.{Logger, LoggerFactory} import com.typesafe.config.{Config, ConfigFactory} import org.apache.spa...

  • 5753 Views
  • 10 replies
  • 1 kudos
Latest Reply
source2sea
Contributor
  • 1 kudos

I've experenced similar issues; please help to answer how to get this working;I've tried using below to be either /dbfs/mnt/blah path or dbfs:/mnt/blah pathin either spark_submit_task or spark_jar_task (via cluster spark_conf for java optinos); no su...

  • 1 kudos
9 More Replies
Herkimer
by New Contributor II
  • 1593 Views
  • 3 replies
  • 1 kudos

dbsqlcli.main ERROR - getaddrinfo failed

I had to rebuild my laptop and am trying to re-install Databricks and Databricks-sql-cli so I can use the CLI to pull info from the databricks repository.When I try to run the command: dbsqlcli -e "select distinct tableName from databases_info.all_db...

  • 1593 Views
  • 3 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @John Zajic​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...

  • 1 kudos
2 More Replies
Arnold_Souza
by New Contributor III
  • 1581 Views
  • 4 replies
  • 2 kudos

SAT - Security Analysis Tool implementation error

I want to implement SAT in my workspace account. I was able to execute the terraform that enable the necessary infra to work on that. When I try to execute the workflow "SAT Initializer Notebook (one-time)" it fails with the error:AnalysisException: ...

1 2
  • 1581 Views
  • 4 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Arnold Souza​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...

  • 2 kudos
3 More Replies
juned
by New Contributor III
  • 6167 Views
  • 4 replies
  • 9 kudos

Resolved! Databricks CLI configure (using AAD-TOKEN) in the Databricks notebook `%sh` mode

Hello everyone, I am trying to setup Databricks CLI by referring to the Databricks CLI documentation. When I setup using the Personal Access Token, it works fine and I am able to access the workspace and fetch the results from the same workspace in D...

image
  • 6167 Views
  • 4 replies
  • 9 kudos
Latest Reply
Anonymous
Not applicable
  • 9 kudos

Hi @Juned Mala​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 9 kudos
3 More Replies
Jkb
by New Contributor II
  • 1297 Views
  • 2 replies
  • 2 kudos

Resolved! Workflow triggered by CLI shown "manually" triggered

We trigger different Worflows by ADF.These workflows will be shown triggered "manually".Is this behaviour intentional? At least for users, this is confusing.ADF-triggered Run: Databricks-Workflows: 

ADF_Monitor manually1 manually2
  • 1297 Views
  • 2 replies
  • 2 kudos
Latest Reply
Kaniz
Community Manager
  • 2 kudos

Hi @J. G.​, Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does. Your feedback will...

  • 2 kudos
1 More Replies
Labels