- 25544 Views
- 5 replies
- 11 kudos
Resolved! How to import a function to another notebook using Repos without %run?
Hi all,I was reading the Repos documentation: https://docs.databricks.com/repos.html#migrate-from-run-commandsIt is explained that, one advantage of Repos is no longer necessary to use %run magic command to make funcions available in one notebook to ...
- 25544 Views
- 5 replies
- 11 kudos
- 11 kudos
Due to new functionalies in Runtime 16.0 regarding autoload i came across this autoload. Performaed a practical test. It works. However had some problems at first.As in solution the key was that definitions are places in a file.py not a notebook.
- 11 kudos
- 23987 Views
- 3 replies
- 4 kudos
Resolved! How to get the Job ID and Run ID and save into a database
We are having Databricks Job running with main class and JAR file in it. Our JAR file code base is in Scala. Now, when our job starts running, we need to log Job ID and Run ID into the database for future purpose. How can we achieve this?
- 23987 Views
- 3 replies
- 4 kudos
- 4 kudos
That article is for members only, can we also specify here how to do it (for those that are not medium members?). Thanks!
- 4 kudos
- 4343 Views
- 4 replies
- 0 kudos
Resolved! Programmatically updating the “run_as_user_name” parameter for jobs
I am trying to write a process that will programmatically update the “run_as_user_name” parameter for all jobs in an Azure Databricks workspace, using powershell to interact with the Jobs API. I have been trying to do this with a test job without suc...
- 4343 Views
- 4 replies
- 0 kudos
- 0 kudos
Solution you've submitted is a solution for different topic (permission to run job, the job still runs as the user in run_as_user_name field). Here is an example of changing "run_as_user_name"Docs:https://docs.databricks.com/api/azure/workspace/job...
- 0 kudos
- 2328 Views
- 4 replies
- 3 kudos
ASP1.2 Error create database in Spark Programming with Databricks training
I'm on Demo and Lab in Dataframes section. I've imported the dbc into my company cluster and has run "%run ./Includes/Classroom-Setup" successfully. When i run the 1st sql command %sql CREATE TABLE IF NOT EXISTS events USING parquet OPTIONS (path "/m...
- 2328 Views
- 4 replies
- 3 kudos
- 3 kudos
I had the same issue and solved it like this:In the includes folder, there is a reset notebook, run the first command, this unmounts all mounted databases.Go back to the ASP 1.2 notebook and run the %run ./Includes/Classroom-Setup codeblock.Then run ...
- 3 kudos
- 6981 Views
- 6 replies
- 3 kudos
Resolved! Is there a way to run a single or selected lines in a notebook?
Assume we have a given cellprint('A') print('B') print('C')I want to run only the below line.print('B')Obviously, I can seperate the cell into three and run the one I want, but this is timely. This is a feature I use so often (e.g. in pycharm) and wo...
- 6981 Views
- 6 replies
- 3 kudos
- 3 kudos
@Volkan_Gumuskay This is also available as an option in the notebook run options.
- 3 kudos
- 1285 Views
- 1 replies
- 3 kudos
How to execute requests later for a number of times that exceeds the Maximum concurrent runs?
I am trying to start the same Jobs multiple times using the python sdk's "run_now" command.If the number of requests exceeds the Maximum concurrent runs, the status of the run will be Skipped and the run will not be executed.Is there any way to queue...
- 1285 Views
- 1 replies
- 3 kudos
- 3 kudos
Hi, We do have a private preview feature which will be enabled shortly for queueing. Please tag me (@Debayan Mukherjee ) with your next update so that I will get notified.
- 3 kudos
- 7594 Views
- 1 replies
- 0 kudos
Run result unavailable: run failed with error message Notebook not found:
I'm trying to create a workflow job that fetches the notebook from a remote git repository (Bitbucket cloud)I tried everything in the Path field and nothing is working. Note that the bitbucket repo is connected to databricks already and no issues che...
- 7594 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @harraz (Customer) , Could you please confirm if files in repos has been enabled? https://docs.databricks.com/files/workspace.html#configure-support-for-files-in-repos.You can use the command %sh pwd in a notebook inside a repo to check if Files ...
- 0 kudos
- 4472 Views
- 8 replies
- 0 kudos
R packages not getting installed on cluster when creating cluster from dockerfile
I'm trying to use dockerfile to create a cluster which has Robyn (https://facebookexperimental.github.io/Robyn/) and other R libraries installed. But it is failing to install the R libraries to the cluster. When I run the container in interactive mod...
- 4472 Views
- 8 replies
- 0 kudos
- 0 kudos
What there has been no answer here! @Debayan Mukherjee @Vartika Nain So I am running into this same problem as the idea of having to wait 45 minutes for libraries to install is absolutely wild as well as I have done everything outside of working w...
- 0 kudos
- 4557 Views
- 2 replies
- 3 kudos
Resolved! Can you use %run or dbutils.notebook.run in a Delta Live Table pipeline?
Hi there, Can you use a %run or dbutils.notebook.run() in a Delta Live Table (DLT) pipeline?When I try, I get the following error: "IllegalArgumentException: requirement failed: To enable notebook workflows, please upgrade your Databricks subscriptio...
- 4557 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi all.@Kaniz Fatma thanks for your answer. I am on the premium pricing tier in Azure.After digging around the logs it would seem that you cannot run magic commands in a Delta Live Table pipeline. Therefore, you cannot use %run in a DLT pipeline - w...
- 3 kudos
- 2343 Views
- 3 replies
- 0 kudos
Resolved! when trying to use %run to a notebook, I'm getting an error.
this error is occurring randomly. sometimes it resolves on its own. not sure what is the cause of the error. The notebook I'm sourcing runs flawlessly. I'm only trying to import dates from this notebook
- 2343 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @viswa p Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we ca...
- 0 kudos
- 4574 Views
- 1 replies
- 4 kudos
Automating the re run of job (with several Tasks) // automate the notification of a failed specific tasks after re trying // Error handling on azure data factory pipeline with DataBricks notebook
Hi DataBricks Experts:I'm using Databricks on Azure.... I'd like to understand the following:1) if there is way of automating the re run some specific failed tasks from a job (with several Tasks), for example if I have 4 tasks, and the task 1 and 2 h...
- 4574 Views
- 1 replies
- 4 kudos
- 4 kudos
You can use "retries".In Workflow, select your job, the task, and in the options below, configure retries.If so, you can also see more options at:https://learn.microsoft.com/pt-br/azure/databricks/dev-tools/api/2.0/jobs?source=recommendations
- 4 kudos
- 4021 Views
- 5 replies
- 3 kudos
Give customized notebook run id/name in console using dbutils.notebook.run()
I am running below code in Azure databricks:Now the running status shows as highlighted in yellow. I want to give a custom name to this running status instead of Notebook job #<job id>. This will help me identify which status is for which notebook so...
- 4021 Views
- 5 replies
- 3 kudos
- 3 kudos
Hi @Gaurav Tanwar I'm sorry you could not find a solution to your problem in the answers provided.Our community strives to provide helpful and accurate information, but sometimes an immediate solution may only be available for some issues.I suggest...
- 3 kudos
- 1517 Views
- 2 replies
- 1 kudos
Running large volume of SQL queries in Python notebooks. How to minimise overheads/maintenance.
I have around 200 SQL queries id like to run in databricks python notebooks. Id like to avoid creating an ETL process for each of the 200 SQL processes.Any suggestions on how to run the queries in a way that it loops through them so i have minimum am...
- 1517 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Chris French Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...
- 1 kudos
- 1863 Views
- 2 replies
- 0 kudos
Delay in retrieving task variables from a notebook utilizing %run
I have a databricks workflow where the first task is to set up task parameters for other notebooks to use later in the process. Since these are variables that are used in all of my notebooks, I have opted to assign them in a shared notebook and call ...
- 1863 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi, Please refer: https://docs.databricks.com/notebooks/notebook-workflows.htmlCould you please provide the error received?
- 0 kudos
- 3510 Views
- 9 replies
- 11 kudos
Databricks start support to run selected text in a cell this will help us a lot during debugging of the code.In windows just select the line of code w...
Databricks start support to run selected text in a cell this will help us a lot during debugging of the code.In windows just select the line of code which you want to execute and press Ctrl+Shift+Enter
- 3510 Views
- 9 replies
- 11 kudos
