cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
goldentown
by • New Contributor III
  • 3195 Views
  • 1 replies
  • 2 kudos

Resolved! The Jupiter note-book doesn't update imports after updating the .py file

Please help. Here's an example:I have one .py file and one .ipynb, and the .py file contains the test function, but after adding the new function test1, it doesn't appear in .ipynb. Even after re-running the .py file and reimporting it in .ipynb. How...

Data Engineering
Reply
  • 3195 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
goldentown
New Contributor III
  • 2 kudos

%load_ext autoreload%autoreload 2

  • 2 kudos
Craig_
by • New Contributor III
  • 3282 Views
  • 4 replies
  • 0 kudos

Resolved! Caveats when importing functions from REPO stored .py files

The ability to import .py files into notebooks looked like a clean and easy way to reuse code and to ensure all notebooks are using the same version of code. However, two items remain unclear after scouring documentation and forums.Are these the rig...

Data Engineering
Reply
  • 3282 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 0 kudos

"does not have access to the spark session by default" yes, that is correct you need to pass the reference to the spark variable inside Class or Function, something like when you call from notebook function_from_file(spark=spark)displayHTML() is desi...

  • 0 kudos
3 More Replies
KrishZ
by • Contributor
  • 4535 Views
  • 4 replies
  • 1 kudos

How to print the path of a .py file or a notebook?

I have stored a test.py in the dbfs at the below location "/dbfs/FileStore/shared_uploads/krishna@company.com/Project_Folder/test.py"I have a print statement in test.py which says the belowprint( os.getcwd() )and it prints the below'/databricks/drive...

Data Engineering
Reply
  • 4535 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 1 kudos

Hey @Krishna Zanwar​  Please use the below code this will work and as you want the specific location you can create a custom code and format the path using a python formatter , it will give you desired result .

  • 1 kudos
3 More Replies
KVNARK
by • Honored Contributor II
  • 1091 Views
  • 1 replies
  • 5 kudos

Resolved! Trigger another .py file by uisng 2 .py files.

Hi,I have 3 .py files - a.py, b.py & c.py files. By joining a.py & b.py, based on the output that I get need to trigger the c.py file.

Data Engineering
Reply
  • 1091 Views
  • 1 replies
  • 5 kudos
View Replies
Latest Reply
Ajay-Pandey
Esteemed Contributor III
  • 5 kudos

Hi @KVNARK .​ refer below link this will help in thisLink

  • 5 kudos
him
by • New Contributor III
  • 1187 Views
  • 1 replies
  • 3 kudos

There is .py file in my local vs code ,how can i upload that file on databrick cluster using rest api and run it !!

Data Engineering
Reply
  • 1187 Views
  • 1 replies
  • 3 kudos
View Replies
Latest Reply
Debayan
Databricks Employee
  • 3 kudos

You can try to refer the example below: https://docs.databricks.com/dev-tools/api/latest/examples.html#upload-a-big-file-into-dbfs

  • 3 kudos
data_engineer_0
by • New Contributor II
  • 14043 Views
  • 3 replies
  • 2 kudos

How to run the .py file in databricks cluster

Hi team,I wants to run the below command in databricks and also need to capture the error and success message.Please help me out here,Thanks in advanceEx: python3 /mnt/users/code/x.py --arguments

Data Engineering
Reply
  • 14043 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
User16764241763
Honored Contributor
  • 2 kudos

Hello @Piper Wilson​ Would this task not help?https://docs.databricks.com/dev-tools/api/latest/examples.html#jobs-api-examples

  • 2 kudos
2 More Replies
jimnaik
by • New Contributor III
  • 19500 Views
  • 2 replies
  • 1 kudos

Resolved! How to execute .sh and .py file in the workspace?

I want to execute shell script which is running .py file. May I know how to run .sh file and .py files in Databricks workspace?

Data Engineering
Reply
  • 19500 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
jimnaik
New Contributor III
  • 1 kudos

I tried executing like this and it worked: %sh /dbfs/***/***/***.sh

  • 1 kudos
1 More Replies
Labels