cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

ae20cg
by New Contributor III
  • 7057 Views
  • 17 replies
  • 12 kudos

How to instantiate Databricks spark context in a python script?

I want to run a block of code in a script and not in a notebook on databricks, however I cannot properly instantiate the spark context without some error.I have tried ` SparkContext.getOrCreate()`, but this does not work.Is there a simple way to do t...

  • 7057 Views
  • 17 replies
  • 12 kudos
Latest Reply
Kaizen
Contributor III
  • 12 kudos

I came across a similar issue. Please detail how you are executing the python script. Are you calling it from the web terminal? or from a notebook?Note: If you are calling it from the web terminal - your spark session wont be passed. You could create...

  • 12 kudos
16 More Replies
Divya_Bhadauria
by New Contributor II
  • 1539 Views
  • 2 replies
  • 2 kudos

Running databricks job with different parameter automatically

I have a python script running as databricks job. Is there a way I can run this job with different set of parameters automatically or programmatically without using run with different parameter option available in UI ?

  • 1539 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Divya Bhadauria​ We haven't heard from you since the last response from @Lakshay Goel​ ​, and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to ...

  • 2 kudos
1 More Replies
drewtoby
by New Contributor II
  • 2872 Views
  • 2 replies
  • 1 kudos

Resolved! How to Pull Cached SQL Table into Python Dictionary?

Hello,I have been working on this issue as a proof of concept - it would be extremely helpful to iterate through tables via loops in a few scenarios. I have a simple three column dimension that I added to a cached table.cache lazy table hedis_cache s...

Method 1 Method 2
  • 2872 Views
  • 2 replies
  • 1 kudos
Latest Reply
drewtoby
New Contributor II
  • 1 kudos

Got it to work, thank you for the tip! I needed to convert the dataframe over to a pandas dataframehttps://www.geeksforgeeks.org/convert-pyspark-dataframe-to-dictionary-in-python/

  • 1 kudos
1 More Replies
Divya_Bhadauria
by New Contributor II
  • 3222 Views
  • 3 replies
  • 2 kudos

Unable to run python script from git repo in Databricks job

I'm getting cannot read python file on running this job which is configured to run a python script from git repo. Run result unavailable: run failed with error message Cannot read the python file /Repos/.internal/7c39d645692_commits/ff669d089cd8f93e9...

  • 3222 Views
  • 3 replies
  • 2 kudos
Latest Reply
Divya_Bhadauria
New Contributor II
  • 2 kudos

Hi Vidula,Yes, the above solution worked out for me. Tried debugging using all of the above steps and it turned out the path I was using in the job config was incorrect.

  • 2 kudos
2 More Replies
Lakshmi_J
by New Contributor II
  • 2695 Views
  • 2 replies
  • 3 kudos

Unable to read data from delta table using a python script after the table properties change.

Renamed the Column to include () in a delta table and set the table properties to the below ​ALTER TABLE test_table SET TBLPROPERTIES (  'delta.minReaderVersion' = '2',  'delta.minWriterVersion' = '5',  'delta.columnMapping.mode' = 'name' ) However w...

Error
  • 2695 Views
  • 2 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Lakshmi Jayaraman​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best ans...

  • 3 kudos
1 More Replies
dslin
by New Contributor III
  • 1614 Views
  • 3 replies
  • 2 kudos

How to deploy a python script with dependencies by dbx?

Hi,I'm quite new here. I'm trying to perform a deployment of python file with dbx command. The file contains libraries to be installed. How may I deploy the file (together with its dependencies) to databricks?Here are the commands I currently run:`db...

  • 1614 Views
  • 3 replies
  • 2 kudos
Latest Reply
Vidula
Honored Contributor
  • 2 kudos

Hi @Di Lin​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 2 kudos
2 More Replies
Manoj
by Contributor II
  • 5253 Views
  • 15 replies
  • 7 kudos

Resolved! Did any one in the community create Permanent Functions using using python script ? I tried but i am getting the below error, Please advise

Hi Team, When i am trying to register a permanant function i am getting the below error.%sqlCREATE FUNCTION simple_udf AS 'SimpleUdf'  USING JAR '/tmp/SimpleUdf.jar';%sqlselect simple_udf(2)Error Details : com.databricks.backend.common.rpc.Databricks...

  • 5253 Views
  • 15 replies
  • 7 kudos
Latest Reply
Manoj
Contributor II
  • 7 kudos

hi @Werner Stinckens​  @Jose Gonzalez​  @Hubert Dudek​ @Kaniz Fatma​ ​Thanks for all the help, Appreciate it. I was able to create permanent functions and use eclipse to create the runnable jar. However, Does anyone have any idea on how to deploy t...

  • 7 kudos
14 More Replies
test_data
by New Contributor III
  • 3823 Views
  • 2 replies
  • 1 kudos
  • 3823 Views
  • 2 replies
  • 1 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 1 kudos

Please use Repos and in admin settings please enable "Files in Repo" than you will be able to import class in notebook:from repo_folder.subfolders.file import your_class

  • 1 kudos
1 More Replies
Labels