cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

sandeep91
by New Contributor III
  • 4906 Views
  • 5 replies
  • 2 kudos

Resolved! Databricks Job: Package Name and EntryPoint parameters for the Python Wheel file

I have created Python wheel file with simple file structure and uploaded into cluster library and was able to run the packages in Notebook but, when I am trying to create a Job using python wheel and provide the package name and run the task it fails...

image
  • 4906 Views
  • 5 replies
  • 2 kudos
Latest Reply
AndréSalvati
New Contributor III
  • 2 kudos

There you can see a complete template project with (the new!!!) Databricks Asset Bundles tool and a python wheel task. Please, follow the instructions for deployment.https://github.com/andre-salvati/databricks-template

  • 2 kudos
4 More Replies
Rajaniesh
by New Contributor III
  • 1383 Views
  • 3 replies
  • 1 kudos

URGENT HELP NEEDED: Python functions deployed in the cluster throwing the error

Hi,I have created a python wheel with the following code. And the package name is rule_engine"""The entry point of the Python Wheel"""import sysfrom pyspark.sql.functions import expr, coldef get_rules(tag): """  loads data quality rules from a table ...

  • 1383 Views
  • 3 replies
  • 1 kudos
Latest Reply
jose_gonzalez
Moderator
  • 1 kudos

You can find more details and examples here https://docs.databricks.com/en/workflows/jobs/how-to/use-python-wheels-in-workflows.html#use-a-python-wheel-in-a-databricks-job

  • 1 kudos
2 More Replies
lcalca95
by New Contributor II
  • 952 Views
  • 0 replies
  • 0 kudos

Azure Databricks job and exception handling

Hi,I'm working on Azure Databricks and I created two jobs, one based on a python wheel and the other based on a notebook, with the same code. The code get data from Azure blob storage, process data with pyspark and send data to EventHub. The whole co...

  • 952 Views
  • 0 replies
  • 0 kudos
vk217
by Contributor
  • 8290 Views
  • 5 replies
  • 17 kudos

Resolved! python wheel cannot be installed as library.

When I try to install the python whl library, I get the below error. However I can install it as a jar and it works fine. One difference is that I am creating my own cluster by cloning an existing cluster and copying the whl to a folder called testin...

image
  • 8290 Views
  • 5 replies
  • 17 kudos
Latest Reply
vk217
Contributor
  • 17 kudos

The issue was that the package was renamed after it was installed to the cluster and hence it was not recognized.

  • 17 kudos
4 More Replies
Sudd
by New Contributor II
  • 1237 Views
  • 1 replies
  • 1 kudos

Permanent UDF in Databricks using Python Wheel

I have a simple Python Program, which takes a Integer as a input and gives a string as a output.I have created the wheel file for this Python code.Then I have uploaded it in the Wheel section of Databricks cluster.After this I want to create a perma...

  • 1237 Views
  • 1 replies
  • 1 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 1 kudos

First, you will need to onboard the unity catalog and sign for Python UDF preview https://www.databricks.com/blog/2022/07/22/power-to-the-sql-people-introducing-python-udfs-in-databricks-sql.htmlBut I doubt it will be possible to use a wheel (but who...

  • 1 kudos
Mr__E
by Contributor II
  • 2832 Views
  • 5 replies
  • 5 kudos

Resolved! Using shared python wheels for job compute clusters

We have a GitHub workflow that generates a python wheel and uploads to a shared S3 available to our Databricks workspaces. When I install the Python wheel to a normal compute cluster using the path approach, it correctly installs the Python wheel and...

  • 2832 Views
  • 5 replies
  • 5 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 5 kudos

You can mount S3 as a DBFS folder then set that library in "cluster" -> "libraries" tab -> "install new" -> "DBFS" 

  • 5 kudos
4 More Replies
Labels