topic Re: Command to install spark package from notebook cell in Data Engineering

Command to install spark package from notebook cell

JJ_LVS1 — Wed, 15 Feb 2023 21:47:04 GMT

Hi All,

I need to install a spark-xml package from a notebook cell (hoping it will work on a DLT cluster).

Maven Package: com.databricks:spark-xml_2.12:0.16.0

Can anyone help me with the command to install from the notebook cell?

Fairly new to all this and anything I've found searching hasn't worked. I've installed it on the interactive cluster from the library install area with no issues but can't get the command from a cell to work.

TIA

Re: Command to install spark package from notebook cell

Ajay-Pandey — Thu, 16 Feb 2023 09:34:08 GMT

Hi @Jason Johnson ,

Please refer below link that might help you-

Libraries CLI | Databricks on AWS

Re: Command to install spark package from notebook cell

JJ_LVS1 — Thu, 16 Feb 2023 15:05:20 GMT

Thanks for the response. Pip install doesn't work for this library.

Re: Command to install spark package from notebook cell

JJ_LVS1 — Thu, 16 Feb 2023 15:07:31 GMT

Because I need this on a DLT cluster, there is no way to interface with it using CLI since it's the cluster created at runtime. I think I can run CLI commands within a notebook cell so I'll try to figure out how to do that. Please reply if you know how.

Re: Command to install spark package from notebook cell

Anonymous — Fri, 17 Feb 2023 05:45:13 GMT

Hi @Jason Johnson

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!

Re: Command to install spark package from notebook cell

JJ_LVS1 — Fri, 17 Feb 2023 15:02:19 GMT

Hi, Have not solved his issue yet. This maven spark-xml library does not install via pip so exploring options to execute the install via databricks cli in a notebook cell (which I don't know how to do so still trying to make it work).

Re: Command to install spark package from notebook cell

JJ_LVS1 — Thu, 23 Feb 2023 20:17:13 GMT

I would call this one closed. Don't see a way to make this work given how the spark-xml library functions.

Re: Command to install spark package from notebook cell

Anonymous — Sat, 11 Mar 2023 03:30:57 GMT

Hi @Jason Johnson

I'm sorry you could not find a solution to your problem in the answers provided.

Our community strives to provide helpful and accurate information, but sometimes an immediate solution may only be available for some issues.

I suggest providing more information about your problem, such as specific error messages, error logs or details about the steps you have taken. This can help our community members better understand the issue and provide more targeted solutions.

Alternatively, you can consider contacting the support team for your product or service. They may be able to provide additional assistance or escalate the issue to the appropriate section for further investigation.

Thank you for your patience and understanding, and please let us know if there is anything else we can do to assist you.

Re: Command to install spark package from notebook cell

Debayan — Thu, 16 Feb 2023 14:29:32 GMT

Hi, you can refer to the notebook scoped libraries: https://docs.databricks.com/libraries/notebooks-python-libraries.html , please let us know if this helps.