topic Re: Need urgent help and guidance on information/details with reference links on below topics: in Data Engineering

Need urgent help and guidance on information/details with reference links on below topics:

shubham_007 — Mon, 27 Jan 2025 15:55:36 GMT

Dear experts,

I need urgent help and guidance on information/details with reference links on below topics:

Steps on Package Installation with Serverless in Databricks.
What are Delta Lake Connector with serverless ? How to run Delta Lake queries outside of Azure Databricks (ADB) ?

Re: Need urgent help and guidance on information/details with reference links on below topics:

brockb — Mon, 27 Jan 2025 17:43:29 GMT

Hi @shubham_007 ,

Thanks for the question.

You can install libraries with Serverless Compute using an "Environment". This also provides the option to configure an environment specification into a YAML file. More detail can be found here: https://docs.databricks.com/en/compute/serverless/dependencies.html#install-notebook-dependencies

Regarding the question on: "What are Delta Lake Connector with serverless?". There is not a constant/fixed Delta library version given that Serverless is considered a versionless product. Could you elaborate on the question here, are you finding that a feature is missing or working differently than expected?

Regarding the question: "How to run Delta Lake queries outside of Azure Databricks (ADB)", the best way to get started with OSS Delta Lake would be to check the docs here: https://docs.delta.io/latest/index.html and use that as a guide to get started. If you encounter issues, it would be best to post a separate question on your specific goals, what you've attempted, and what issues have been encountered.

Thank you.

Re: Need urgent help and guidance on information/details with reference links on below topics:

shubham007 — Tue, 28 Jan 2025 02:18:51 GMT

Thanks for information.

About second question ! I want to ask that What are delta lake connectors available for serverless compute to run delta lake queries outside Azure Databricks ? How to integrate it ? Any documentation available please share.

Re: Need urgent help and guidance on information/details with reference links on below topics:

shubham_007 — Tue, 28 Jan 2025 02:25:26 GMT

Thanks for information.

About second question ! I want to ask that What are delta lake connectors available for serverless compute to run delta lake queries outside Azure Databricks ? How to integrate it ? Any documentation available please share.

Re: Need urgent help and guidance on information/details with reference links on below topics:

brockb — Tue, 28 Jan 2025 03:39:34 GMT

Sure, I will presume you are using OSS Apache Spark outside of Azure Databricks. If so, you can use the delta-spark library at these maven coordinates: https://mvnrepository.com/artifact/io.delta/delta-spark_2.12.

Further information is at the Delta Quick Start in the OSS Delta docs: https://docs.delta.io/latest/quick-start.html#set-up-apache-spark-with-delta-lake

Re: Need urgent help and guidance on information/details with reference links on below topics:

shubham_007 — Tue, 28 Jan 2025 03:49:09 GMT

Thanks !!

What are steps to install package in serverless ? Any reference other than above given ?

Re: Need urgent help and guidance on information/details with reference links on below topics:

brockb — Tue, 28 Jan 2025 18:12:17 GMT

Were you able to review the documentation provided here: https://docs.databricks.com/en/compute/serverless/dependencies.html#install-notebook-dependencies?