topic Re: How do we add a certificate file in Databricks for sparksubmit type of job? in Data Engineering

How do we add a certificate file in Databricks for sparksubmit type of job?

geetha_venkates — Wed, 08 Dec 2021 16:26:47 GMT

How do we add a certificate file in Databricks for sparksubmit type of job?

Re: How do we add a certificate file in Databricks for sparksubmit type of job?

Anonymous — Wed, 08 Dec 2021 23:44:08 GMT

Hello, @geetha Venkatesan! Welcome and thank you for asking. Let's give the community a bit longer to answer. We'll circle back if we need to.

Re: How do we add a certificate file in Databricks for sparksubmit type of job?

BilalAslamDbrx — Mon, 13 Dec 2021 00:58:49 GMT

@geetha Venkatesan can you explain what sort of certificate you're trying to use? spark-submit works over the Jobs API, which does not require submitting any cert.

Re: How do we add a certificate file in Databricks for sparksubmit type of job?

franco_patano — Mon, 13 Dec 2021 19:08:39 GMT

Here are directions on how to add a custom CA Cert. Does this help?

https://kb.databricks.com/python/import-custom-ca-cert.html

Re: How do we add a certificate file in Databricks for sparksubmit type of job?

geetha_venkates — Tue, 14 Dec 2021 06:55:22 GMT

ECS is hosted on our private cloud and it uses internal certificate owned by us. So from databricks when we need to connect to ECS, we need to import our internal certificate in databricks, so that connection happens between databricks and ECS .

Re: How do we add a certificate file in Databricks for sparksubmit type of job?

geetha_venkates — Tue, 14 Dec 2021 07:02:49 GMT

As per this document, we need to use DButils and we can use this only on notebook. But ours is spark scala application jar that runs in spark submit mode. So we want to know how we can import the certificate and make it use at the startup of the cluster.

Re: How do we add a certificate file in Databricks for sparksubmit type of job?

DBLearner74 — Thu, 07 Dec 2023 20:02:34 GMT

Since DBFS based files getting deprecated, is there a newer method to this?

Re: How do we add a certificate file in Databricks for sparksubmit type of job?

nicozambelli — Wed, 13 Dec 2023 16:22:45 GMT

I have the same problem... when i worked with the hive_metastore in past, i was able tu use file system and also use API certs.

Now i'm using the unity catalog and i can't upload a certificate, can somebody help me?