Databricks Community

FerArribas · 06-19-2024

FerArribas · 02-02-2024

Does anyone know what the first digits of the jobgroup that are shown in the spark ui mean when using all purpose clusters to launch multiple jobs?Right now the pattern is something like: [id_random]_job_[jod_id]_run-[run_id]_action_[action].

FerArribas · 01-02-2023

Hi, Thoroughly investigating the best security practices for accessing the Databricks WEB UI. I have doubts about the difference between protecting the WEB UI with (1) IP Access list (https://learn.microsoft.com/en-us/azure/databricks/security/networ...

FerArribas · 11-28-2022

We need to be able to import a custom certificate (https://learn.microsoft.com/en-us/azure/databricks/kb/python/import-custom-ca-cert) in the same way as in the "data engineering" module but in the Databricks SQL module

FerArribas · 10-13-2022

I have created a workspace with private endpoint in Azure following this guide:https://learn.microsoft.com/en-us/azure/databricks/administration-guide/cloud-configurations/azure/private-linkOnce I have created the private link of type browser_authent...

FerArribas · 04-04-2023

Hi @Helen Morgen,Is it possible that you have enabled unity catalog after creating the cluster? You can try to create a new cluster as single user and runtime 12.x for example.Regards,

FerArribas · 04-04-2023

Hi @Helen Morgen,Is it possible that you have enabled unity catalog after creating the cluster? You can try to create a new cluster as single user and runtime 12.x for example.Regards,

FerArribas · 01-03-2023

I'm not sure about that. When you call the function to_excel all the data is loaded into the driver (as if you were doing a collect). So, the writing is not distributed and you can have memory and performance problems as I mentioned. Reference: https...

FerArribas · 01-03-2023

Have you tried writing to the local file system (for example, in the path /databricks/...)Anyway, i recommend you to tyr writing with Spark (without dataframes). Pandas without additional libraries, doesn't distribute and with high volumes you will...

FerArribas · 01-02-2023

Hi @Anastasiia Polianska,I agree, it looks like a concurrency issue. Very possibly this concurrency problem will be caused by an erroneous ETAG in the HTTP call to the Azure Storage API (https://azure.microsoft.com/de-de/blog/managing-concurrency-in...

Databricks Community

User Stats

User Activity

In Azure Databricks, for when to use Kubernetes instead of Virtual Machines as compute backend?

Custom JobGroup in Spark UI for cluster with multiple executions

Azure Databricks - Difference between protecting the WEB UI with IP Access list or disabling public access?

How to import a custom CA certificate into the Databricks SQL module?

Redirect error in access to web app in Azure Databricks with private front endpoint

Re: [UC_NOT_ENABLED] Unity Catalog is not enabled on this cluster

Re: [UC_NOT_ENABLED] Unity Catalog is not enabled on this cluster

Re: How can we write a pandas dataframe into azure adls as excel file, when trying to write it is showing error as protocol not known 'abfss' like that.

Re: How can we write a pandas dataframe into azure adls as excel file, when trying to write it is showing error as protocol not known 'abfss' like that.

Re: Read/Write concurrency issue