Databricks Community

Avinash_Narala

Hi,I want to convert my python udf to pyspark udf, is there any guide/article on suggesting the best practices and avoid miscalculations if any

Avinash_Narala

Are your SQL queries running slow or consuming excessive resources? As data grows in size and complexity, writing optimized SQL queries becomes more crucial than ever. Knowing the right techniques can make all the difference.In my latest blog, I expl...

Avinash_Narala

Hi,Recently, I had some logic to collect the dataframe and process row by row. I am using 128GB driver node but it is taking significantly more time (like 2 hours for just 700 rows of data).May I know which type of cluster should I use and the driver...

Avinash_Narala · 11-13-2024

I'm exploring Databricks' fully serverless SaaS option, as shown in the attached image, which promises quick setup and $400 in initial credits. I'm curious about the pros and cons of using this fully serverless setup.Specifically, would this option b...

Avinash_Narala · 08-22-2024

We have installed Overwatch in Azure Databricks environment.As mentioned in documentation, we did create storage account for cluster logs and mounted it in databricks. Similarly, enabled system tables for audit logs and created delta table for stori...

Avinash_Narala

You can compare your databricks data with on-prem sql server data in two ways:Firstly, you have to make connection between sql-server and databricks using volumes. Using volumes we can mount sql server data into databricks unity catalog1.we can comp...

Avinash_Narala

Hi @Miguel_Suarez ,So when I have multiple jobs to run, is it suggested to use multiple clusters for each job instead of one standalone high concurrency cluster? Does it improve performance?

Avinash_Narala

you can use volumes and mount the abfss location to unity catalog and can access the files present in azure in databricks. Regards,Avinash N

Avinash_Narala

Maybe, but instead of focusing on connecting to local VM's I think Databricks will focus on broader audience, but will get to know once they announces more supported sources.If my answer helps you, please accept it as solution. Regards,Avinash N

Avinash_Narala

As for Lakehouse Federation, only some sources are supported as of now. In that, connecting to HiveMetastore data which is in local VM is not supported, so you can migrate that data to ADLS and mount that as external location in unity catalog and que...

Databricks Community

User Stats

User Activity

Python udf to pyspark udf conversion

Best practices for writing efficient SQL queries in Databricks

which type of cluster to use

Fully serverless databricks SaaS

Few Tables not getting creating in Overwatch installation in Azure Databricks

Re: Export data from databricks to prem

Re: Drive memory utilization cleanup

Re: Working with Pandas through Abfss

Re: Lakehouse Federation

Re: Lakehouse Federation