Databricks Community

Brahmareddy

As someone who frequently works with large datasets in Power BI, I’ve had my fair share of frustrations with slow query performance, especially when pulling data from Databricks. After countless hours of tweaking and experimenting, I’ve finally found...

Brahmareddy

Hey Everyone, Hope you’re all doing great!A friend of mine recently went through a data engineering interview at Walmart and shared his experience with me. I thought it would be really useful to pass along what he encountered. The interview had some ...

Brahmareddy

PySpark UDFs offer a powerful mechanism for applying custom transformations to data within Spark DataFrames. While they provide flexibility and code reusability, they also come with performance overhead and debugging challenges. By understanding the ...

Brahmareddy

When I first started handling schema management in Databricks, I realized that a little bit of planning could save me a lot of headaches down the road. Here’s what I’ve learned and some simple tips that helped me manage schema changes effectively. On...

Brahmareddy

In the airline industry, it’s important to manage flights efficiently. Knowing why flights get canceled or rescheduled helps improve customer satisfaction and operational performance. In this article, I’ll show you how to use Databricks and PySpark t...

Brahmareddy

Hi @jay971, How are you doing today?To answer your question, To pass job parameters using Databricks CLI, create a JSON file that contains your job ID and parameters in the format expected by Databricks. Then, use the --json-file option in the CLI co...

Brahmareddy

Hi @weilin0323,Good morning.To avoid multiple results with masking, try using a hash function like sha2 in your function instead of partial masking. This ensures each value is unique, even when masked. When joining tables, apply the same hash to both...

Brahmareddy

Hi @weilin0323, How are you doing today?As per my understanding, When creating a downstream table from a masked table, you’ll need to reapply the mask function to the necessary columns in the new table if you want to maintain the same level of data p...

Brahmareddy

Hi @anirudh286, How are you doing today?As per my understanding, The HA option in Databricks picks instances from multiple zones right from the start, not just when one zone is unavailable. This ensures your cluster is always highly available but cou...

Brahmareddy

Hi @jg, How are you doing today?Try installing the wheel package with pip install wheel to resolve the missing bdist_wheel command. Make sure your environment has setuptools and wheel installed and up-to-date by running pip install --upgrade setuptoo...

Databricks Community

User Stats

User Activity

How I Tuned Databricks Query Performance from Power BI Desktop

Pyspark Interview Question: Inside a Walmart Data Engineering Interview

PySpark UDFs: Leveraging Custom Functions for Data Transformation

My Journey with Schema Management in Databricks

Understanding Flight Cancellations and Rescheduling in Airlines Using Databricks and PySpark

Re: Error: Cannot use legacy parameters because the job has job parameters configured.

Re: Data Masking Techniques and Issues with Creating Tables

Re: Data Masking Techniques and Issues with Creating Tables

Re: Info on Databricks AWS High Availability during zone selection

Re: build failed