Databricks Community

Brahmareddy

Hey Austin Databricks Group,If you're passionate about data and AI, here's an event you won't want to miss: Data Day Texas + AI 2025, happening on Saturday, January 25. This event brings together experts and enthusiasts to share insights, trends, and...

Brahmareddy

Hey Austin Databricks Group,I just started exploring Databricks Assistant, and it’s genuinely amazing! It’s a tool that lets you ask questions about your data in plain language and get instant answers, whether it’s code snippets, SQL help, or guidanc...

Brahmareddy

Hi Everyone,Exciting news! There is an upcoming hands-on virtual lab with Databricks and AWS on Wednesday, November 13, 2024, from 9:00 AM to 12:00 PM PT. Don’t miss this opportunity to dive deep into the Databricks Data Intelligence Platform, which ...

Brahmareddy

Hello Austin Databricks Community,I wanted to take a moment to reflect on why Databricks is such a powerful platform compared to other solutions out there and why it’s worth exploring further together.When we look at Databricks versus its competitors...

Brahmareddy

Hello Austin Databricks Community,I wanted to take a moment to reflect on why Databricks is such a powerful platform compared to other solutions out there and why it’s worth exploring further together.When we look at Databricks versus its competitors...

Brahmareddy

Hi @RoelofvS,I have gone through your response and here is my suggestion below.Make sure to allow for a slight delay or checkpoint refresh after each schema change to ensure Auto Loader registers updates fully. Given that renaming 0 to 1 prompted sch...

Brahmareddy

Hi @calvinchan_iot, How are you doing today?As per my understanding, It sounds like the error may be due to environment instability when running the UDF after enabling Unity Catalog. The [UDF_ERROR.ENV_LOST] error often points to the UDF execution en...

Brahmareddy

Hi @RoelofvS,How are you doing today? As per my understanding, You just make sure cloudFiles.schemaEvolutionMode is set to addNewColumns to enable automatic schema updates for new columns. If schema versions aren't updating in the same location, try ...

Brahmareddy

You’re right @KUMAR__111—tracking costs for jobs on ALL_PURPOSE_COMPUTE clusters can be tricky since DBU usage isn’t directly tied to specific jobs. When multiple jobs run in parallel on the same cluster, it’s challenging to allocate costs accurately...

Brahmareddy

Hello @zed, May be you can try consider using partition pruning to limit the scope of the merge operation when calling write_table. Partitioning your feature table by daily or monthly increments will help reduce the data scanned during each merge, as...

Databricks Community

User Stats

User Activity

Mark Your Calendar: Data Day Texas + AI 2025 on January 25!

Try Databricks Assistant – Get Insights with Simple Questions!

Reserve Your Spot for the Databricks Data Intelligence Workshop on Nov 13!

Let’s Learn, Collaborate, and Grow Together!

Let’s Learn, Collaborate, and Grow Together

Re: Schema evolution in Autoloader not evolving beyond version 0

Re: SparkRuntimeException: [UDF_ERROR.ENV_LOST] the execution environment was lost during execution

Re: Schema evolution in Autoloader not evolving beyond version 0

Re: How to get cost per job which runs on ALL_PURPOSE_COMPUTE ??

Re: Consequences of Not Using write_table with Feature Engineering Client and INSERT OVERWRITE