Databricks Community

joao_augusto · 07-11-2025

Hi, everyone!Does anyone know the reason for this problem? It says that I do not need to do anything, but if I don't restart the warehouse manually, it will not start.Is there a way to fix it? Or at least, create monitoring for it? We have some jobs ...

joao_augusto · 04-04-2025

Hi guys!I'm facing a problem, and I have no idea where it came from. My process is not appending all the topic data into my bronze table. I checked the topic, and the data is there.For example, I have some rows that are still in my Kafka topic but do...

joao_augusto · 07-11-2025

That's really cool! You have a really mature environment right there! Hope to get this level!!

joao_augusto · 04-04-2025

EDIT: I checked that the data that is inserted into the kafka topic while my stream is running is neither process by the current stream nor the next job run

joao_augusto · 03-19-2025

Hmmmmmmmmm that makes sense!!!!! I will study this possibility! You are amazing and we are best friends from now.Thank you for this discussion Brahma.Regards

joao_augusto · 03-19-2025

That's interesting, but I think I'm missing something.To define the schema as json, you would probably read it from a bucket (example), let the spark/autoloader define the schema and save it in a file for validation. In this case, you are also relyin...

joao_augusto · 03-18-2025

Hi Brah! Thanks for your post Currently, I'm trying to get away from schema inference. How do you store the DDL, schema and data type of each table? I guess that it is not write manually for each one.Do you store it in yaml, json or SQL tables?It wou...

Databricks Community

User Stats

User Activity

The warehouse fails to start

Data loss with spark streaming and kafka

Re: My Journey with Schema Management in Databricks

Re: Data loss with spark streaming and kafka

Re: My Journey with Schema Management in Databricks

Re: My Journey with Schema Management in Databricks

Re: My Journey with Schema Management in Databricks