cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
ImAbhishekTomar
by • New Contributor III
  • 8782 Views
  • 7 replies
  • 4 kudos

kafkashaded.org.apache.kafka.common.errors.TimeoutException: topic-downstream-data-nonprod not present in metadata after 60000 ms.

I am facing an error when trying to write data on Kafka using spark stream.#Extract source_stream_df= (spark.readStream .format("cosmos.oltp.changeFeed") .option("spark.cosmos.container", PARM_CONTAINER_NAME) .option("spark.cosmos.read.inferSchema.en...

Data Engineering
Reply
  • 8782 Views
  • 7 replies
  • 4 kudos
View Replies
Latest Reply
devmehta
New Contributor III
  • 4 kudos

What event hub namespace you were using?I had same problem and resolved by changing pricing plan from basic to standard as Kafka apps is not supporting in basic planLet me know if you had anything else. Thanks

  • 4 kudos
6 More Replies
DwrCymru
by • New Contributor III
  • 3036 Views
  • 4 replies
  • 4 kudos

Data bricks issue writing to Azure synapse

Process is to pickup data from an external table in Azure Synapse and combine with other tables and write to another synapse tables. Data for external table is in Azure storage. It was fine for months ,all of sudden last week it errored out with err...

Data Engineering
Reply
  • 3036 Views
  • 4 replies
  • 4 kudos
View Replies
Latest Reply
UmaMahesh1
Honored Contributor III
  • 4 kudos

Also do check out this blog from this stack overflow question.https://stackoverflow.com/questions/66747544/databricks-write-back-to-azure-synapse-errorhttps://chinnychukwudozie.com/2020/11/13/write-data-from-azure-databricks-to-azure-synapse-analytic...

  • 4 kudos
3 More Replies
pine
by • New Contributor III
  • 3558 Views
  • 5 replies
  • 4 kudos

Resolved! Databricks fails writing after writing ~30 files

Good day, Copy of https://stackoverflow.com/questions/69974301/looping-through-files-in-databricks-failsI got 100 files of csv data on adls-gen1 store. I want to do some processing to them and save results to same drive, different directory. def look...

Data Engineering
Reply
  • 3558 Views
  • 5 replies
  • 4 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 4 kudos

was actually anything created by script in directory <my_output_dir>?The best would be to permanently mount ADSL storage and use azure app for that.In Azure please go to App registrations - register app with name for example "databricks_mount" . Ad...

  • 4 kudos
4 More Replies
Labels