Databricks Community

Mathias_Peters

Hi, I am implementing a Spark Job in Kotlin (unfortunately a must-have) which reads from and writes to MongoDB. The reason for this is to reuse existing code in a MapFunction. The result of applying that map is a DataSet of type Consumer, a custom ob...

Mathias_Peters

Hi, I need to join three streams/streamed data sets in a DLT pipeline. I am reading from a Kinesis data stream a sequence of events per group key. The logically first of the events per group contains a marker which determines whether that group is re...

Mathias_Peters · 11-03-2025

Hi, for a Spark job which does some custom computation, I need to access data from a MongoDB collection and access the elements as of type Document. The reason for this is, that I want to apply some custom type serialization which is already implemen...

Mathias_Peters · 10-09-2024

Hi, we are using asset bundles to deploy artefacts in Databricks. We use a service account and github workflows for that. Currently, we deploy in dev mode whenever a PR is created against the main branch of our gh repo. So far so good. However, we ar...

Mathias_Peters · 08-15-2024

Hi, we are managing our Databricks Workspaces entirely using terraform. As of today, the module to read the current clusters fails with a timeout. We use this:data "databricks_clusters" "this" { }This has worked flawlessly before. However, now we are...

Mathias_Peters

hi @mark_ott , thank you for your help. I have a follow up question regarding data completeness and out of order processing. I have decided to go with the delta table option since super low latency is not an issue and since this option has (seemingly...

Mathias_Peters · 06-17-2025

Hi, did you find answer to this question? I am having similar problems and a slow solution, which I need to improve upon. Thanks in advance

Mathias_Peters · 06-17-2025

This code snippet seems to have no relationship with the question whatsoever. Is this generated?

Mathias_Peters · 01-06-2025

Hi, we are having similar issues with schema hints formulated in fully qualified DDL, e.g. "a STRUCT<b INT>" etc. Did you find a solution? Also, did you specify the schema hint using the dot-notation, e.g. "a.b INT" before ingesting any data or after...

Mathias_Peters · 11-13-2024

Hi, I had a similar problem today. I changed the way, that we deploy our main bundle using pull requests and in order to play around with it locally, I copied python and dbt code into the databricks src dir (that is normally done during a github work...

Databricks Community

User Stats

User Activity

Question on how to properly write a dataset of custom objects to MonogDB

Streamed DLT Pipeline using a lookup table

Reading MongoDB collections into an RDD

Is there a way to use name prefixes for asset bundle deployments to prevent collisions

Databricks Terraform Module for reading the current clusters times out

Re: Streamed DLT Pipeline using a lookup table

Re: Clarification Needed: Ensuring Correct Pagination with Offset and Limit in PySpark

Re: Clarification Needed: Ensuring Correct Pagination with Offset and Limit in PySpark

Re: Auto Loader Schema Hint Behavior: Addressing Nested Field Errors

Re: Asset bundle doesn't sync files to workspace