Databricks Community

SumeshKashyap

Hi everyone,Part 2 of my Helsinki live transit project is out. In part 1 I built a Databricks App that follows every tram and metro train in Helsinki in real time: a Lakeflow pipeline on HSL's live MQTT feed, Lakebase behind the live map, and a Genie...

SumeshKashyap

Hi everyone! I built a near-real-time app that follows every tram and metro train in Helsinki using only Databricks: no Kafka, no extra servers, and the whole thing deploys from one repository.What the app doesA live map of every tram and metro train...

SumeshKashyap

Hello @A_Motia Correct, there's no GA idle-timeout or auto-stop for Apps today. An app is billed for its compute size the whole time it's in the Running state, whether anyone uses it or not. The supported workaround is to stop it yourself on a schedu...

SumeshKashyap

Hello @capoorhimanshu On standard access mode this is expected. Scala/Java closures (foreachPartition, mapPartitions, UDFs) don't run in the executor JVM where the cluster env vars are set. They run in an isolated per-user sandbox process with a fixe...

SumeshKashyap

Hello @ASH1243434 You're right that it's the wrong tool for this. binaryFile loads each whole file into one content byte-array cell. That is capped at about 2 GB (spark.sql.sources.binaryFile.maxLength), and even below the cap a single task has to pu...

SumeshKashyap

hello @sudopradip , if the response was helpful, please accept the reply as answer so that it is beneficial for other community members

SumeshKashyap

Hi @sudopradip Short version: the SQL connector's PUT/GET is a staging path meant for external clients. From inside serverless, the storage-gateway hostname resolves through the serverless plane's private DNS to an endpoint that doesn't serve that tr...

Databricks Community

User Stats

User Activity

Should I wait for the tram? Adding agents to a live Helsinki transit app on Databricks (part 2)

Built a live Helsinki tram tracker on Databricks with Lakeflow, Lakebase, Genie and Apps

Re: Query on Databricks apps idle time out

Re: Cluster environment variables not visible to Java Dataset.foreachPartition / mapPartitions

Re: Best pattern for ingesting large (multi-GB) zip files from an external SFTP into Databricks

Re: Serverless notebook: UC volume PUT via SQL connector fails *.storage.azuredatabricks.net resolve

Re: Serverless notebook: UC volume PUT via SQL connector fails *.storage.azuredatabricks.net resolve