Databricks Community

Nice design. One option for the gate: instead of paging through the Jobs API, query the jobs system tables. It's one SQL query and works across workspaces:SELECT job_id FROM system.lakeflow.job_run_timeline WHERE job_id IN (<dimjob1..5 ids>) AND resu...

On the open question about an existing MV: the docs only describe setting those two properties when the MV is created. So I'd treat it as a create-time setting and not count on ALTER working on an existing MV. What I'd do:1. Add the properties to the...

Agree there's no timezone option on the connection or foreign catalog. Rather than casting in every query, I'd fix it once in a governed view:CREATE VIEW main.gold.orders_v ASSELECTorder_id,to_utc_timestamp(created_at, 'America/Chicago') AS created_a...

Your point about the delegation hop is a critical detail. It means that even if you want to keep workloads on dedicated classic compute, you still have to enable the serverless path workspace-wide just to handle policy evaluation.We didn’t hit this o...

Congrats on the first post, Juan. Treating logic, storage, and publishing as three separate decisions is the most useful part, because most layer debates go wrong when those get collapsed into one.My default is simplicity. Start with bronze, silver a...

@ASH1243434 For this workload, I'd recommend a scheduled Databricks Job using Paramiko → UC Volume → ZIP extraction → Auto Loader.Although the native Lakeflow SFTP connector supports binaryFile, it doesn't provide a direct raw ZIP-to-Volume transfer ...

Hello @capoorhimanshu On standard access mode this is expected. Scala/Java closures (foreachPartition, mapPartitions, UDFs) don't run in the executor JVM where the cluster env vars are set. They run in an isolated per-user sandbox process with a fixe...

I tested this independently and was able to reproduce a very similar failure pattern on AWS us-east-2, using a Databricks serverless.Here are the results against the same UC Volume:Upload method ResultDirect /Volumes/.. writePASS (byte readback verif...

Hi Is it possible ti share the full error message (eg. AuthorizationHeaderMalformed/ InvalidToken etc.) so we can determine the root cause. Also, did you add a regional endpoint for this S3 bucket to your serverless NCC?

Using a Unity Catalog Volume to write and read back the DataFrame seems like a practical alternative to avoid mounting cloud storage. Just make sure to account for the extra I/O overhead and schema preservation when reloading the data.

Hi Reza, Sorry you’ve been passed between support channels. The three-App limit and automatic stopping are likely caused by the account still being treated as Free Edition. Free Edition is limited to three Apps and may stop them automatically. A Prem...

Hi, I've just done some digging into this and as far as I can tell there shoulnd't be any charges for the Lakeflow Ingestion until you hit the 100dbu limit. The only thing I want to clarify is that the 100DBU limit is a total for all ingestion pipeli...

its not a wrong identity, since i used google sign in - now its not showing my workspace, what happen ? i try in multiple browser. not working.1st I log with email its redirect to the genie one page also its not working there when i type its connect ...