Databricks

alonisser · 08-27-2023

and it seems that older data (From the "replaced" table) isn't being removed, long after the retention period I'd be glad for clues on how to handle this

alonisser · 11-13-2022

is there an API for that? as I couldn't find a way to do this through the UIClassic serving for now (didn't get access to the new "serverless" offering)

alonisser · 05-22-2022

We've setup a premium workspace with passthrough credentials cluster , while they do work and access my adls gen 2 storageI can't make it install a library on the cluster from there. and keeping getting"Library installation attempted on the driver no...

alonisser · 03-14-2022

Currently we're on an azure databricks workspace, we've setup during the POC, a long time ago. In the meanwhile we have built quite a production workload above databricks.Now we want to split workspaces - one for analysts and one for data engineeri...

alonisser · 02-22-2022

Question about spark checkpoints and offsets in a running streamWhen the stream started I needed tons of partitions, so we've set it with spark.conf to 5000As expected offsets in the checkpoint contain this info and the job used this value. Then we'...

alonisser · 09-23-2023

Thanks, I know that, but the table history shows 30 days, but the actual data size and number of files and all other indicators , correlate to 170 days.

alonisser · 11-19-2022

any clues?

alonisser · 06-16-2022

Sorry I can't figure this out, the link you've added is irrelevant for passthrough credentials, if we add it the cluster won't be passthrough, Is there a way to add this just for a specific folder? while keeping passthrough for the rest?

alonisser · 05-25-2022

@Nancy Gupta , As far as I can trace this issue, it's about the token isn't set up yet when the cluster is starting; I assume it does work with pass-through credentials after starting the collection regularly? My hack was to copy the library to the ...

alonisser · 05-24-2022

I currently got a hack, of copying the library from the data lake to root dbfs. and from there . but I don't like it

Databricks

User Stats

User Activity

Trying to vacuum a table that is constantly being "createdOrReplaced"

ML inference server (with REST api) how can I specify a GPU cluster.?

Failing to install a library from dbfs mounted storage (adls2) with pass through credentials cluster

How to migrate an existing workspace for an external metastore

Changing shuffle.partitions with spark.conf in a spark stream - isn't respected even after a checkpoint

Re: Trying to vacuum a table that is constantly being "createdOrReplaced"

Re: ML inference server (with REST api) how can I specify a GPU cluster.?

Re: Failing to install a library from dbfs mounted storage (adls2) with pass through credentials cluster

Re: Failing to install a library from dbfs mounted storage (adls2) with pass through credentials cluster

Re: Failing to install a library from dbfs mounted storage (adls2) with pass through credentials cluster