Databricks Community

Mr__E · 08-29-2022

We have a couple sources we'd already set up to stream to prod using a 3p system. Is there a way to sync this directly to our dev workspace to build pipelines? eg. directly connecting to a cluster in prod and pull with a job cluster, dump to S3 and u...

Mr__E · 05-11-2022

I built a model which is used for ranking and I have a notebook that takes that model to generate rankings and then uses a UDF-based metric to evaluate those rankings. Is there any way that I can have this ranking / UDF be used during the AutoML trai...

Mr__E · 05-02-2022

Accounts added after we turned on SSO don't allow me to restrict their cluster creation abilities. How can I undo this, so I can prevent business people from writing to ETLed data?

Mr__E · 04-15-2022

I created a table that aggregates data by year and week of year and display this in a chart over time. As part of the query, I order by the year, then week columns. In the visualization on the query (in the SQL editor), I disabled the sort, because i...

Mr__E · 04-02-2022

We have a GitHub workflow that generates a python wheel and uploads to a shared S3 available to our Databricks workspaces. When I install the Python wheel to a normal compute cluster using the path approach, it correctly installs the Python wheel and...

Mr__E · 05-02-2022

Nevermind. Turns out someone was giving everyone admin privileges when they weren't supposed to and I didn't notice.

Mr__E · 04-02-2022

Yeah, it was an authentication issue. Turns out the compute clusters were set up with instance profiles, but never the job profiles, so when the wheel process was set up it failed for jobs.TL;DR: you need to apply instance profiles for access to shar...

Mr__E · 04-02-2022

We set up a GitHub workflow to upload the built python wheels to a shared S3 cluster, which is accessible from the workspaces page, then we can add using the dialogue you show. However, I'm having an issue using it for job clusters, so YMMV.

Mr__E · 04-02-2022

Thanks! This is what I'm already doing. It works fine for normal compute clusters, but it doesn't work for job clusters and gives the error mentioned above.

Mr__E · 04-02-2022

Thanks! I used this pattern of adding underscores to simplify raw dumping.

Databricks Community

User Stats

User Activity

Sync prod WS DBs to dev WS DBs

Custom AutoML evaluation metric for ranking model

SSO and cluster creation restriction

Databricks dashboard removing order, incorrectly sorted.

Using shared python wheels for job compute clusters

Re: SSO and cluster creation restriction

Re: Using shared python wheels for job compute clusters

Re: How to refer the ".whl" files from repos into a Databricks "python wheel" task in Databricks jobs?

Re: Using shared python wheels for job compute clusters

Re: Importing MongoDB with field names containing spaces