Databricks Community

pawelmitrus · 05-17-2024

Hi,does anyone know about any existing issue with azure databricks account api? I cannot do below:1. login with cli `databricks auth login --account-id <acc_id>, this is what I get https://adb-MY_ID.azuredatabricks.net/oidc/accounts/MY_ACC_ID/v1/auth...

pawelmitrus · 07-24-2022

I have a Delta table spark101.airlines (sourced from `/databricks-datasets/airlines/`) partitioned by `Year`. My `spark.sql.shuffle.partitions` is set to default 200. I run a simple query:select Origin, count(*) from spark101.airlines group by Origi...

pawelmitrus · 07-20-2022

Hi,As for now we already know that our application will be running 24/7 streaming constantly incoming data. The stream pipeline is very basic, however as for now it's enough to run this pipeline 1x per day (to save the costs of constantly running clu...

pawelmitrus · 02-27-2022

Do I always need to manually invite members of my AAD tenant to ADB workspace, if I don't have SCIM integration configured?EDIT: solved, it works when you go through Azure Portal and get in with "Launch Workspace" button on the ADB resource overview ...

pawelmitrus · 12-13-2021

I'm looking for some information regarding log4j vulnerability - if any databricks runtime should be changed manually by the client or when specific update will be applied.I know I can go through the docs by myself, finding out which log4j library is...

pawelmitrus · 05-20-2024

UPDATE Solved, the very same solution started to work today from running a pipeline with tf - M2M auth. with a service principal with fed auth. That's the 2 from my above post.When trying to follow these steps https://learn.microsoft.com/en-us/azure/...

pawelmitrus · 05-20-2024

Hi Yesh,could you please be more specific which part of that docs I'd like me to check? I've been using this docs already, that's why I'm confusedThanks, Pawel

pawelmitrus · 08-02-2022

I've been struggling myslef with it, but after installing pypmml for spark, I can use the other library, maybe it will work for you:runtime 10.4 LTS MLinstall pypmml-spark (https://github.com/autodeployai/pypmml-spark)install pmml4s-spark (org.pmml4s...

pawelmitrus · 07-31-2022

I think it is something that Databricks does when running a query which result is returned to the notebook. When I write this sql statement to the storage, then it's only 1 job with 2 stages - as expected.

pawelmitrus · 07-26-2022

Yeah, this is all I've got. The things I should also mention:databricks runtime 10.4 LTSI have disabled AQEIt looks like databricks have some kind of approach of creating jobs / stages in the way that:start with 1multiply by 4, if not enough then...m...

Databricks Community

User Stats

User Activity

Azure Databricks account api can't auth

Why Databricks spawns multiple jobs

Shutting down a job cluster, when streaming is over

github.com

log4j vulnerability - action plan for clients

Re: Azure Databricks account api can't auth

Re: Azure Databricks account api can't auth

Re: Getting Py4J "Could not find py4j jar" error when trying to use pypmml, solution doesn't work

Re: Why Databricks spawns multiple jobs

Re: Why Databricks spawns multiple jobs