Databricks Community

Mirko · 03-13-2024

Hello community,i have the following problem: I am using automl to solve a regression model, but in the preprocessing my dataset is sampled to ~30% of the original amount.I am using runtime 14.2 ML Driver: Standard_DS4_v2 28GB Memory 8 coresWorker: S...

Mirko · 08-22-2022

I detected that there ist a limit of 25 in /2.1/jobs/list. While from what i know /2.0/jobs/list had no limit? Why is this the case? Is it planned to increase the limit at some point?I know that the offset concept exist, but from my standpoint that i...

Mirko · 02-02-2022

The following situation: I am creating a Database with location somewhere in my Azure Lake Gen 2.CREATE SCHEMA IF NOT EXISTS curated LOCATION 'somelocation'Then i want a specific Table in curated to be in a subfolder in 'somelocation':CREATE TABLE IF...

Group vs User rights · 01-26-2022

I have a small question: How does the combination of group and user rights work? Is it like in azure, that if i have for example databricks sql acces threw a (databricks) group i am member of, but in my personal account databricks sql is not enabled...

Mirko · 01-17-2022

The situation is as following: i have a sheduled job, which uses dbutils.notebook.run(path,timeout) . During the last week everything worked smooth. During the weekend the job began to fail, at the dbutils.notebook.run(path,timeout) command. I get th...

Mirko · 03-19-2024

I am pretty sure that i know what the problem was. I had a timestamp column (with second precision) as a feature. If they get one hot encoded, the dataset can get pretty large.

Mirko · 03-15-2024

Thank you for your detailed answer. I followed your sugestions with the following result:- repartioing of the data didnt change anything- i checked the metrics of the workers and the memory is indeed nearly fully used (10gig is used, nearly 17gig is ...

Mirko · 08-24-2022

@Rohit Rajendran Thank you for the technical explanation. Do you think it would be really a problem to increase the limit? From my perspective listining inofrmation from 500 jobs is still not a lot of data.

Mirko · 08-24-2022

@Joseph Kambourakis thank you for the quick respond. Very cool that the soulution is so flexibel.

Mirko · 02-02-2022

Hi Piper, i tried it again. This time on 10.2 instead of 9.0. Now it works like desired. But still a little bit strange.

Databricks Community

User Stats

User Activity

AutoMl Dataset too large

Why is there a limit in /2.1/jobs/list?

Location for DB and for specific tables in DB

Group vs User rights

strange error with dbutils.notebook.run(...)

Re: AutoMl Dataset too large

Re: AutoMl Dataset too large

Re: Why is there a limit in /2.1/jobs/list?

Re: Why is there a limit in /2.1/jobs/list?

Re: Location for DB and for specific tables in DB