03-13-2024 06:58 AM
Hello community,
i have the following problem: I am using automl to solve a regression model, but in the preprocessing my dataset is sampled to ~30% of the original amount.
I am using runtime 14.2 ML
Driver: Standard_DS4_v2 28GB Memory 8 cores
Worker: Standard_DS4_v2 28GB Memory 8 cores (min 1, max 2)
i allready set spark.task.cpus = 8, but my dataset is still down sampled 😞
03-14-2024 03:06 AM
Hi @Mirko, I understand the challenge you’re facing with your AutoML regression model and the downsampling of your dataset.
Let’s break down the situation and explore potential solutions:
Dataset Size and Downsampling:
AutoML Behavior:
Potential Solutions:
Next Steps:
Remember that AutoML aims to simplify the model-building process, but understanding its behavior and tuning parameters can lead to better results. If you encounter any specific issues during these steps, feel free to ask for further assistance! 😊
03-14-2024 03:06 AM
Hi @Mirko, I understand the challenge you’re facing with your AutoML regression model and the downsampling of your dataset.
Let’s break down the situation and explore potential solutions:
Dataset Size and Downsampling:
AutoML Behavior:
Potential Solutions:
Next Steps:
Remember that AutoML aims to simplify the model-building process, but understanding its behavior and tuning parameters can lead to better results. If you encounter any specific issues during these steps, feel free to ask for further assistance! 😊
03-15-2024 01:56 AM
Thank you for your detailed answer. I followed your sugestions with the following result:
- repartioing of the data didnt change anything
- i checked the metrics of the workers and the memory is indeed nearly fully used (10gig is used, nearly 17gig is cached)
- i do not fully understand why my relativ small dataset creates such a big memory demand, maybe it results in the amount of categorial features. One hot encoding could result in many "extra columns"
03-19-2024 04:50 AM
I am pretty sure that i know what the problem was. I had a timestamp column (with second precision) as a feature. If they get one hot encoded, the dataset can get pretty large.