cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Science & Machine Learning

Forum Posts

bruno_valero
by New Contributor II
  • 4258 Views
  • 2 replies
  • 1 kudos

How to download a .csv or .pkl file from databricks?

When I save files on "dbfs:/FileStore/shared_uploads/brunofvn6@gmail.com/", it doesn't appear anywhere in my workspace. I've tried to copy the path of the workspace with the right mouse button, pasted on ("my pandas dataframe").to_csv('path'), but wh...

  • 4258 Views
  • 2 replies
  • 1 kudos
Latest Reply
bruno_valero
New Contributor II
  • 1 kudos

I think I discover how to do this. Is in the label called data in the left menu of the databricks environment, in the top left of the menu there are two labels "Database Tables" and "DBFS" in which "Database Table" is the default label. So it is just...

  • 1 kudos
1 More Replies
Data_Analytics1
by Contributor III
  • 969 Views
  • 1 replies
  • 2 kudos

Resolved! File not found error. Does OPTIMIZE deletes initial versions of the delta table?

df = (spark.readStream.format("delta")\              .option("readChangeFeed", "true")\              .option("startingVersion", 1)\              .table("CatalogName.SchemaName.TableName")             )display(df)A file referenced in the transaction l...

  • 969 Views
  • 1 replies
  • 2 kudos
Latest Reply
swethaNandan
New Contributor III
  • 2 kudos

Had you run vacuum on the table? Vacuum can clean up data files marked for removal and are older than retention period.Optimize compacts files and marks the small files for removal, but does not physically remove the data files

  • 2 kudos
User16461610613
by New Contributor II
  • 1472 Views
  • 1 replies
  • 2 kudos

Free Databricks Training on AWS, Azure, or Google Cloud Good news! You can now access free, in-depth Databricks training on AWS, Azure or Google Cloud...

Free Databricks Training on AWS, Azure, or Google CloudGood news! You can now access free, in-depth Databricks training on AWS, Azure or Google Cloud.  Our on-demand training series walks through how to:Streamline data ingest and management to build ...

image
  • 1472 Views
  • 1 replies
  • 2 kudos
Latest Reply
jose_gonzalez
Moderator
  • 2 kudos

Thank you for sharing this!!

  • 2 kudos
obiamaka
by New Contributor III
  • 3019 Views
  • 6 replies
  • 2 kudos

Resolved! Unable to view jobs in Databricks Workflow

This has been happening for about 2 days now. Any ideas on why this occurs and how it can be resolved. I attached a screenshot.

  • 3019 Views
  • 6 replies
  • 2 kudos
Latest Reply
obiamaka
New Contributor III
  • 2 kudos

This issue got resolved on it's own, but not sure what the problem was, probably a bug from a software update?

  • 2 kudos
5 More Replies
zachclem
by New Contributor III
  • 2748 Views
  • 2 replies
  • 1 kudos

Resolved! Logging model to MLflow using Feature Store API. Getting TypeError: join() argument must be str, bytes, or os.PathLike object, not 'dict'

I'm using databricks. Trying to log a model to MLflow using the Feature Store log_model function. but I have this error: TypeError: join() argument must be str, bytes, or os.PathLike object, not 'dict' I'am using the Databricks runtime ml (10.4 LTS M...

  • 2748 Views
  • 2 replies
  • 1 kudos
Latest Reply
zachclem
New Contributor III
  • 1 kudos

I updated by Databricks Run Time from 10.4 to 12.1 and this solved the issue.

  • 1 kudos
1 More Replies
gaponte
by New Contributor III
  • 1587 Views
  • 2 replies
  • 1 kudos

Resolved! What are the best resources for learning how to tune/optimize Spark?

I know this question/topic is not very specific, but perhaps it asking the question would be useful for people other than me.I am a newbie to Spark, and while I've been able to get my current model training and data transformations running, they are ...

  • 1587 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Greg Aponte​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...

  • 1 kudos
1 More Replies
gaponte
by New Contributor III
  • 3496 Views
  • 5 replies
  • 5 kudos

Unable to install SynapseML on clusters

I would like to run a distributed training using LightGBM but I cannot install SynapseML. I have tried doing so on a few different clusters (note: our clusters are running on AWS, not sure if that matters. Also, I am running the Databricks ML Runtime...

  • 3496 Views
  • 5 replies
  • 5 kudos
Latest Reply
Anonymous
Not applicable
  • 5 kudos

Hi @Greg Aponte​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers y...

  • 5 kudos
4 More Replies
Troy
by New Contributor II
  • 3130 Views
  • 8 replies
  • 7 kudos

databricks-connect error when executing sparkml

I use databricks-connect, and spark jobs related spark dataframe works good. But, when I trigger spark ml code, I am getting errors.For example, after executing in the code: https://docs.databricks.com/_static/notebooks/gbt-regression.htmlpipelineMod...

  • 3130 Views
  • 8 replies
  • 7 kudos
Latest Reply
Oliver_Floyd
Contributor
  • 7 kudos

For information, upgrading python libraries does not resolve all problems.This code works fine on databricks in a notebook :import mlflow model = mlflow.spark.load_model('runs:/cb6ff62587a0404cabeadd47e4c9408a/model')Whereas it failed on intelliJ wit...

  • 7 kudos
7 More Replies
RJ
by New Contributor II
  • 2088 Views
  • 2 replies
  • 2 kudos

Serverless Inference Setup Error

Hello, I am using Azure Databricks Premium and am an Admin on the Workspace. I am trying to create a Serving Endpoint for a registered model created with MLFlow. I can make a traditional endpoint without issues, but when I try to make a serverless en...

  • 2088 Views
  • 2 replies
  • 2 kudos
Latest Reply
Debayan
Esteemed Contributor III
  • 2 kudos

Hi, Model serving was available generally from March 7th 2023 in Azure Databricks. (https://azure.microsoft.com/en-us/updates/generally-available-serverless-realtime-inference-for-azure-databricks/) Also, there are region availabilities for the endpo...

  • 2 kudos
1 More Replies
Tom12
by New Contributor III
  • 2654 Views
  • 5 replies
  • 4 kudos

DBFS REST API - unable to access or upload experiment artifacts - permission denied

Hello,we are trying to achieve artifacts upload to MLflow experiments via REST API. (We have an edge case when we need to do that)But if we try to use DBFS API to upload an artifact, we are not allowed. Always ends up with:`PERMISSION_DENIED: No oper...

  • 2654 Views
  • 5 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Tomas Hanzlik​  I'm sorry you could not find a solution to your problem in the answers provided.Our community strives to provide helpful and accurate information, but sometimes an immediate solution may only be available for some issues.I suggest...

  • 4 kudos
4 More Replies
sanjay
by Valued Contributor II
  • 27858 Views
  • 1 replies
  • 1 kudos

Resolved! torch.cuda.OutOfMemoryError: CUDA out of memory

Hi,I am using pynote/whisper large model and trying to process data using spark UDF and getting following error.torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 172.00 MiB (GPU 0; 14.76 GiB total capacity; 6.07 GiB already allocated...

  • 27858 Views
  • 1 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

@Sanjay Jain​ : The error message suggests that there is not enough available memory on the GPU to allocate for the PyTorch model. This error can occur if the model is too large to fit into the available memory on the GPU, or if the GPU memory is bei...

  • 1 kudos
Hubert-Dudek
by Esteemed Contributor III
  • 932 Views
  • 1 replies
  • 5 kudos

Databricks has introduced new functionality for serving machine learning models through a serverless REST API, enabling the consumption of models outs...

Databricks has introduced new functionality for serving machine learning models through a serverless REST API, enabling the consumption of models outside of Databricks. While serving the model via REST API is ideal for external use cases, it is recom...

1677238735536
  • 932 Views
  • 1 replies
  • 5 kudos
Latest Reply
jose_gonzalez
Moderator
  • 5 kudos

Thank you for sharing this!!!

  • 5 kudos
Youssef1985
by New Contributor
  • 1428 Views
  • 2 replies
  • 0 kudos

Pushing SparkNLP Model on Mlflow

Hello Everyone, I am trying to load a SparkNLP (link for more details about the model if required) from Mlflow Registry. To this end, I have followed one tutorial and implemented below codes:import mlflow.pyfunc   class LangDetectionModel(mlflow.pyfu...

  • 1428 Views
  • 2 replies
  • 0 kudos
Latest Reply
tala
New Contributor II
  • 0 kudos

آموزش طراحی سایت https://arzgu.ir/blog/What%20is%20website%20design

  • 0 kudos
1 More Replies
youssefmrini
by Honored Contributor III
  • 794 Views
  • 1 replies
  • 1 kudos
  • 794 Views
  • 1 replies
  • 1 kudos
Latest Reply
youssefmrini
Honored Contributor III
  • 1 kudos

Yes You can. With Databricks Runtime 12.2 LTS ML and above, you can use existing feature tables in Feature Store to augment the original input dataset for all of your AutoML problems: classification, regression, and forecasting.This capability requi...

  • 1 kudos
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels