- 4639 Views
- 7 replies
- 13 kudos
Resolved! Getting started with Databricks Machine Learning
hello all,I am fairly new to Databricks technologies and I have taken the Lakehouse Fundamentals course but I am interested in Machine Learning technologies. I will appreciate any help with materials and curated free study paths and packs that can he...
- 4639 Views
- 7 replies
- 13 kudos
- 13 kudos
https://pages.databricks.com/rs/094-YMS-629/images/LearningSpark2.0.pdf is a free book and has some machine learning examples. The way I learned was mostly from the docs, which are good and have good coding examples.
- 13 kudos
- 3507 Views
- 2 replies
- 1 kudos
MLflow stopped working after one a few successful runs
Both the mlflow.log_metrics() call and the web UI worked in the first few days but started failing at some point. The log doesn't give any clue why this is happening. It's suspicious but is there a limit of mlflow requests? It's quite annoying that t...
- 3507 Views
- 2 replies
- 1 kudos
- 6677 Views
- 3 replies
- 15 kudos
Resolved! Do One-Hot-Encoding (OHE) before or after split data to train and test dataframe
Hi,I wonder that I should do OHE before or after I split data to build up a ML model.Please give some advise.
- 6677 Views
- 3 replies
- 15 kudos
- 15 kudos
Hi @Nhat Hoang​ ,While not Databricks-specific, here's a good answer:"If you perform the encoding before the split, it will lead to data leakage (train-test contamination). In this sense, you will introduce new data (integers of Label Encoders) and u...
- 15 kudos
- 686 Views
- 0 replies
- 0 kudos
Databricks-hosted MLFlow ignores the `mlflow.user` tag when set in the "runs/create" REST API call and doesn't let me change it after a ...
Databricks-hosted MLFlow ignores the `mlflow.user` tag when set in the "runs/create" REST API call and doesn't let me change it after a run is created.Open source MLFlow respects the tag.Could you please change the server to respect this field when i...
- 686 Views
- 0 replies
- 0 kudos
- 5196 Views
- 7 replies
- 15 kudos
What programming frameworks and languages can be used with Databricks Feature Store
To leverage Databricks feature store, can only Python be utilized? If otherwise, what other language frameworks are supported. Below is my question in 2 partsPart 1) What languages can be utilized to write data frames as feature tables in the Feature...
- 5196 Views
- 7 replies
- 15 kudos
- 2221 Views
- 3 replies
- 10 kudos
I am Avi, a Solutions Architect at Databricks. We have built an application to demonstrate how AI-capabilities could be easily integrated to deliver n...
I am Avi, a Solutions Architect at Databricks. We have built an application to demonstrate how AI-capabilities could be easily integrated to deliver novel user experiences. The application allows users to submit images and text, and uses these inputs...
- 2221 Views
- 3 replies
- 10 kudos
- 2089 Views
- 3 replies
- 1 kudos
inability to consume model
Hello, I would like to ask where the problem may be. I want to create a real time endpoint to real time model infering. . i have created a simple cluster but i am not able to deploy the model i still get a yellow status - pending. the whole pro...
- 2089 Views
- 3 replies
- 1 kudos
- 1233 Views
- 0 replies
- 5 kudos
youtu.be
I'm Avi, a Solutions Architect at Databricks working at the intersection of Data Engineering and Machine Learning.Streaming data processing has moved from niche to mainstream, and deploying machine learning models in such data streams opens up a mult...
- 1233 Views
- 0 replies
- 5 kudos
- 8072 Views
- 3 replies
- 3 kudos
Resolved! Spark Error/Exception Handling
I am creating new application and looking for ideas how to handle exceptions in Spark, for example ThreadPoolExecution. Are there any good practice in terms of error handling and dealing with specific exceptions ?
- 8072 Views
- 3 replies
- 3 kudos
- 3 kudos
@Krzysztof Nojman​ Can you please click on the "Select As Best" button if you find the information provided helps resolve your question.
- 3 kudos
- 14192 Views
- 7 replies
- 16 kudos
Resolved! Way of using pymc.model_to_graphviz into a Databricks notebook
Hi everybody,I created a simple bayesian model using the pymc library in Python. I would like to graphically represent my model using the pymc.model_to_graphviz(model=model) method.However, it seems it does not work within a databrcks notebook, even ...
- 14192 Views
- 7 replies
- 16 kudos
- 5192 Views
- 1 replies
- 4 kudos
Resolved! Insert into delta table fails
Hello experts. We are trying to execute an insert command with less columns than the target table:Insert into table_name( col1, col2, col10)Select col1, col2, col10from table_name2However the above fails with:Error in SQL statement: DeltaAnalysisExce...
- 5192 Views
- 1 replies
- 4 kudos
- 4 kudos
Hi @ELENI GEORGOUSI​ Yes. When you are doing an insert, your provided schema should match with the target schema else it would throw an error.But you can still insert the data using another approach. Create a dataframe with your data having less colu...
- 4 kudos
- 2433 Views
- 3 replies
- 1 kudos
How does mlflow determine if a pyfunc model uses SparkContext?
I've been getting this error pretty regularly while working with mlflow:"It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that ...
- 2433 Views
- 3 replies
- 1 kudos
- 1 kudos
I checked the page and it looks like there is no integration with Datarobot and Datarobot doesn't contribute to mlflow. https://mlflow.org/ has all the integrations listed
- 1 kudos
- 2848 Views
- 1 replies
- 2 kudos
Resolved! Unable to create feature table using databricks API .FeatureStoreClient()
I am following example steps from databricks documentation https://docs.databricks.com/_static/notebooks/machine-learning/feature-store-taxi-example.htmlI am using Feature Store client v0.3.6 and above.However on trying to create feature table with f...
- 2848 Views
- 1 replies
- 2 kudos
- 2 kudos
After much digging, observed i was using standard runtime. Once i switched to ML runtime of databricks, issue was resolved. To use Feature Store capability, ensure that you select a Databricks Runtime ML version from the Databricks Runtime Version dr...
- 2 kudos
- 1631 Views
- 1 replies
- 4 kudos
Stream data from Delta tables replicated with Fivetran into DLT
I'm attempting to stream into a DLT pipeline with data replicated from Fivetran directly into Delta tables in another database than the one that the DLT pipeline uses. This is not an aggregate, and I don't want to recompute the entire data model eac...
- 1631 Views
- 1 replies
- 4 kudos
- 4 kudos
Hi @M A​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else bricksters will get back to you soon. Thanks
- 4 kudos
- 819 Views
- 0 replies
- 2 kudos
What is "git source" supposed to be in the run page for an mlflow experiment?
I have an mlflow experiment with runs in it. When I go to a run's page (with the parameters/metrics/logged artifacts), there is a part that says `git source: my_project_name@some_letters` and I was wondering what that was supposed to point to.When I ...
- 819 Views
- 0 replies
- 2 kudos
