- 4490 Views
- 1 replies
- 0 kudos
What's the best way to implement long term data versioning?
I'm a data scientist creating versioned ML models. For compliance reasons, I need to be able to replicate the training data for each model version. I've seen that you can version datasets by using delta, but the default retention period is around 30 ...
- 4490 Views
- 1 replies
- 0 kudos
- 0 kudos
Delta, as you mentioned has a feature to do time travel and by default, delta tables retain the commit history for 30 days. Operations on history of the table are parallel but will become more expensive as the log size increasesNow, in this case - s...
- 0 kudos
- 1343 Views
- 1 replies
- 0 kudos
- 1343 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes.Please refer to our docshttps://docs.databricks.com/applications/machine-learning/manage-model-lifecycle/multiple-workspaces.html
- 0 kudos
- 2028 Views
- 1 replies
- 0 kudos
- 2028 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes!You will have to pip install mlflowin your environment as a first step. For more details, see: https://docs.databricks.com/applications/mlflow/access-hosted-tracking-server.html
- 0 kudos
- 1718 Views
- 1 replies
- 0 kudos
Resolved! How is Databricks AutoML different than other AutoML products out there?
How does it provide a glass box view?
- 1718 Views
- 1 replies
- 0 kudos
- 0 kudos
Depending on which solution you use, GlassBox means that any interactive work you do via point & click, we automatically generate the code behind the scene and generate notebooks used for each experiment that was ran under the hood, in addition for a...
- 0 kudos
- 1768 Views
- 1 replies
- 0 kudos
What are the differences between Open Source and Hosted MLFlow?
We have been using open source MLflow, how will it benefit us to move to Databricks mlflow?
- 1768 Views
- 1 replies
- 0 kudos
- 0 kudos
Please see https://databricks.com/product/managed-mlflow
- 0 kudos
- 562 Views
- 0 replies
- 0 kudos
- 562 Views
- 0 replies
- 0 kudos
- 1515 Views
- 1 replies
- 1 kudos
What algorithms does Databricks AutoML use?
AutoML presumably tries a few different algorithms while hyperparameter searching. What model types are considered?
- 1515 Views
- 1 replies
- 1 kudos
- 1 kudos
At the moment, it's really just xgboost, and sklearn implemenations like random forests, logistic regression, and linear regression as applicable. More possibilities are coming.
- 1 kudos
- 1556 Views
- 1 replies
- 0 kudos
How can I use Non- Spark related libraries like spacy with Databricks and Spark
I have an NLP application that I build on my local machine using spacy and pandas, but now I would like to scale my application to a large production dataset and utilize the benefits of sparks distributed compute. How do I import and utilize a librar...
- 1556 Views
- 1 replies
- 0 kudos
- 0 kudos
It depends on what you mean, but if you're just trying to (say) tokenize and process data with spacy in parallel, then that's trivial. Write a 'pandas UDF' function that expresses how you want to transform data using spacy, in terms of a pandas DataF...
- 0 kudos
- 2712 Views
- 1 replies
- 0 kudos
Resolved! Is there documentation about the feature store API and how it's architected under the hood?
- 2712 Views
- 1 replies
- 0 kudos
- 0 kudos
I don't think we have a lot of internal docs, just high-level explanations like https://databricks.com/blog/2021/05/27/databricks-announces-the-first-feature-store-integrated-with-delta-lake-and-mlflow.htmlHowever I don't think there's much to it. Th...
- 0 kudos
- 2084 Views
- 1 replies
- 0 kudos
- 2084 Views
- 1 replies
- 0 kudos
- 0 kudos
The feature store has both online / offline components. The offline feature store is used for feature discovery, model training, and batch inference and is backed by Delta tables. You could read/write to offline store from Databricks clusters that...
- 0 kudos
- 2125 Views
- 1 replies
- 1 kudos
What are best NLP libraries to use with Spark
Best NLP APIs to use with Spark which gives better performance
- 2125 Views
- 1 replies
- 1 kudos
- 1 kudos
By far the most popular and comprehensive library, to my knowledge, for Spark-native distributed NLP, is spark-nlp from John Snow Labs. https://nlp.johnsnowlabs.com/ It is open source (but with commercial support options) and has a whole lot of funct...
- 1 kudos
- 3212 Views
- 1 replies
- 0 kudos
- 3212 Views
- 1 replies
- 0 kudos
- 0 kudos
These terms are borrowed from scikit-learn, and the idea is the same. A transformer is just a component of a pipeline that transforms the data in some way. An estimator is also a transfomer, but one that additionally needs to be 'fit' on data before ...
- 0 kudos
- 9972 Views
- 1 replies
- 0 kudos
Resolved! How can I embed image to my notebook?
- 9972 Views
- 1 replies
- 0 kudos
- 0 kudos
If the image is a result of, for example, a plotting library's output, it should just render as-is.If it's not, then one simple approach is to write a markdown (%md) cell and include a link to the image: Of course this requires t...
- 0 kudos
- 2231 Views
- 1 replies
- 0 kudos
Resolved! Best practice for Image manipulation
Can you please recommend suggestions for image manipulation once you read the data as an image ? Any specific library to use?
- 2231 Views
- 1 replies
- 0 kudos
- 0 kudos
Spark has a built-in 'image' data source which will read a directory of images files as a DataFrame: spark.read.format("image").load(...). The resulting DataFrame has the pixel data, dimensions, channels, etc.You can also read image files 'manually' ...
- 0 kudos
- 6441 Views
- 2 replies
- 0 kudos
Can I access Delta tables outside of Databricks Runtime?
Is it possible to write same table with Databricks and from OSS too, Also what if I want to read the data from Map redeuce or hive
- 6441 Views
- 2 replies
- 0 kudos
- 0 kudos
Yes. The Delta client is open source, and lets you read/write Delta tables if you add it to your external application. See https://docs.delta.io/latest/index.html
- 0 kudos
