- 3243 Views
- 1 replies
- 1 kudos
Unable to install sf and rgeos R packages on the cluster
Got following errorjava.lang.RuntimeException: Installation failed with message:Error installing R package: Could not install package with error: installation of package ‘rgdal’ had non-zero exit status Full error log available at /databricks/drive...
- 3243 Views
- 1 replies
- 1 kudos
- 1 kudos
We can use the below init script to install the packages in the cluster:%scala dbutils.fs.put("dbfs:/databricks/init_scripts/rlib.sh", """ #!/bin/bash sudo apt-get install -y libudunits2-dev sudo add-apt-repository ppa:ubuntugis/ubuntugis-uns...
- 1 kudos
- 5824 Views
- 1 replies
- 0 kudos
Error importing pip package s3fs
A job recently began failing with the following error when a python notebook imports the pip package s3fs.ImportError: cannot import name 'maybe_sync' from 'fsspec.asyn' (/databricks/python/lib/python3.8/site-packages/fsspec/asyn.py) ImportError Tr...
- 5824 Views
- 1 replies
- 0 kudos
- 0 kudos
While checking the init script is installing the s3fs version 0.5.2.This version has issues at the moment from the pypi. I have tested version 0.6.0 that works fine. please change your requirement.txt file with a newer version of s3fs. Below is the p...
- 0 kudos
- 2083 Views
- 1 replies
- 1 kudos
- 2083 Views
- 1 replies
- 1 kudos
- 1 kudos
You can find a lot more info on this at this MLflow product page, including a comparison table at the bottom. I'd summarize that comparison as: Databricks provides three key things in its managed MLflow service.Security: MLflow experiments, models, ...
- 1 kudos
- 884 Views
- 0 replies
- 0 kudos
Feature Discovery
How would one discover features here and also know how to make sense of these features?Ideally, we can trace the usage of features in code as well.
- 884 Views
- 0 replies
- 0 kudos
- 2528 Views
- 1 replies
- 0 kudos
- 2528 Views
- 1 replies
- 0 kudos
- 0 kudos
You can find the MLflow version in the runtime release notes, along with a list of every other library provided. E.g., for DBR 8.3 ML, you can look at the release notes for AWS, Azure, or GCP.The MLflow client API (i.e., the API provided by installi...
- 0 kudos
- 1654 Views
- 1 replies
- 0 kudos
Muliple Where condition vs AND && in Pyspark
.where((col('state')==state) & (col('month')>startmonth)I can do the where conditions both ways. I think the one below add readability. Is there any other difference and which is the best?.where(col('state')==state).where(col('month')>startmonth)
- 1654 Views
- 1 replies
- 0 kudos
- 0 kudos
You can use explain to see what type of physical and logical plans are getting created . This is the best way to see difference , but as mentioned in the question , it should give the same physical plan
- 0 kudos
- 1497 Views
- 2 replies
- 0 kudos
How do I efficiently read image data for a deep learning model?
How do I efficiently read image data for a deep learning model?
- 1497 Views
- 2 replies
- 0 kudos
- 0 kudos
Our documentation provides nice examples of preparing image data for training and inference.Training: See docs for AWS, Azure, GCPInference: See reference solution for AWS, Azure, GCP
- 0 kudos
- 2068 Views
- 4 replies
- 0 kudos
How do you control the cost of provisioning a cluster?
How do you govern the cost of running clusters in Databricks so you're not sticker shocked?
- 2068 Views
- 4 replies
- 0 kudos
- 0 kudos
Less use of Interactive cluster and more use of job cluster can one of the way above others
- 0 kudos
- 2167 Views
- 1 replies
- 0 kudos
- 2167 Views
- 1 replies
- 0 kudos
- 0 kudos
Follow the instruction at Share models across workspaces.
- 0 kudos
- 2936 Views
- 1 replies
- 0 kudos
When should we use offline store vs online store for Feature Store?
Looking at the docs we see both options, can we use both e.g.?
- 2936 Views
- 1 replies
- 0 kudos
- 0 kudos
Online store is for real time inferencing, in most case you will use the offline store.
- 0 kudos
- 2271 Views
- 2 replies
- 0 kudos
What is Databricks' model deployment framework?
How do you do deploy a model in Databricks.
- 2271 Views
- 2 replies
- 0 kudos
- 0 kudos
The following resources provide more detail on this:Databricks model registry example notebook: https://docs.databricks.com/_static/notebooks/mlflow/mlflow-model-registry-example.htmlDatabricks model lifecycle - https://docs.databricks.com/applicatio...
- 0 kudos
- 1699 Views
- 1 replies
- 0 kudos
- 1699 Views
- 1 replies
- 0 kudos
- 0 kudos
https://databricks.com/blog/2017/05/18/taking-apache-sparks-structured-structured-streaming-to-production.html
- 0 kudos
- 4675 Views
- 1 replies
- 0 kudos
Text length limitations in the display() function
Is there a way to change the limit to the length of strings that can be shown using the display() function in notebooks? If I'm noticing truncation, what can I do?
- 4675 Views
- 1 replies
- 0 kudos
- 0 kudos
There is a 500 character limit to strings in columns which is non-configurable. To see the full contents of the column, you can either use the tooltip to expand the cell or download the full results.
- 0 kudos
