- 1673 Views
- 1 replies
- 2 kudos
Mapping Control data - Maintained by Business User
We are ingesting our data from ADLS into databricks as delta table. After raw layer we need to refer to a control\mapping layer which defines certain logic\measure definition. This would be incorporated in the subsequent silver or gold layer. This co...
- 1673 Views
- 1 replies
- 2 kudos
- 2 kudos
Thanks for your response. Can business user without the help of any script modify any rows in the table after loading it onetime from CSV fiels?
- 2 kudos
- 3357 Views
- 1 replies
- 32 kudos
Databricks Roadmap Azure There are a lot of excitement new features coming in 2022. I tried to put them all on one list: Unity catalog (seems that it ...
Databricks Roadmap AzureThere are a lot of excitement new features coming in 2022. I tried to put them all on one list:Unity catalog (seems that it will exists next to hive metastore and it will be possible to migrate)Control metastore, unity creatio...
- 3357 Views
- 1 replies
- 32 kudos
- 32 kudos
Databricks Roadmap AzureThere are a lot of excitement new features coming in 2022. I tried to put them all on one list:Unity catalog (seems that it will exists next to hive metastore and it will be possible to migrate)Control metastore, unity creatio...
- 32 kudos
- 2117 Views
- 2 replies
- 6 kudos
- 2117 Views
- 2 replies
- 6 kudos
- 6 kudos
Here are the supported data types for the Feature Store:https://docs.databricks.com/applications/machine-learning/feature-store/feature-tables.html#supported-data-typesAs you can see, image is not between them, but you could use BinaryType.
- 6 kudos
- 1724 Views
- 1 replies
- 7 kudos
2021-07-Webinar--Hassle-Free-Data-Ingestion-Social-1200x628
Thanks to everyone who joined the Hassle-Free Data Ingestion webinar. You can access the on-demand recording here. We're sharing a subset of the phenomenal questions asked and answered throughout the session. You'll find Ingestion Q&A listed first, f...
- 1724 Views
- 1 replies
- 7 kudos
- 7 kudos
Check out Part 2 of this Data Ingestion webinar to find out how to easily ingest semi-structured data at scale into your Delta Lake, including how to use Databricks Auto Loader to ingest JSON data into Delta Lake.
- 7 kudos
- 1437 Views
- 1 replies
- 2 kudos
- 1437 Views
- 1 replies
- 2 kudos
- 2 kudos
Not yet, but stay-tuned it's being cooked in the kitchen
- 2 kudos
- 1185 Views
- 2 replies
- 1 kudos
- 1185 Views
- 2 replies
- 1 kudos
- 1 kudos
I can find a link which can help https://docs.databricks.com/dev-tools/databricks-connect.html
- 1 kudos
- 985 Views
- 1 replies
- 0 kudos
- 985 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes.Please refer to our docshttps://docs.databricks.com/applications/machine-learning/manage-model-lifecycle/multiple-workspaces.html
- 0 kudos
- 1005 Views
- 1 replies
- 1 kudos
What algorithms does Databricks AutoML use?
AutoML presumably tries a few different algorithms while hyperparameter searching. What model types are considered?
- 1005 Views
- 1 replies
- 1 kudos
- 1 kudos
At the moment, it's really just xgboost, and sklearn implemenations like random forests, logistic regression, and linear regression as applicable. More possibilities are coming.
- 1 kudos
- 1005 Views
- 1 replies
- 0 kudos
How can I use Non- Spark related libraries like spacy with Databricks and Spark
I have an NLP application that I build on my local machine using spacy and pandas, but now I would like to scale my application to a large production dataset and utilize the benefits of sparks distributed compute. How do I import and utilize a librar...
- 1005 Views
- 1 replies
- 0 kudos
- 0 kudos
It depends on what you mean, but if you're just trying to (say) tokenize and process data with spacy in parallel, then that's trivial. Write a 'pandas UDF' function that expresses how you want to transform data using spacy, in terms of a pandas DataF...
- 0 kudos
- 1830 Views
- 1 replies
- 0 kudos
Muliple Where condition vs AND && in Pyspark
.where((col('state')==state) & (col('month')>startmonth)I can do the where conditions both ways. I think the one below add readability. Is there any other difference and which is the best?.where(col('state')==state).where(col('month')>startmonth)
- 1830 Views
- 1 replies
- 0 kudos
- 0 kudos
You can use explain to see what type of physical and logical plans are getting created . This is the best way to see difference , but as mentioned in the question , it should give the same physical plan
- 0 kudos
- 2392 Views
- 2 replies
- 0 kudos
What is Databricks' model deployment framework?
How do you do deploy a model in Databricks.
- 2392 Views
- 2 replies
- 0 kudos
- 0 kudos
The following resources provide more detail on this:Databricks model registry example notebook: https://docs.databricks.com/_static/notebooks/mlflow/mlflow-model-registry-example.htmlDatabricks model lifecycle - https://docs.databricks.com/applicatio...
- 0 kudos