- 1698 Views
- 1 replies
- 7 kudos
Materialized views are a powerful feature soon available on databricks. Unlike traditional views, which store the query definition, materialized views...
Materialized views are a powerful feature soon available on databricks. Unlike traditional views, which store the query definition, materialized views physically store the data, making it available for faster querying. This translates to significantl...
- 1698 Views
- 1 replies
- 7 kudos
- 7 kudos
Very informative, Thanks for sharing
- 7 kudos
- 1587 Views
- 1 replies
- 2 kudos
Run with UUID *** is already active when running automl
Hi, I'm tried using databricks autoML API following the documentation and example notebook. The documentation and example are pretty straight forward however I encountered the following error:Exception: Run with UUID 1315376a0cbb4657b5d23fa552efba4b ...
- 1587 Views
- 1 replies
- 2 kudos
- 2 kudos
@Al IDI​ - could you please let us know the ML runtime version you have ran into this? could you please try setting and see if it works? spark.conf.set("spark.databricks.mlflow.trackHyperopt.enabled", "false")
- 2 kudos
- 1485 Views
- 1 replies
- 0 kudos
how does the data science workflow change in databricks if you start with a nosql database (specifically document store) instead of something more traditional/rdbms type source?
I'm sorry if this is a bad question. The tl;dr is are there any concrete examples of a nosql data science workflows specifically in databricks and if so what are they?is it always the case that our end goal is a dataframe?For us we start as a bunch o...
- 1485 Views
- 1 replies
- 0 kudos
- 0 kudos
Nice sharing, thanks!
- 0 kudos
- 1540 Views
- 1 replies
- 1 kudos
Error on pandas udf usage in databricks, sc.broadcasting random forest loaded from Kedro MLFlow Logger DataSet, cannot pickle '_thread.RLock' object
I'm trying to broadcast a Random forest (sklearn 1.2.0) recently loaded from mlflow, and using Pandas UDF to predict a model.​However, the same code works perfectly on Spark 2.4 + our OnPrem cluster.​I thought it was due to Spark 2.4 to 3 changes, an...
- 1540 Views
- 1 replies
- 1 kudos
- 1 kudos
Anyone?
- 1 kudos
- 1774 Views
- 5 replies
- 1 kudos
Latest Blog PostsJanuary 13 - 20Â Did you get a chance to look at the most recent blog posts? Here are some happening content from the past week that i...
Latest Blog PostsJanuary 13 - 20Did you get a chance to look at the most recent blog posts? Here are some happening content from the past week that is worth the read. What’s New With SQL User-Defined Functions In this blog, we describe several enhanc...
- 1774 Views
- 5 replies
- 1 kudos
- 1 kudos
Thanks @Sujitha Ramamoorthy​ , for sharing with the community these are worth reading and insightful.
- 1 kudos
- 4367 Views
- 8 replies
- 3 kudos
com.amazonaws.services.s3.model.AmazonS3Exception: The bucket is in this region: *** when using S3 Select
Hello,I have a cluster running in us-east-1 region.I hava a Spark job loading data in a DataFrame using s3select format on a bucket in eu-west-1 region.Access and Secret keys are encoded in URI s3a://$AccessKey:$SecretKey@bucket/path/to/dirJob fails ...
- 4367 Views
- 8 replies
- 3 kudos
- 3 kudos
Hello,I tried your suggestion by setting up the peering connection between the 2 VPC but issue remains the same.The error message The bucket is in this region: .... please use this region to retry the requestmakes me think that the root cause is not ...
- 3 kudos
- 9181 Views
- 1 replies
- 1 kudos
Unity Catalog Pricing
Hi All, I would like to understand the pricing model of the Unity Catalog. Earlier I remember there was some mention of the data lineage and a few other features that will have a cost associated with it. If that's true, what other features cost us? W...
- 9181 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Venkadeshwaran K​ ,All Unity Catalog features are provided at no charge to customers, provided they are using a Premium or Enterprise SKU.
- 1 kudos
- 2525 Views
- 3 replies
- 3 kudos
How to circumvent Py4JSecurityException for spark-nlp : Constructor public com.johnsnowlabs.nlp.***(java.lang.String) is not whitelisted.
Running into the following error on our company's cluster. py4j.security.Py4JSecurityException: Constructor public com.johnsnowlabs.nlp.DocumentAssembler(java.lang.String) is not whitelisted.For the following code(which is just tutorial code from the...
- 2525 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Kenan Spruill​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...
- 3 kudos
- 1865 Views
- 3 replies
- 3 kudos
Is it possible to access online feature store (Cosmos DB) outside databricks?
We are building an machine learning application with feature store enabled. Once the model is built, we are trying to move the model artifacts and deploy it in azure ml as online endpoint. Does it possible to access the online store in azure ml endpo...
- 1865 Views
- 3 replies
- 3 kudos
- 3 kudos
if you want databricks to use the feature store, which is in Cosmos DB, yes, it is possible https://learn.microsoft.com/en-us/azure/databricks/machine-learning/feature-store/online-feature-storessuppose you want to consume a future store in Databrick...
- 3 kudos
- 2607 Views
- 7 replies
- 0 kudos
Databricks not able to create table in minio bucket
Trying to create table in minio bucket using databricks.spark.sql("create database if not exists minio_db_1 managed location 's3a://my-bucket/minio_db_1'");I am passing the s3 configurations using spark context.access_key = 'XXXX'secret_key = 'XXXXXX...
- 2607 Views
- 7 replies
- 0 kudos
- 0 kudos
MANAGED LOCATION is for Unity Catalog. Please check if you are under the unity catalog, not under hive metastore. Additionally, with Unity, you are not using sc._jsc.hadoopConfiguration() etc. but just register storage credentials and external locati...
- 0 kudos
- 17015 Views
- 5 replies
- 6 kudos
Resolved! Access the environment variable from the custom container base cluster
Hi Databricks Community, I want to set environment variables for all clusters in my workspace. The goal is to the have environment variable, available in all notebooks executed on the cluster.The environment variable is generated in global init scrip...
- 17015 Views
- 5 replies
- 6 kudos
- 6 kudos
Thanks @Lukasz Lu​ - that worked for me as well. When I used the following script:#!/bin/bash echo MY_TEST_VAR=value1 | tee -a /etc/environment >> /databricks/spark/conf/spark-env.shfor non-docker clusters, MY_TEST_VAR shows up twice in ` /databrick...
- 6 kudos
- 906 Views
- 1 replies
- 0 kudos
Weekly Release Notes Recap Here’s a quick recap of the latest release notes updates from the past one week. Databricks platform release notesJanuary 1...
Weekly Release Notes RecapHere’s a quick recap of the latest release notes updates from the past one week.Databricks platform release notesJanuary 13 - 19, 2023Cluster policies now support limiting the max number of clusters per userYou can now use c...
- 906 Views
- 1 replies
- 0 kudos
- 0 kudos
thanks for the details
- 0 kudos
- 4438 Views
- 3 replies
- 1 kudos
Permission denied: Lightning Logs
I'm doing parameter tuning for a NeuralProphet model (you can see in the image the parameters and code for training)When I try to parallelize the training, it gives me Permission Error.Why can't I access the folder '/databricks/spark/work/*'? Do I ne...
- 4438 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi, Could you please check on cluster-level permissions and let us know if it helps? Please refer: https://docs.databricks.com/security/access-control/cluster-acl.html#cluster-level-permissions
- 1 kudos
- 2077 Views
- 3 replies
- 2 kudos
Cannot write Feature Table because of invalid access token
From a notebook I created a new feature store via:%sql CREATE DATABASE IF NOT EXISTS feature_store_ebp;Within that feature store I fill my table with:feature_store_name = "feature_store_ebp.table_1" try: fs.write_table( name=feature_stor...
- 2077 Views
- 3 replies
- 2 kudos
- 1452 Views
- 2 replies
- 0 kudos
I need to access the json file in the github repo from the databricks notebookI have a repo integrated with Databricks workspace. Â When I run %sh pwd ...
I need to access the json file in the github repo from the databricks notebookI have a repo integrated with Databricks workspace. When I run %sh pwd it returns this path /Workspace/Repos/chris@myemail/Repo/folder/test.json. I'm not able to access the...
- 1452 Views
- 2 replies
- 0 kudos
