- 2350 Views
- 3 replies
- 3 kudos
Is it possible to access online feature store (Cosmos DB) outside databricks?
We are building an machine learning application with feature store enabled. Once the model is built, we are trying to move the model artifacts and deploy it in azure ml as online endpoint. Does it possible to access the online store in azure ml endpo...
- 2350 Views
- 3 replies
- 3 kudos
- 3 kudos
if you want databricks to use the feature store, which is in Cosmos DB, yes, it is possible https://learn.microsoft.com/en-us/azure/databricks/machine-learning/feature-store/online-feature-storessuppose you want to consume a future store in Databrick...
- 3 kudos
- 3464 Views
- 7 replies
- 0 kudos
Databricks not able to create table in minio bucket
Trying to create table in minio bucket using databricks.spark.sql("create database if not exists minio_db_1 managed location 's3a://my-bucket/minio_db_1'");I am passing the s3 configurations using spark context.access_key = 'XXXX'secret_key = 'XXXXXX...
- 3464 Views
- 7 replies
- 0 kudos
- 0 kudos
MANAGED LOCATION is for Unity Catalog. Please check if you are under the unity catalog, not under hive metastore. Additionally, with Unity, you are not using sc._jsc.hadoopConfiguration() etc. but just register storage credentials and external locati...
- 0 kudos
- 1329 Views
- 1 replies
- 0 kudos
Weekly Release Notes Recap Here’s a quick recap of the latest release notes updates from the past one week. Databricks platform release notesJanuary 1...
Weekly Release Notes RecapHere’s a quick recap of the latest release notes updates from the past one week.Databricks platform release notesJanuary 13 - 19, 2023Cluster policies now support limiting the max number of clusters per userYou can now use c...
- 1329 Views
- 1 replies
- 0 kudos
- 0 kudos
thanks for the details
- 0 kudos
- 2806 Views
- 3 replies
- 2 kudos
Cannot write Feature Table because of invalid access token
From a notebook I created a new feature store via:%sql CREATE DATABASE IF NOT EXISTS feature_store_ebp;Within that feature store I fill my table with:feature_store_name = "feature_store_ebp.table_1" try: fs.write_table( name=feature_stor...
- 2806 Views
- 3 replies
- 2 kudos
- 2174 Views
- 2 replies
- 0 kudos
I need to access the json file in the github repo from the databricks notebookI have a repo integrated with Databricks workspace. Â When I run %sh pwd ...
I need to access the json file in the github repo from the databricks notebookI have a repo integrated with Databricks workspace. When I run %sh pwd it returns this path /Workspace/Repos/chris@myemail/Repo/folder/test.json. I'm not able to access the...
- 2174 Views
- 2 replies
- 0 kudos
- 7466 Views
- 6 replies
- 2 kudos
Solution for - "PythonException: 'ModuleNotFoundError: No module named 'spacy'
I am actually trying to extract the adjective and noun phrases from the text column in spark data frame for which I've written the udf and applying on cleaned text column. However, I am getting this error.from pyspark.sql.functions import udffrom pys...
- 7466 Views
- 6 replies
- 2 kudos
- 1669 Views
- 2 replies
- 4 kudos
Implementation of best practices in authorization to a service application
we are working on authentication mechanism for our model service application using python framework fastAPI deployed on azure cloud, Need help on end to end auth mechanism(either through inbuild mechanism in python like jwt etc., or with azure).kindl...
- 1669 Views
- 2 replies
- 4 kudos
- 4 kudos
Could you please explain it a little more?For authentication, please refer: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/api/latest/authenticationhttps://learn.microsoft.com/en-us/azure/databricks/security/security-overview-azure#--au...
- 4 kudos
- 2460 Views
- 2 replies
- 2 kudos
Number of epochs/epoch loss widget not visible while training the model
I am training a N-BEATS forecasting model using darts library. After I define all my hyper parameters and execute the code to fit my model and have set the ''verbose'' parameter to true according to the documentation to show the progress of the train...
- 2460 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Mrinmoy Gupta​, what happens when you detach the notebook from the cluster (and optionally clear the state) and then rerun the code? I've seen this happen once and it was a solved by re-running the code
- 2 kudos
- 2808 Views
- 1 replies
- 1 kudos
Permission errors.
I have some permission erros when I want to modify some sql queries in SQL module. We are two colleauge working on one project so we build da data model. Sometimes we need to correct each other and access the SQL code but unfortunelty we dont have a...
- 2808 Views
- 1 replies
- 1 kudos
- 1 kudos
you not worry here , give permission as a viewer only , or else they will use your creds there This doc will be helpful - https://docs.databricks.com/security/access-control/query-acl.htmlThanksAviral Bhardwaj
- 1 kudos
- 13228 Views
- 8 replies
- 0 kudos
Resolved! Inheritance model in Unity Catalog is not working as per documentation.
As per the documentation "Securable objects in Unity Catalog are hierarchical and privileges are inherited downward. The highest level object that privileges are inherited from is the catalog". Executed following statement "GRANT SELECT ON CATALOG uc...
- 13228 Views
- 8 replies
- 0 kudos
- 1881 Views
- 1 replies
- 0 kudos
Why is mounting storage no longer considered best practice?
As the title describes. I think it's really nice to work with mounted storage, but I've typically had an IaC team take care of setting it up. Now I'm not that lucky. Why is it no longer best practice? Security reasons?
- 1881 Views
- 1 replies
- 0 kudos
- 0 kudos
I think so, mount is like a local storage, other users in the same workspace will have the access to any mounted storage too.Access Azure Data Lake Storage Gen2 and Blob Storage | Databricks on AWS
- 0 kudos
- 6023 Views
- 4 replies
- 0 kudos
How to fix "WARNING mlflow.utils.environment" when run mlflow in Databricks?
I'm running the following python code from one of the databricks training materials. import mlflow import mlflow.spark from pyspark.ml.regression import LinearRegression from pyspark.ml.feature import VectorAssembler from pyspark.ml import Pipeline f...
- 6023 Views
- 4 replies
- 0 kudos
- 0 kudos
I've encountered the same warning when running this notebook from DA.https://github.com/databricks-academy/scalable-machine-learning-with-apache-spark-english/blob/published/ML%2002%20-%20Linear%20Regression%20I.pyI've managed to get rid of that war...
- 0 kudos
- 2763 Views
- 3 replies
- 3 kudos
Databricks AutoML (Forecasting) Python SDK for Model Serving
I am using Databricks AutoML ( Python SDK) to forecast bed occupancy. (Actually, Databricks used MLflow experiments for AutoML run). After training with different iterations, I registered the best model in the Databricks Model registry. Now I am tryi...
- 2763 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi, It can be a bug if the python version is 3.9.5 and still the error is on compatibility. Could you please raise a support case to look into it further?
- 3 kudos
- 3807 Views
- 3 replies
- 2 kudos
How to solve cluster break down due to GC when training a pyspark.ml Random Forest
I am trying to train and optimize a random forest. At first the cluster handles the garbage collection fine, but after a couple of hours the cluster breaks down as Garbage Collection has gone up significantly.The train_df has a size of 6,365,018 reco...
- 3807 Views
- 3 replies
- 2 kudos
- 2 kudos
The cache is expensive and wants to save that data to memory and disk (id there is no more space left in memory). I know that, in theory, it should improve, but it can make things worse. I would just putscaled_train_data = pipeline_data.transform(tra...
- 2 kudos
- 1503 Views
- 2 replies
- 3 kudos
Resolved! Can i change the Managed Mlflow to work with a postgresql server?
We are using the managed mlflow, but we want to access the metadata of the models and show it in another application. There is already a server that I can query?Can I re-create/configure the databricks workspace to make the managed mlflow use a post...
- 1503 Views
- 2 replies
- 3 kudos
- 3 kudos
Ideas which I have is:periodically export/import mlflow models and experiments https://github.com/mlflow/mlflow-export-import#why-use-mlflow-export-importget metadata through API https://docs.databricks.com/dev-tools/api/latest/mlflow.html#operation/...
- 3 kudos
