- 9169 Views
- 8 replies
- 3 kudos
com.amazonaws.services.s3.model.AmazonS3Exception: The bucket is in this region: *** when using S3 Select
Hello,I have a cluster running in us-east-1 region.I hava a Spark job loading data in a DataFrame using s3select format on a bucket in eu-west-1 region.Access and Secret keys are encoded in URI s3a://$AccessKey:$SecretKey@bucket/path/to/dirJob fails ...
- 9169 Views
- 8 replies
- 3 kudos
- 3 kudos
Hello,I tried your suggestion by setting up the peering connection between the 2 VPC but issue remains the same.The error message The bucket is in this region: .... please use this region to retry the requestmakes me think that the root cause is not ...
- 3 kudos
- 16347 Views
- 1 replies
- 1 kudos
Unity Catalog Pricing
Hi All, I would like to understand the pricing model of the Unity Catalog. Earlier I remember there was some mention of the data lineage and a few other features that will have a cost associated with it. If that's true, what other features cost us? W...
- 16347 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Venkadeshwaran K​ ,All Unity Catalog features are provided at no charge to customers, provided they are using a Premium or Enterprise SKU.
- 1 kudos
- 5098 Views
- 3 replies
- 3 kudos
How to circumvent Py4JSecurityException for spark-nlp : Constructor public com.johnsnowlabs.nlp.***(java.lang.String) is not whitelisted.
Running into the following error on our company's cluster. py4j.security.Py4JSecurityException: Constructor public com.johnsnowlabs.nlp.DocumentAssembler(java.lang.String) is not whitelisted.For the following code(which is just tutorial code from the...
- 5098 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Kenan Spruill​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...
- 3 kudos
- 3434 Views
- 3 replies
- 3 kudos
Is it possible to access online feature store (Cosmos DB) outside databricks?
We are building an machine learning application with feature store enabled. Once the model is built, we are trying to move the model artifacts and deploy it in azure ml as online endpoint. Does it possible to access the online store in azure ml endpo...
- 3434 Views
- 3 replies
- 3 kudos
- 3 kudos
if you want databricks to use the feature store, which is in Cosmos DB, yes, it is possible https://learn.microsoft.com/en-us/azure/databricks/machine-learning/feature-store/online-feature-storessuppose you want to consume a future store in Databrick...
- 3 kudos
- 5101 Views
- 7 replies
- 0 kudos
Databricks not able to create table in minio bucket
Trying to create table in minio bucket using databricks.spark.sql("create database if not exists minio_db_1 managed location 's3a://my-bucket/minio_db_1'");I am passing the s3 configurations using spark context.access_key = 'XXXX'secret_key = 'XXXXXX...
- 5101 Views
- 7 replies
- 0 kudos
- 0 kudos
MANAGED LOCATION is for Unity Catalog. Please check if you are under the unity catalog, not under hive metastore. Additionally, with Unity, you are not using sc._jsc.hadoopConfiguration() etc. but just register storage credentials and external locati...
- 0 kudos
- 2194 Views
- 1 replies
- 0 kudos
Weekly Release Notes Recap Here’s a quick recap of the latest release notes updates from the past one week. Databricks platform release notesJanuary 1...
Weekly Release Notes RecapHere’s a quick recap of the latest release notes updates from the past one week.Databricks platform release notesJanuary 13 - 19, 2023Cluster policies now support limiting the max number of clusters per userYou can now use c...
- 2194 Views
- 1 replies
- 0 kudos
- 0 kudos
thanks for the details
- 0 kudos
- 3830 Views
- 3 replies
- 2 kudos
Cannot write Feature Table because of invalid access token
From a notebook I created a new feature store via:%sql CREATE DATABASE IF NOT EXISTS feature_store_ebp;Within that feature store I fill my table with:feature_store_name = "feature_store_ebp.table_1" try: fs.write_table( name=feature_stor...
- 3830 Views
- 3 replies
- 2 kudos
- 3386 Views
- 2 replies
- 0 kudos
I need to access the json file in the github repo from the databricks notebookI have a repo integrated with Databricks workspace. Â When I run %sh pwd ...
I need to access the json file in the github repo from the databricks notebookI have a repo integrated with Databricks workspace. When I run %sh pwd it returns this path /Workspace/Repos/chris@myemail/Repo/folder/test.json. I'm not able to access the...
- 3386 Views
- 2 replies
- 0 kudos
- 10014 Views
- 6 replies
- 2 kudos
Solution for - "PythonException: 'ModuleNotFoundError: No module named 'spacy'
I am actually trying to extract the adjective and noun phrases from the text column in spark data frame for which I've written the udf and applying on cleaned text column. However, I am getting this error.from pyspark.sql.functions import udffrom pys...
- 10014 Views
- 6 replies
- 2 kudos
- 2431 Views
- 2 replies
- 4 kudos
Implementation of best practices in authorization to a service application
we are working on authentication mechanism for our model service application using python framework fastAPI deployed on azure cloud, Need help on end to end auth mechanism(either through inbuild mechanism in python like jwt etc., or with azure).kindl...
- 2431 Views
- 2 replies
- 4 kudos
- 4 kudos
Could you please explain it a little more?For authentication, please refer: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/api/latest/authenticationhttps://learn.microsoft.com/en-us/azure/databricks/security/security-overview-azure#--au...
- 4 kudos
- 4049 Views
- 2 replies
- 2 kudos
Number of epochs/epoch loss widget not visible while training the model
I am training a N-BEATS forecasting model using darts library. After I define all my hyper parameters and execute the code to fit my model and have set the ''verbose'' parameter to true according to the documentation to show the progress of the train...
- 4049 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Mrinmoy Gupta​, what happens when you detach the notebook from the cluster (and optionally clear the state) and then rerun the code? I've seen this happen once and it was a solved by re-running the code
- 2 kudos
- 4226 Views
- 1 replies
- 1 kudos
Permission errors.
I have some permission erros when I want to modify some sql queries in SQL module. We are two colleauge working on one project so we build da data model. Sometimes we need to correct each other and access the SQL code but unfortunelty we dont have a...
- 4226 Views
- 1 replies
- 1 kudos
- 1 kudos
you not worry here , give permission as a viewer only , or else they will use your creds there This doc will be helpful - https://docs.databricks.com/security/access-control/query-acl.htmlThanksAviral Bhardwaj
- 1 kudos
- 20537 Views
- 8 replies
- 0 kudos
Resolved! Inheritance model in Unity Catalog is not working as per documentation.
As per the documentation "Securable objects in Unity Catalog are hierarchical and privileges are inherited downward. The highest level object that privileges are inherited from is the catalog". Executed following statement "GRANT SELECT ON CATALOG uc...
- 20537 Views
- 8 replies
- 0 kudos
- 2755 Views
- 1 replies
- 0 kudos
Why is mounting storage no longer considered best practice?
As the title describes. I think it's really nice to work with mounted storage, but I've typically had an IaC team take care of setting it up. Now I'm not that lucky. Why is it no longer best practice? Security reasons?
- 2755 Views
- 1 replies
- 0 kudos
- 0 kudos
I think so, mount is like a local storage, other users in the same workspace will have the access to any mounted storage too.Access Azure Data Lake Storage Gen2 and Blob Storage | Databricks on AWS
- 0 kudos
- 8296 Views
- 4 replies
- 0 kudos
How to fix "WARNING mlflow.utils.environment" when run mlflow in Databricks?
I'm running the following python code from one of the databricks training materials. import mlflow import mlflow.spark from pyspark.ml.regression import LinearRegression from pyspark.ml.feature import VectorAssembler from pyspark.ml import Pipeline f...
- 8296 Views
- 4 replies
- 0 kudos
- 0 kudos
I've encountered the same warning when running this notebook from DA.https://github.com/databricks-academy/scalable-machine-learning-with-apache-spark-english/blob/published/ML%2002%20-%20Linear%20Regression%20I.pyI've managed to get rid of that war...
- 0 kudos
