- 2488 Views
- 1 replies
- 1 kudos
Error on pandas udf usage in databricks, sc.broadcasting random forest loaded from Kedro MLFlow Logger DataSet, cannot pickle '_thread.RLock' object
I'm trying to broadcast a Random forest (sklearn 1.2.0) recently loaded from mlflow, and using Pandas UDF to predict a model.​However, the same code works perfectly on Spark 2.4 + our OnPrem cluster.​I thought it was due to Spark 2.4 to 3 changes, an...
- 2488 Views
- 1 replies
- 1 kudos
- 1 kudos
Anyone?
- 1 kudos
- 2926 Views
- 5 replies
- 1 kudos
Latest Blog PostsJanuary 13 - 20Â Did you get a chance to look at the most recent blog posts? Here are some happening content from the past week that i...
Latest Blog PostsJanuary 13 - 20Did you get a chance to look at the most recent blog posts? Here are some happening content from the past week that is worth the read. What’s New With SQL User-Defined Functions In this blog, we describe several enhanc...
- 2926 Views
- 5 replies
- 1 kudos
- 1 kudos
Thanks @Sujitha Ramamoorthy​ , for sharing with the community these are worth reading and insightful.
- 1 kudos
- 7227 Views
- 8 replies
- 3 kudos
com.amazonaws.services.s3.model.AmazonS3Exception: The bucket is in this region: *** when using S3 Select
Hello,I have a cluster running in us-east-1 region.I hava a Spark job loading data in a DataFrame using s3select format on a bucket in eu-west-1 region.Access and Secret keys are encoded in URI s3a://$AccessKey:$SecretKey@bucket/path/to/dirJob fails ...
- 7227 Views
- 8 replies
- 3 kudos
- 3 kudos
Hello,I tried your suggestion by setting up the peering connection between the 2 VPC but issue remains the same.The error message The bucket is in this region: .... please use this region to retry the requestmakes me think that the root cause is not ...
- 3 kudos
- 13997 Views
- 1 replies
- 1 kudos
Unity Catalog Pricing
Hi All, I would like to understand the pricing model of the Unity Catalog. Earlier I remember there was some mention of the data lineage and a few other features that will have a cost associated with it. If that's true, what other features cost us? W...
- 13997 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Venkadeshwaran K​ ,All Unity Catalog features are provided at no charge to customers, provided they are using a Premium or Enterprise SKU.
- 1 kudos
- 4226 Views
- 3 replies
- 3 kudos
How to circumvent Py4JSecurityException for spark-nlp : Constructor public com.johnsnowlabs.nlp.***(java.lang.String) is not whitelisted.
Running into the following error on our company's cluster. py4j.security.Py4JSecurityException: Constructor public com.johnsnowlabs.nlp.DocumentAssembler(java.lang.String) is not whitelisted.For the following code(which is just tutorial code from the...
- 4226 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Kenan Spruill​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...
- 3 kudos
- 2831 Views
- 3 replies
- 3 kudos
Is it possible to access online feature store (Cosmos DB) outside databricks?
We are building an machine learning application with feature store enabled. Once the model is built, we are trying to move the model artifacts and deploy it in azure ml as online endpoint. Does it possible to access the online store in azure ml endpo...
- 2831 Views
- 3 replies
- 3 kudos
- 3 kudos
if you want databricks to use the feature store, which is in Cosmos DB, yes, it is possible https://learn.microsoft.com/en-us/azure/databricks/machine-learning/feature-store/online-feature-storessuppose you want to consume a future store in Databrick...
- 3 kudos
- 4119 Views
- 7 replies
- 0 kudos
Databricks not able to create table in minio bucket
Trying to create table in minio bucket using databricks.spark.sql("create database if not exists minio_db_1 managed location 's3a://my-bucket/minio_db_1'");I am passing the s3 configurations using spark context.access_key = 'XXXX'secret_key = 'XXXXXX...
- 4119 Views
- 7 replies
- 0 kudos
- 0 kudos
MANAGED LOCATION is for Unity Catalog. Please check if you are under the unity catalog, not under hive metastore. Additionally, with Unity, you are not using sc._jsc.hadoopConfiguration() etc. but just register storage credentials and external locati...
- 0 kudos
- 1653 Views
- 1 replies
- 0 kudos
Weekly Release Notes Recap Here’s a quick recap of the latest release notes updates from the past one week. Databricks platform release notesJanuary 1...
Weekly Release Notes RecapHere’s a quick recap of the latest release notes updates from the past one week.Databricks platform release notesJanuary 13 - 19, 2023Cluster policies now support limiting the max number of clusters per userYou can now use c...
- 1653 Views
- 1 replies
- 0 kudos
- 0 kudos
thanks for the details
- 0 kudos
- 3354 Views
- 3 replies
- 2 kudos
Cannot write Feature Table because of invalid access token
From a notebook I created a new feature store via:%sql CREATE DATABASE IF NOT EXISTS feature_store_ebp;Within that feature store I fill my table with:feature_store_name = "feature_store_ebp.table_1" try: fs.write_table( name=feature_stor...
- 3354 Views
- 3 replies
- 2 kudos
- 2684 Views
- 2 replies
- 0 kudos
I need to access the json file in the github repo from the databricks notebookI have a repo integrated with Databricks workspace. Â When I run %sh pwd ...
I need to access the json file in the github repo from the databricks notebookI have a repo integrated with Databricks workspace. When I run %sh pwd it returns this path /Workspace/Repos/chris@myemail/Repo/folder/test.json. I'm not able to access the...
- 2684 Views
- 2 replies
- 0 kudos
- 8718 Views
- 6 replies
- 2 kudos
Solution for - "PythonException: 'ModuleNotFoundError: No module named 'spacy'
I am actually trying to extract the adjective and noun phrases from the text column in spark data frame for which I've written the udf and applying on cleaned text column. However, I am getting this error.from pyspark.sql.functions import udffrom pys...
- 8718 Views
- 6 replies
- 2 kudos
- 1944 Views
- 2 replies
- 4 kudos
Implementation of best practices in authorization to a service application
we are working on authentication mechanism for our model service application using python framework fastAPI deployed on azure cloud, Need help on end to end auth mechanism(either through inbuild mechanism in python like jwt etc., or with azure).kindl...
- 1944 Views
- 2 replies
- 4 kudos
- 4 kudos
Could you please explain it a little more?For authentication, please refer: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/api/latest/authenticationhttps://learn.microsoft.com/en-us/azure/databricks/security/security-overview-azure#--au...
- 4 kudos
- 3196 Views
- 2 replies
- 2 kudos
Number of epochs/epoch loss widget not visible while training the model
I am training a N-BEATS forecasting model using darts library. After I define all my hyper parameters and execute the code to fit my model and have set the ''verbose'' parameter to true according to the documentation to show the progress of the train...
- 3196 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Mrinmoy Gupta​, what happens when you detach the notebook from the cluster (and optionally clear the state) and then rerun the code? I've seen this happen once and it was a solved by re-running the code
- 2 kudos
- 3706 Views
- 1 replies
- 1 kudos
Permission errors.
I have some permission erros when I want to modify some sql queries in SQL module. We are two colleauge working on one project so we build da data model. Sometimes we need to correct each other and access the SQL code but unfortunelty we dont have a...
- 3706 Views
- 1 replies
- 1 kudos
- 1 kudos
you not worry here , give permission as a viewer only , or else they will use your creds there This doc will be helpful - https://docs.databricks.com/security/access-control/query-acl.htmlThanksAviral Bhardwaj
- 1 kudos
- 16614 Views
- 8 replies
- 0 kudos
Resolved! Inheritance model in Unity Catalog is not working as per documentation.
As per the documentation "Securable objects in Unity Catalog are hierarchical and privileges are inherited downward. The highest level object that privileges are inherited from is the catalog". Executed following statement "GRANT SELECT ON CATALOG uc...
- 16614 Views
- 8 replies
- 0 kudos
