- 3011 Views
- 3 replies
- 7 kudos
Spark context not implemented Error when using Databricks connect
I am developing an application using databricks connect and when I try to use VectorAssembler I get the Error sc is not none Assertion Error. is there a workaround for this ?
- 3011 Views
- 3 replies
- 7 kudos
- 7 kudos
Ran into exactly the same issue as @Łukasz1 After some googling, I found this SO post explaining the issue: later versions of databricks connect no longer support the SparkContext API. Our code is failing because the underlying library is trying to f...
- 7 kudos
- 53 Views
- 1 replies
- 1 kudos
Best Practices for Collaborative Notebook Development in Databricks
Hi everyone! I’m looking to learn more about effective strategies for collaborative development in Databricks notebooks. Since notebooks are often used by multiple data scientists, analysts, and engineers, managing collaboration efficiently is critic...
- 53 Views
- 1 replies
- 1 kudos
- 1 kudos
For version control, use this approach.Git Integration with Databricks ReposCore Features:Databricks Git Folders (Repos) provides native Git integration with visual UI and REST API access Supports all major providers: GitHub, GitLab, Azure DevOps, Bi...
- 1 kudos
- 2031 Views
- 4 replies
- 2 kudos
Resolved! Unable to Access Delta View from Azure Machine Learning via Delta Sharing – Is View Access Supported
Unable to Access Delta View from Azure Machine Learning via Delta Sharing – Is View Access Supported?I am able to access the tables but while accessing the view I am getting below error.Response from server: { 'details': [ { '@type': 'type.googleapis...
- 2031 Views
- 4 replies
- 2 kudos
- 2 kudos
View sharing is supported (launched GA) in Databricks. See https://docs.databricks.com/aws/en/delta-sharing/create-share#add-views-to-a-share. You likely need a workspace id override. Creating the recipient from a workspace with proper access and res...
- 2 kudos
- 143 Views
- 1 replies
- 0 kudos
GenAI experiment tracing does not render markdown images
When traces include base64 encoded images in Markdown, they do not render properly. This makes the analysis of traces including images difficult.Just for context, the same trace in other tracing tools like LangSmith renders as expected. An example of...
- 143 Views
- 1 replies
- 0 kudos
- 0 kudos
Thank you for the for the flag juandados! I will ping my product team to get a timeline for you.
- 0 kudos
- 683 Views
- 1 replies
- 1 kudos
AutoML Forecast fails when using feature_store_lookups with timestamp key
We are running AutoML Forecast on Databricks Runtime 15.4 ML LTS and 16.4 ML LTS, using a time series dataset with temporal covariates from the Feature Store (e.g. a corona_dummy feature). We use feature_store_lookups with lookup_key and timestamp_lo...
- 683 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @ostae911 , are you still facing this issue? It looks like your usage of the timestamp column is correct. It can be used as a primary key on the time series feature table. Is it possible that there are other duplicate columns between the training ...
- 1 kudos
- 1268 Views
- 3 replies
- 1 kudos
Resolved! Serving Endpoint Disappears After One Day
I'm encountering an issue where a serving endpoint I create disappears from the list of serving endpoints after a day. This has happened both when I created the endpoint from the Databricks UI and using the Databricks SDK.
- 1268 Views
- 3 replies
- 1 kudos
- 1 kudos
Hey @prashant_089 , what you are experiencing should not happen on its own except for some extremely outlying circumstanctes. IF YOU ARE USING Databricks Free Edition you shold ignore everything below. Here are some troubleshooting suggestions/tips: ...
- 1 kudos
- 2036 Views
- 3 replies
- 0 kudos
Resolved! Problem loading a pyfunc model in job run
Hi, I'm currently working on a automated job to predict forecasts using a notebook than work just fine when I run it manually, but keep failling when schedueled, here is my code: import mlflow # Load model as a PyFuncModel. loaded_model = mlflow.pyf...
- 2036 Views
- 3 replies
- 0 kudos
- 0 kudos
Hey AmineM! If your MLflow model loads fine in a Databricks notebook but fails in a scheduled job on serverless compute with an error like: TypeError: code() argument 13 must be str, not int the root cause is almost always a mismatch between the ...
- 0 kudos
- 823 Views
- 4 replies
- 2 kudos
Resolved! What is the most efficient way of running sentence-transformers on a Spark DataFrame column?
We're trying to run the bundled sentence-transformers library from SBert in a notebook running Databricks ML 16.4 on an AWS g4dn.2xlarge [T4] instance.However, we're experiencing out of memory crashes and are wondering what the optimal to run sentenc...
- 823 Views
- 4 replies
- 2 kudos
- 2 kudos
If you didn't get this to work with Pandas API on Spark, you might also try importing and instantiating the SentenceTransformer model inside the pandas UDF for proper distributed execution. Each executor runs code independently, and when Spark execut...
- 2 kudos
- 188 Views
- 1 replies
- 0 kudos
Inference Tables Empty
Hello,I have been using Databricks Free Platform for a while. Everything seems to work well. However, I've been trying to generate the payload from the deployed endpoint and I got always an empty inference table.When I check the configuration, I got ...
- 188 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @salesbrj ,Most probably this will be related to limitation in Free Edition. In limitations section I can see following entry:No custom models on GPU or batch inferencehttps://docs.databricks.com/aws/en/getting-started/free-edition-limitations
- 0 kudos
- 1273 Views
- 3 replies
- 1 kudos
Distributed SparkXGBRanker training: failed barrier ResultStage
I'm following a variation of the tutorial [here](https://assets.docs.databricks.com/_extras/notebooks/source/xgboost-pyspark-new.html) to train an `SparkXGBRanker` in distributed mode. However, the line:pipeline_model = pipeline.fit(data) Is throwing...
- 1273 Views
- 3 replies
- 1 kudos
- 1 kudos
You have already mentioned you did turn off autoscaling, please try the num_workers too Step 1: Disable Dynamic Resource Allocation: Use spark.dynamicAllocation.enabled = false Step 2: Configure num_workers to Match Fixed Resources After disabling dy...
- 1 kudos
- 6379 Views
- 7 replies
- 3 kudos
Unable to Use VectorAssembler in PySpark 3.5.0 Due to Whitelisting
Hi,I am currently using PySpark version 3.5.0 on my Databricks cluster. Despite setting the required configuration using the command: spark.conf.set("spark.databricks.ml.whitelist", "true"), I am still encountering an issue while trying to use the Ve...
- 6379 Views
- 7 replies
- 3 kudos
- 3 kudos
I also had this error trying to use ML on free edition. Is ML features working for free edition.
- 3 kudos
- 318 Views
- 2 replies
- 4 kudos
Resolved! Can't use pyspark bucketizer
As title suggests, I am struggling to use pyspark bucketizer as I repeatedly get the following error:File <command-8301298062763331>, line 4 2 from pyspark.ml.feature import Bucketizer 3 spark = SparkSession.builder.appName("test").getOrC...
- 318 Views
- 2 replies
- 4 kudos
- 4 kudos
Hi @wise_centipede ,In your Serverless compute select Environment Version: 4 and it will work With version below 4 I've got the same error as you:And when I've upgrade serverless environment ot version 4 it works as expected
- 4 kudos
- 764 Views
- 1 replies
- 0 kudos
Lakehouse monitoring generates broken queries
Hi everyone,I’m setting up Databricks Lakehouse Monitoring to track my model’s performance using an inference-regression monitor. I’ve completed all the required configuration and successfully launched my first monitoring run.The quality tables are g...
- 764 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @the_p_l ,I want to confirm that I understand your situation correctly. You mentioned that you are not adding any custom code to the deployed Lakehouse Monitoring setup, and you believe the issue is related to the inline comments generated during ...
- 0 kudos
- 1230 Views
- 2 replies
- 0 kudos
Issue Importing transformers Library on Databricks
I'm experiencing an issue when trying to import the "transformers" library in a Databricks notebook. The import statement causes the notebook to hang indefinitely without any error messages. The library works perfectly on my local machine using Anaco...
- 1230 Views
- 2 replies
- 0 kudos
- 0 kudos
@Deniz_Bilgin yeah some packages are not compatible with runtime. Use a Stable Version Try installing a known working version:%pip install transformers==4.41.2Dependency Issues:Conflicts with preinstalled libraries like urllib3 (e.g., version ...
- 0 kudos
- 207 Views
- 1 replies
- 0 kudos
Help Finding Course Notebook Machine Learning Practitioner Learning Plan
Hi,This is Vandna. I’m currently taking the Machine Learning Practitioner Learning Plan course, but I’m unable to locate the corresponding notebook. Could you please share the link to the notebook or guide me on where and how I can access it?Thank yo...
- 207 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Vandna-Shobran! The Machine Learning Practitioner Learning Plan modules are free self-paced and do not include hands-on labs. To access the labs, you would need to either: Enroll in the ILT (Instructor-Led Training) courses - This will grant y...
- 0 kudos
