Hi community, I am getting below warning when I try using pyspark code for some of my use-cases using databricks-connect. Is this a critical warning, and any idea what does it mean?Logs: WARN DatabricksConnectConf: Could not parse /root/.databricks-c...
Hi, @Surajv, The warning you’re encountering is related to using Databricks Connect with PySpark.
Databricks Connect: Databricks Connect is a Python library that allows you to connect your local development environment to a Databricks cluster. I...
Hi everyone,I'm currently facing an issue with handling a large amount of data using the Databricks API. Specifically, I have a query that returns a significant volume of data, sometimes resulting in over 200 chunks.My initial approach was to retriev...
Hi @rafal_walisko, Handling large volumes of data using the Databricks API can indeed be challenging, especially when dealing with numerous chunks.
Let’s explore some strategies that might help you optimize your approach:
Rate Limits and Paral...
Hi team,This is how I connect to Snowflake from Jupyter Notebook:import snowflake.connector
snowflake_connection = snowflake.connector.connect(
authenticator='externalbrowser',
user='U1',
account='company1.us-east-1',
database='db1',...
Hi @ymt, It seems you’ve encountered an issue while connecting to Snowflake from your Databricks Notebook.
The error message you received is:
ImportError: cannot import name 'NamedTuple' from 'typing_extensions' (/databricks/python/lib/python3.9/s...
Hello I am trying to run the SparkXGBoostRegressor and I am getting the following error:SpoilerPy4JError: An error occurred while calling o992.resourceProfileManager. Trace: py4j.security.Py4JSecurityException: Method public org.apache.spark.resource...
Hi @rahuja, The error you’re encountering might be related to the interaction between PySpark and XGBoost.
Let’s explore some potential solutions:
PySpark Version Compatibility:
Ensure that your PySpark version is compatible with the XGBoost vers...
I'm trying to set up a workflow in databricks and I need my job parameter to get the date and time. I see in the documentation there's some options for dynamic values.I'm trying to use this one: {{job.start_time.[argument]}}For the "argument" there, ...
Hi community, When I use pyspark rdd related functions in my environment using databricks connect, I get below error: Databricks cluster version: 12.2. `RuntimeError: Python in worker has different version 3.9 than that in driver 3.10, PySpark cannot...
Got it. As a side note, I tried above methods, but the error persisted, hence upon reading docs again, there was this statement: You must install Python 3 on your development machine, and the minor version of your client Python installation must be t...
Hello,I've recently embarked on integrating Splunk with Databricks. My aim is to efficiently ingest data from Splunk into Databricks. While I've reviewed the available documentation on Splunk Integration, it primarily covers basic information. Howeve...
I have a scheduled task running in workflow.Task 1 computes some parameters then these are picked up by a dependent reporting task: Task 2.I want Task 2 to report "Failure" if Task 1 fails. Yet creating a dependency in workflows means that Task 2 wil...
Hi @sharpbetty , Any suggestions how I can keep the parameter sharing and dependency from Task 1 to Task 2, yet also allow Task 2 to fire even on failure of Task 1?Setup:Task 2 dependent on Task1 Challenge: To Fire Task 2 even on Task 1 FailureSoluti...
Whenever I try to open my company file over a network or multi-user mode, I keep getting QB Desktop Error 6000 and something after that. The error messages on my screen vary every time I attempt to access the data file. I cannot understand the error,...
Hi, @faithlawrence98 and @judithphillips5 I appreciate you both taking the time to share your expertise. This is a well-written and insightful post. Keep up the great work!Thanks Regards!Larson Kristen
I just learnt, the above is a LEGACY support and hence must not be used. This isn't supported syntax, so there would be a lot of restrictions on the usage of this.
Internally it is just a view and hence we should go for create temp view instead.
I k...
Context:I've developed a DLT (Data Lifecycle Tool) pipeline where I create several temporary tables. Initially, when I ran these tables individually in separate notebooks, they functioned correctly within the DLT framework.However, after merging the ...
Hi All,Looking for suggestions to see if it is possible to control users via Azure AD (outside of Azure Databricks). As i want to create a new users in Azure and then I want to give RBAC to individual users and rather than control their permissions f...
I am getting an "INTERNAL_ERROR" on a databricks job submitted through the API. Which says:"Run result unavailable: run failed with error message All access to AWS S3 resource has been disabled"However, when I click on the notebook created by the job...
Hi @jvk, The “INTERNAL_ERROR” you’re encountering in your Databricks job, along with the message “Run result unavailable: run failed with error message All access to AWS S3 resource has been disabled,” indicates that there’s an issue related to acces...
I am looking for some help on getting databricks cluster metrics such as memory utilization, CPU utilization, memory swap utilization, free file system using REST API.I am trying it in postman using databricks token and with my Service Principal bear...
Hi, I'm trying to work on VS code remotely on my machine instead of using the Databricks environment on my browser. I have went through documentation to set up the Databricks. extension and also setup Databricks Connect but don't feel like they work ...
