- 8478 Views
- 4 replies
- 0 kudos
Resolved! Unable to call UDF inside the Spark SQL: RuntimeError: SparkSession should be create
Here is how I define the UDF inside the file udf_define.py:from pyspark.sql.functions import length, udf from pyspark.sql.types import IntegerType from pyspark.sql import SparkSession spark = SparkSession.builder.getOrCreate() def strlen(s): ret...
- 8478 Views
- 4 replies
- 0 kudos
- 7522 Views
- 3 replies
- 2 kudos
Problems with Azure Databricks
Hi,I want to use Databricks the first time, and am having many problems and confusions. Please help me resolve them.1. I created a free Databricks Community account on Azure and get error when creating the cluster/compute"Azure Quota Exceeded Excepti...
- 7522 Views
- 3 replies
- 2 kudos
- 2 kudos
you must have created some resources with public IP addresses in your azure subscription, ie: storage account, etc...Try to avoid using public IPs as much as possible to secure your tenant/subscription.try to find which of your Azure resources are us...
- 2 kudos
- 2541 Views
- 1 replies
- 1 kudos
Creation of logs in a file
Good afternoon.I am trying to add logs in the creation of my code. The issue is that I haven't yet found a way to write the logs to a separate file, rather than having them output to the terminal; I want them to be stored in a file (example.log).I ha...
- 2541 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @mac08_flo ,Use logging library. You can configure to log to terminal, to files etc.https://www.highlight.io/blog/5-best-python-logging-libraries
- 1 kudos
- 4840 Views
- 2 replies
- 0 kudos
Failed to read job commit marker error
Recently, we migrate from DBR 11.3 LTS ML to DBR 14.3 LTS ML. We are struggling on one data source where we consume parquet files. New data are appended every 30 minutes to that data source. The data are generated by Databricks notebook which runs on...
- 4840 Views
- 2 replies
- 0 kudos
- 0 kudos
Thanks @menotron from your reply!Interestingly, we have been using REFRESH TABLE command even before this issue and it worked well so far. However, now with new runtime, it doesn't work anymore. I should specify the code which we use. It actually fai...
- 0 kudos
- 4188 Views
- 7 replies
- 3 kudos
Delt Live Table Path/Directory help
Hello, I am working on a dlt pipeline and I've been facing an issue. I hope someone here can help me find a solution.My files are json in azure storage. These files are stored in dircctory like this ( blobName/FolderName/xx.csv).The folder name is li...
- 4188 Views
- 7 replies
- 3 kudos
- 2851 Views
- 5 replies
- 1 kudos
Data Engineering with Databricks 3.1.12 - Unable to run Classroom-Setup-01.2
Receiving the following error when attempting to run the classroom setup for lesson 1.2 of the Data Engineering with Databricks 3.1.12. This has been tested with multiple accounts, both admins and non-admins.Below is the error message I am receiving....
- 2851 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi @JR61276126 ,Since your workspace is deployed in azure with vent injection I assume it might be a network/firewall related issue. Could you check your driver logs also?
- 1 kudos
- 5619 Views
- 1 replies
- 1 kudos
Databricks Asset Bundle "Credential was not sent or was of an unsupported type"
I am working on setting up an asset bundle and it is failing when I try to validate the bundle. I am getting an error saying "Credential was not sent or was of an unsupported type for this API."I have a profile created and am using an access token t...
- 5619 Views
- 1 replies
- 1 kudos
- 1 kudos
I am having a similar issue, when trying to deploy my asset bundle.I ran the following:databricks auth login --host <hostname>I then was authenticated just fine, without issue. I then pointed to the relevant directory containing the asset bundle and ...
- 1 kudos
- 4053 Views
- 2 replies
- 0 kudos
How to properly implement incremental batching from Kinesis Data Streams
Hi, I implemented a job that should incrementally read all the available data from a Kinesis Data Stream and terminate afterwards. I schedule the job daily. The data retention period of the data stream is 7 days, i.e., there should be enough time to ...
- 4053 Views
- 2 replies
- 0 kudos
- 0 kudos
It seems like the issue might be caused by potential data loss in the Kinesis stream. Even though you're using checkpoints and specifying the "earliest" position, data can expire due to the 7-day retention period, especially if there's a delay in job...
- 0 kudos
- 6753 Views
- 3 replies
- 1 kudos
Resolved! Suddenly Getting Timeout Errors Across All Environments while waiting for Python REPL to start.
Hey - we currently have 4 environments spread out across separate workspaces, and as of Monday we've began to have transient failures in our DLT pipeline runs with the following error:"java.util.concurrent.TimeoutException: Timed out after 60 seconds...
- 6753 Views
- 3 replies
- 1 kudos
- 1 kudos
For anyone else who may be experiencing this issue - it seems to have been related to serverless compute for notebooks/workflows, which we had enabled for the account, but WERE NOT using for our DLT pipelines. After noticing references to serverless ...
- 1 kudos
- 7725 Views
- 14 replies
- 2 kudos
- 7725 Views
- 14 replies
- 2 kudos
- 2 kudos
Hi @varshini_reddy ,There is no option to stop all the other iterations when for each is running and one of the iterations failed.This is why the shared workaround, that will simply skip/fail all the next iterations without doing anything.You can fai...
- 2 kudos
- 4606 Views
- 9 replies
- 0 kudos
WHERE 1=0, Error message from Server
Hi ,I am getting this Error:WHERE 1=0, Error message from Server: Configuration db table is not available. I am using PySpark and JDBC connection. Please help on this.
- 4606 Views
- 9 replies
- 0 kudos
- 0 kudos
@szymon_dybczak Can you help us on this? Or could you provide a complete structure/steps how to connect with databricks using PySpark and JDBC step by step . Like initiate spark session then JDBC connection url then sql read all these in details.Also...
- 0 kudos
- 2237 Views
- 1 replies
- 1 kudos
azure pipeline databricks bundle deploy duplicating jobs
I am deploying an asset bundle using an azure pipeline.I use # Databricks Bundle Validate- bash: | databricks bundle validate -t $(BUNDLE_TARGET) displayName: 'Validate Asset Bundle' # Databricks Bundle Deploy- bash: | databricks bundle deploy...
- 2237 Views
- 1 replies
- 1 kudos
- 1 kudos
Hey! Same problem over here, tried upgrading to the latest version of the Databricks CLI but to no avail.I did find the issue on Github: https://github.com/databricks/cli/issues/1650
- 1 kudos
- 24138 Views
- 27 replies
- 20 kudos
Resolved! Databricks connect 14.3.2 SparkConnectGrpcException Not found any cached local relation withthe hash
Hi All,I am using Databricks Connect 14.3.2 with Databricks Runtime 14.3 LTS to execute the code below. The CSV file is only 7MB, the code runs without issues on Databricks Runtime 15+ clusters but consistently produces the error message shown below ...
- 24138 Views
- 27 replies
- 20 kudos
- 20 kudos
As a workaround, please try the following Spark configuration, which seems to have resolved the issue for me on both 14.3 LTS and 15.4 LTS.spark.conf.set("spark.sql.session.localRelationCacheThreshold", 64 * 1024 * 1024)
- 20 kudos
- 7209 Views
- 5 replies
- 3 kudos
Resolved! PARSE_EMPTY_STATEMENT error when trying to use spark.sql via Databricks Connect
I'm trying to use Databricks Connect to run queries on Delta Tables locally. However, SQL queries using spark.sql don't seem to work properly, even though spark.read.table works.>>> from databricks.connect import DatabricksSession>>> spark = Databric...
- 7209 Views
- 5 replies
- 3 kudos
- 3 kudos
Hi everyone! I am an engineer working on Databricks Connect. This error appears because of the incompatibility between the Serverless Compute and Databricks Connect versions. The current Serverless Compute release roughly corresponds to Databricks Ru...
- 3 kudos
- 3452 Views
- 4 replies
- 5 kudos
Create New Table, Infer Schema gives error: Invalid column type {colSchemaType}
I'm doing a the course 'Distributed Computing with Spark SQL' on Coursera, and need to create a table by uploading a csv file. That seems to work at first, but as soon as I check the box for 'Infer schema' for the preview table, I get the following m...
- 3452 Views
- 4 replies
- 5 kudos
- 5 kudos
Hi @lauracoursera ,In Databricks Community Edition I am getting the same error as you:The Community Edition is very limited - the UI is not updated to the newest version, it has old runtimes, missing features etc.My recommendation is to register a fr...
- 5 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
