Running almost any notebook with a merge statement in Databricks with DBR 14.3 I get the following error and the notebook exists:"Fatal error: The Python kernel is unresponsive."I would provide more code, but like I said, it is pretty much anything w...
Hi @ac0 , Hope you are doing well!
The "Fatal error: The Python kernel is unresponsive" error generally means the Jupyter Kernel is still running but is not responsive within a specific time frame.
Please try to increase the timeout by specifying co...
I have the Databricks VS code extension setup to develop and run jobs remotely. (with Databricks Connect).I enjoy working on notebooks within the native Databricks workspace, especially for exploratory work because I can execute blocks of code step b...
Need to understand this i got 2 sources which currently writes to the Single S3 bucket in csv and json format daily is it possible to ingest both the file types in a single build to ingest in Databricks, file size <300 mb ?or Alternately i should cre...
Hey there! Thanks a bunch for being part of our awesome community! We love having you around and appreciate all your questions. Take a moment to check out the responses – you'll find some great info. Your input is valuable, so pick the best solution...
hello,I have a spark StructuredStreaming job : the source is a kafka topic in json.it work find with databricks 14.2, but when a change to 14.3LTS, I have a NPE in CreateJacksonParser:Caused by: NullPointerException:
at org.apache.spark.sql.catalys...
Hi ,thank you for your quick reply,i found the problem : val newSchema = spark.read.json(df.select("data").as[String]).schemaif "data" have 1 value to null, in 14.2 it work, but with 14.3LTS this function return a NPEI don't know if it is a bug
Hi Team,We recently migrated event files from our previous S3 bucket to a new one. While utilizing Autoloader for batch ingestion, we've encountered an issue where the migrated data is being processed as new events. This leads to duplicate records in...
Hey there! Thanks a bunch for being part of our awesome community! We love having you around and appreciate all your questions. Take a moment to check out the responses – you'll find some great info. Your input is valuable, so pick the best solution...
Q1. My source is SQL server tables, I want to identify only latest records(incremental rows) and load those into BRNZE layer. Instead of full load to ADLS, we want to capture only incremental rows and load into ADLS for further processing. NOTE: Prob...
Hi @Govind3331, Handling incremental data without a primary key or date column can be challenging, but there are some strategies you can consider.
Row Number Approach:
You can use the ROW_NUMBER() function to assign a unique number to each row bas...
We're using Databricks to incrementally extract data from SQL Server tables into S3. The data contains a timestamp column. We need a place to store the maximum retrieved timestamp per table so it can retrieved during the next run.Does Databricks cont...
Hi @demost11, Databricks provides several options for managing state or storing metadata. Databricks Secrets:Databricks allows you to create and manage secrets, which are key-value pairs storing secret material (such as API keys, passwords, etc.). E...
In spark UI, I can see the application running with the application ID, from this spark UI, could I able to see the which notebook is running with that applications is this possible?I am interested in learning more about the jobs, stage how it works ...
Hey there! Thanks a bunch for being part of our awesome community! We love having you around and appreciate all your questions. Take a moment to check out the responses – you'll find some great info. Your input is valuable, so pick the best solution...
Hi1: I am reading a parquet file from AWS s3 storage using spark.read.parquet(<s3 path>) 2: An autoloader job has been configured to load this data into a external delta table.3: But before loading into this autoloader I need to do some typecasting o...
Hi @srinivas_001 , When working with Parquet files in Spark, you can read the file, perform typecasting on the columns, and then write the transformed data directly to your external Delta table.
Let’s break down the steps:
Read the Parquet file:...
Hello.I'm getting a cluster validation error while trying to deploy a DLT pipeline via DAB.See attached screenshots for config and error.Hoping someone has run into this before and can guide me.Thanks.
Hi @ahab, Deploying a Delta Live Tables (DLT) pipeline via Databricks can sometimes be tricky, especially when dealing with cluster validation errors. Let’s troubleshoot this together. Cluster Validation Error:When encountering a cluster validation e...
Hi, Rencently, I am seeing issue Could not reach driver of cluster <some_id> with my structure streaming job when migrating to unity catalog and found this when checking the traceback:Traceback (most recent call last):File "/databricks/python_shell/...
@Latonya86Dodson , Thank you for the reply. I have done a test, and it seems that double the memory of the driver cluster and change to use a instance with bigger memory works for this issue. However I do question why is this happen after I swap to p...
When I use the current "result table" option it does not show the table results. This occurs when running SQL commands and the display() function for DataFrames.It is not linked to a Databricks runtime, since it occurs on all runtimes. I am not allow...
I followed steps listed in this article.After creating and validation of bundle with default template, during deployment using this command:databricks bundle deploy -t dev --profile zzI get this message:Building mySecPrj...Error: build failed mySecPr...
I have a Databricks cluster with 10.4 runtime, when I configure databricks-connect configure I put all the information needed and using the default port 15001, databricks-connect test works.But changing the port to 443 does not work, I tried to do a ...
@daniel_sahal Thank you for the reply, indeed port 443 is used by a lot of applucations and could be problematic. But I also tried port `15002` and it didn't work. No other ports than default one works
How do I contact billing support? I am billed through AWS Marketplace and noticed last month the SQL Pro discount is not being reflected in my statement.
Hi, could anybody provide a contact email? I have sent emails to many contacts described in the support page here and in AWS, but no response from any channel. My problem is that databricks charged me by the resources used during a free trial, what i...
