Hi, we have several clusters that keep giving this error:Failure starting repl. Try detaching and re-attaching the notebook.All the investigation I've done points to this issue being related to the number of concurrent connections but we only have 1 ...
After some initial skepticism, Barker agreed to give Plinko https://plnkgame.com a try. The game was an instant hit with both the audience and the contestants. The concept was simple but exciting - players would drop a disc down a large pegboard, and...
I'm trying to create a task where the source is a Python script located in remote GitLab repo. I'm following the instructions HERE and this is how I have the task set up:However, no matter what path I specify all I get is the error below:Cannot read ...
Hi @ChingizK, The issue you are experiencing might be because you are starting your path with a /. According to the provided information, when you enter the relative path, you should not begin it with / or ./.
For example, if the absolute path for th...
Hi all,I ran a code to join 3 tables in Azure Databricks using SQL. When I run the code it is indicated "93 million rows read (1GB). It will be showing me " and does not go beyond this. Who knows what the issue could be?
Hi @space25 , The issue you're facing could be due to a variety of reasons. It's hard to pinpoint the exact cause without more details, but here are a few possibilities.
1. **Large Volume of Data**: The operation might be taking a long time due to t...
Hello All,I have successfully created a databricks account and went to login to the community edition with the exact same login credentials as my account, but it tells me that the email/password are invalid. I can login with these same exact credenti...
Hi @Arihant ,
Please look at this link related to the Community - Edition, which might solve your problem.
I appreciate your interest in sharing your Community-Edition query with us. However, at this time, we are not entertaining any Community-Edi...
Hi - I have tried my level best to go through both elasticsearch documentation as well as Databricks documentation to get an answer for my question - is it possible to connect to AWS elasticsearch of a different AWS account from Databricks? I did no...
Hi @niladri , It's possible to connect to AWS Elasticsearch of a different AWS account from Databricks.- The error is related to permissions, indicating the user or role lacks necessary access permissions.- To solve this, use AWS SDK boto3 to assume ...
I am hoping someone can help me remove the WHERE 1=0 that is constantly getting added onto the end of my Query (see below). Please let me know if I can provide more info here.This is running a notebook, in Azure Databricks against a cluster that has...
Hi @DanBrown , The WHERE 1=0 clause is being added to your query by the Spark SQL engine during the query planning phase. This is a common optimization technique used to create an empty DataFrame with the same schema as the original data source.
It'...
I need to retrieve the event logs of deleted All Purpose clusters of a certain workspace.databricks list API ({workspace_url}/api/2.0/clusters/list) provides me with the list of all active/terminated clusters but not the clusters that are deleted. I ...
Hi @RiyuLite, To retrieve the event logs of deleted All Purpose clusters without using the root account details, you can use Databricks audit logs. These logs record the activities in your workspace, allowing you to monitor detailed Databricks usage ...
Hey ,I am trying to fetch data from mongo and write to databricks table.I have read data from mongo using pymongo library, then flattened nested struct objects along with renaming columns(since there were few duplicates) and then writing to databrick...
Hi @Divyanshu ,
The error message "org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 12.0 failed 4 times, most recent failure: Lost task 2.3 in stage 12.0 (TID 53) (192.168.23.122 executor 0): org.apache.spark.SparkR...
Hi @Alex006 , No, a Delta Live Tables (DLT) pipeline does not use a single SparkSession for all notebooks. DLT evaluates and runs all code defined in notebooks but has a different execution model than a notebook 'Run all' command. You cannot rely on ...
Hi Team,Is there a way that we can add data manually to the tables that are generated by DLT?We have done a PoC using DLT for Sep 15 to current data. Now, that they are happy, they wanted the previous data from Synapse and put into Databricks.I can e...
Hi @Gilg, Yes, you can add data manually to the tables generated by DLT (Delta Live Tables).
However, it would be best to be careful not to directly modify, add, or delete Parquet data files in a Delta table, as this can lead to lost data or table c...
Hello!I am currently exploring the possibility of implementing incremental changes in our company's ETL pipeline and looking into Change Data Feed option. There are a couple of challenges I'm uncertain about.For instance, we have a piece of logic lik...
Hi @mike_engineer ,
- Use the Change Data Feed feature in Databricks to track row-level changes in a Delta table.- Change Data Feed records change events for all data written into the table, including row data and metadata.
- Use case scenarios: 1. ...
Hi Team,I have a DLT pipeline running in Production for quite some time now. When I check the pipeline, a couple of jobs took longer than expected. Usually, 1 job only took 10-15 minutes to complete with 2 to 3 mins to provision a resource. Then I ha...
Hi @Gilg, The issue you're experiencing with your DLT pipeline could be due to a couple of factors:
1. **Development Optimizations**: As per the Databricks release notes from September 7-13, 2021, new pipelines run in development mode by default. Thi...
I am trying to read data into a dataframe from Azure SQL DB, using jdbc. Here is the code I am using.driver = "com.microsoft.sqlserver.jdbc.SQLServerDriver"
database_host = "server.database.windows.net"
database_port = "1433"
database_name = "dat...
With the introduction of the Unity Catalog in databricks, many of us have become familiar with creating catalogs. However, did you know that the Unity Catalog also allows you to create foreign catalogs? You can register databases from the following s...
