- 2171 Views
- 1 replies
- 0 kudos
Silent failure in DataFrameWriter when loading data to Redshift
Context:I'm using DataFrameWriter to load the dataSet into the Redshift. DataFrameWriter writes the dataSet to S3, and loads data from S3 to Redshift by issuing the Redshift copy command. Issue:In frequently we are observing, the data is present in t...
- 2171 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Kishorekumar Somasundaram Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 0 kudos
- 3050 Views
- 1 replies
- 0 kudos
Getting JsonParseException: Unexpected character ('<' (code 60))
I have a scheduled job that is executed using a notebook. Within one of the notebook cells, there is a check to determine if a table exists. However, even when the table does exist, it incorrectly identifies it as non-existent and proceeds to execut...
- 3050 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Mahesh Chahare Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 0 kudos
- 2731 Views
- 1 replies
- 1 kudos
Having issue trying to download a csv file from a website using FireFox Selenium.
Hi, When I clicked on the download button from a website thru Firefox selenium using element.click(), and the download destination is being set as Azure datalake storage. Then, after the download started, those .csv and .csv.part files never gotten m...
- 2731 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Brandon Lim Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 1 kudos
- 1266 Views
- 0 replies
- 2 kudos
Your guide to data and AI governance talks at the Data + AI Summit 2023
The highly anticipated Data+AI summit is just around the corner. Are you curious about redefining your data governance strategy in the era of LLMs and generative AI? Look no further! This year's Data+AI summit offers a dynamic lineup of keynotes, dem...
- 1266 Views
- 0 replies
- 2 kudos
- 9378 Views
- 3 replies
- 0 kudos
Parquet column cannot be converted. Column: [Rainfall_Value], Expected: DoubleType, Found: INT64
df.printSchema()root |-- Device_ID: string (nullable = true) |-- Location: string (nullable = true) |-- Latitude: double (nullable = true) |-- Longitude: double (nullable = true) |-- DateTime: timestamp (nullable = true) |-- Rainfall_Value: double (n...
- 9378 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @THIAM HUAT TAN , The issue is because the schema defined for the column "Rainfall_Value" is of DoubleType and the values present in the data frame are of Integer type. This could be because of one or multiple values. Depending on the data, you ...
- 0 kudos
- 7279 Views
- 1 replies
- 0 kudos
StreamingQueryException: Read timed out // Reading from delta share'd dataset
I have a workspace in GCP that's reading from a delta-shared dataset hosted in S3. When trying to run a very basic DLT pipeline, I'm getting the below error. Any help would be awesome!Code:import dlt @dlt.table def fn(): return (spark.readStr...
- 7279 Views
- 1 replies
- 0 kudos
- 0 kudos
@Charlie You :The error message you're encountering suggests a timeout issue when reading from the Delta-shared dataset hosted in S3. There are a few potential reasons and solutions you can explore:Network connectivity: Verify that the network conne...
- 0 kudos
- 1519 Views
- 1 replies
- 0 kudos
If we do z-order on a particular column will delta log stats collection be affected?
Let's assume a table contains more than 40 columns, now we know it automatically collects stat for the first 32 columns. If we run a z-order on a particular column(let's say column 1), then will the log file collect stats for all the 32 columns or wi...
- 1519 Views
- 1 replies
- 0 kudos
- 0 kudos
@Shubhadip Ghosh : Hope this helps. In Delta Lake, when you perform Z-Ordering on a particular column, it reorganizes the data within the files based on the values of that column. However, Z-Ordering itself does not directly affect the statistics co...
- 0 kudos
- 1521 Views
- 1 replies
- 0 kudos
Will consecutive delete insert affect z-ordering?
Let's say there is a delta table with a date field as its partition. In a table where condition, we delete all the rows according to the division. The data is currently being inserted into the same date field. If we do a z-order after inserting the d...
- 1521 Views
- 1 replies
- 0 kudos
- 0 kudos
@Shubhadip Ghosh :In Delta Lake, when you perform a delete operation on a table, it doesn't physically remove the data from the files. Instead, it marks the affected rows for deletion by adding a tombstone marker to the Delta transaction log. This e...
- 0 kudos
- 1878 Views
- 0 replies
- 1 kudos
Presenting top 3 members who contributed to Community last week between 11th June-17th June- @Tyler Heflin @Werner Stinckens and @Bharathan K ...
Presenting top 3 members who contributed to Community last week between 11th June-17th June- @Tyler Heflin @Werner Stinckens and @Bharathan K We would like to express our gratitude for your participation and dedication in the Databricks Commun...
- 1878 Views
- 0 replies
- 1 kudos
- 62762 Views
- 8 replies
- 8 kudos
Resolved! Failure starting repl
Hi, we have several clusters that keep giving this error:Failure starting repl. Try detaching and re-attaching the notebook.All the investigation I've done points to this issue being related to the number of concurrent connections but we only have 1 ...
- 62762 Views
- 8 replies
- 8 kudos
- 8 kudos
@Aviral Bhardwaj thanks, this seemed to fix the issue, we had an innit script that was potentially conflicting with UI set libraries (in cluster settings).
- 8 kudos
- 3431 Views
- 4 replies
- 2 kudos
Resolved! Unable to lunch cluster in Databricks as my azure subscription has been disabled
Azure subscription- disabledDatabricks subscription- free trial 13 day leftDatabricks host- AzureThe cluster is not getting created as my Azure subscription has been disabled after a month of free trial. However, Databricks subscription has still got...
- 3431 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @Aanchal Soni We haven't heard from you since the last response from @Tyler Retzlaff , and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to o...
- 2 kudos
- 1899 Views
- 2 replies
- 2 kudos
- 1899 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Pavan Kumar Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...
- 2 kudos
- 11162 Views
- 1 replies
- 0 kudos
Model Serving error - Java gateway process exited before sending its port number
Hello, I am trying to serve a model endpoint (using Databricks GUI) for a model that was successfully logged to the Model Registry. However, the endpoint creation failed with the following errors: Endpoint logs with error messagesEndpoint events with...
- 11162 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Nikhil Gajghate We haven't heard from you since the last response from @Kaniz Fatma , and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to o...
- 0 kudos
- 1882 Views
- 2 replies
- 5 kudos
Resolved! Databricks voucher
I have attended the training last year of databricks to gain knowledge and help the clients but later i got to know that there are vouchers also available for which survey needs to complete which i completed now. I have already given some of the exam...
- 1882 Views
- 2 replies
- 5 kudos
- 5 kudos
Hi @Shishir Shivhare Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answ...
- 5 kudos
- 1378 Views
- 1 replies
- 3 kudos
delta table
can we connect delta table of databricks from one workspace to another in different subscription and run vacuum command?
- 1378 Views
- 1 replies
- 3 kudos
- 3 kudos
Hi @ASHUTOSH YADAV Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 3 kudos
| User | Count |
|---|---|
| 1616 | |
| 787 | |
| 383 | |
| 349 | |
| 287 |
