- 2582 Views
- 2 replies
- 1 kudos
pandas issue
We identify a potential bug in either DBFS or Pandas that when writting a dataframe using Pandas `to_csv`, `to_parquet`, `to_pickle` etc to a mounted ADLS location with read-only service principle didn't throw permission deny exceptions. However, met...
- 2582 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Yung-Hang Chang​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...
- 1 kudos
- 2619 Views
- 1 replies
- 1 kudos
Concurrent Insert on a delta table fails if the table contains Identity Columns. Error message are added below.MetadataChangedException: The metadata ...
Concurrent Insert on a delta table fails if the table contains Identity Columns. Error message are added below.MetadataChangedException: The metadata of the Delta table has been changed by a concurrent update. Please try the operation again.
- 2619 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @SK ASIF ALI​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 1 kudos
- 2068 Views
- 1 replies
- 0 kudos
We have concern with respect to the feasibility of installing the Databricks runtime supported version on-premise. This is because we are planning to...
We have concern with respect to the feasibility of installing the Databricks runtime supported version on-premise. This is because we are planning to migrate the existing Azure Databricks application code to an on-premise environment based on the cl...
- 2068 Views
- 1 replies
- 0 kudos
- 0 kudos
Any update?
- 0 kudos
- 4321 Views
- 2 replies
- 2 kudos
DLT config/setting json support
hi team,There used to be option to provide DLT pipeline settings either via UI or JSON, but I do not see it anymore after switching to new UI. Is this something expected ? am I missing something ? here is screenshot for reference.
- 4321 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Chhaya Vishwakarma​ This option is available, could you please clear the browser cache and try ? or can you try in an incognito window?
- 2 kudos
- 10271 Views
- 2 replies
- 1 kudos
Trying to mount Azure Data Lake Storage Gen 2 to Azure Databricks
I have validated many many many times all my credentials and I am still getting the following error (at the very end). ChatGPT said to basically recheck everything and I did. The one thing I didn't do was grant the permissions which I have since done...
- 10271 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Chris Sarrico​ Could you please specify the container name before the storage account name like this :so the source looks like this. :source = "abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/"https://learn.microsoft.com/en-u...
- 1 kudos
- 2561 Views
- 1 replies
- 0 kudos
0 byte files appearing in blob storage
When writing out files to azure blob storage using spark there are some 0 byte files being created on the web portal but not when I view it using storage explorer.I was wondering if there is there any work around for this?
- 2561 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Ryan Low​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 0 kudos
- 2349 Views
- 1 replies
- 0 kudos
Spark read not working in parallel
Hi All,Am trying to run a spark jdbc query and the read is running on only 1 worker and is taking a lot of time. Usually, it takes 4 minutes in aws lambda but now it is taking approx 10 minutes while using databricks. There are only 4m records (write...
- 2349 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Saloni Bhatia​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 0 kudos
- 2756 Views
- 1 replies
- 0 kudos
Silent failure in DataFrameWriter when loading data to Redshift
Context:I'm using DataFrameWriter to load the dataSet into the Redshift. DataFrameWriter writes the dataSet to S3, and loads data from S3 to Redshift by issuing the Redshift copy command. Issue:In frequently we are observing, the data is present in t...
- 2756 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Kishorekumar Somasundaram​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 0 kudos
- 3739 Views
- 1 replies
- 0 kudos
Getting JsonParseException: Unexpected character ('<' (code 60))
I have a scheduled job that is executed using a notebook. Within one of the notebook cells, there is a check to determine if a table exists. However, even when the table does exist, it incorrectly identifies it as non-existent and proceeds to execut...
- 3739 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Mahesh Chahare​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 0 kudos
- 3485 Views
- 1 replies
- 1 kudos
Having issue trying to download a csv file from a website using FireFox Selenium.
Hi, When I clicked on the download button from a website thru Firefox selenium using element.click(), and the download destination is being set as Azure datalake storage. Then, after the download started, those .csv and .csv.part files never gotten m...
- 3485 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Brandon Lim​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 1 kudos
- 1898 Views
- 0 replies
- 2 kudos
Your guide to data and AI governance talks at the Data + AI Summit 2023
The highly anticipated Data+AI summit is just around the corner. Are you curious about redefining your data governance strategy in the era of LLMs and generative AI? Look no further! This year's Data+AI summit offers a dynamic lineup of keynotes, dem...
- 1898 Views
- 0 replies
- 2 kudos
- 11042 Views
- 3 replies
- 0 kudos
Parquet column cannot be converted. Column: [Rainfall_Value], Expected: DoubleType, Found: INT64
df.printSchema()root |-- Device_ID: string (nullable = true) |-- Location: string (nullable = true) |-- Latitude: double (nullable = true) |-- Longitude: double (nullable = true) |-- DateTime: timestamp (nullable = true) |-- Rainfall_Value: double (n...
- 11042 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @THIAM HUAT TAN​ , The issue is because the schema defined for the column "Rainfall_Value" is of DoubleType and the values present in the data frame are of Integer type. This could be because of one or multiple values. Depending on the data, you ...
- 0 kudos
- 8274 Views
- 1 replies
- 0 kudos
StreamingQueryException: Read timed out // Reading from delta share'd dataset
I have a workspace in GCP that's reading from a delta-shared dataset hosted in S3. When trying to run a very basic DLT pipeline, I'm getting the below error. Any help would be awesome!Code:import dlt @dlt.table def fn(): return (spark.readStr...
- 8274 Views
- 1 replies
- 0 kudos
- 0 kudos
@Charlie You​ :The error message you're encountering suggests a timeout issue when reading from the Delta-shared dataset hosted in S3. There are a few potential reasons and solutions you can explore:Network connectivity: Verify that the network conne...
- 0 kudos
- 2390 Views
- 1 replies
- 0 kudos
If we do z-order on a particular column will delta log stats collection be affected?
Let's assume a table contains more than 40 columns, now we know it automatically collects stat for the first 32 columns. If we run a z-order on a particular column(let's say column 1), then will the log file collect stats for all the 32 columns or wi...
- 2390 Views
- 1 replies
- 0 kudos
- 0 kudos
@Shubhadip Ghosh​ : Hope this helps. In Delta Lake, when you perform Z-Ordering on a particular column, it reorganizes the data within the files based on the values of that column. However, Z-Ordering itself does not directly affect the statistics co...
- 0 kudos
- 2139 Views
- 1 replies
- 0 kudos
Will consecutive delete insert affect z-ordering?
Let's say there is a delta table with a date field as its partition. In a table where condition, we delete all the rows according to the division. The data is currently being inserted into the same date field. If we do a z-order after inserting the d...
- 2139 Views
- 1 replies
- 0 kudos
- 0 kudos
@Shubhadip Ghosh​ :In Delta Lake, when you perform a delete operation on a table, it doesn't physically remove the data from the files. Instead, it marks the affected rows for deletion by adding a tombstone marker to the Delta transaction log. This e...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
