- 2251 Views
- 1 replies
- 0 kudos
Free edition and serverless edition are not loading and are really slow
Good morning,I am trying since last week to access two workspaces that I have (one in the free edition), and the other one in the paid serverless edition.Both of the workspaces are not loading, if they load they are very very slow, and I cannot see ...
- 2251 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @DanielaHello ,That's weird. According to status page, there is no outage in any region currently. Could you try to use different browser? Or try to log in incognito mode.For instance, Databricks Free Editon is currently available only in one regi...
- 0 kudos
- 3601 Views
- 1 replies
- 1 kudos
Resolved! Extracting cost by user (run_by) for All-purpose clusters and SQL warehouse usage
Hi,I'm trying to extract usage cost per user (run_by) for workloads that utilize all-purpose clusters and SQL warehouses. I’ve been exploring the system.billing.usage table but noticed some challenges:1. For records related to all-purpose clusters an...
- 3601 Views
- 1 replies
- 1 kudos
- 1 kudos
Attribution of compute usage to individual users for all-purpose clusters and SQL warehouses is only partially supported. Job compute (including serverless jobs) and workflows are reliably attributable to the job owner/service principal. For interact...
- 1 kudos
- 3441 Views
- 2 replies
- 6 kudos
Resolved! Unity Catalog blocks DML (UPDATE, DELETE) on static Delta tables — unable to use spark.sql
Hello,We’ve started migrating from Azure Databricks (Hive Metastore) to AWS Databricks with Unity Catalog. Our entire codebase was deliberately designed around spark.sql('...') using DML operations (UPDATE, DELETE, MERGE) for two reasons:In many case...
- 3441 Views
- 2 replies
- 6 kudos
- 6 kudos
[RESOLVED] The issue was caused by the source tables being in Parquet format. After rewriting them as Delta tables, everything worked fine — including DML operations like UPDATE via DataFrame logic. Thanks!
- 6 kudos
- 3362 Views
- 4 replies
- 0 kudos
Executor OOM Error with AQE enabled
We have Databricks Spark Job. After migration from Databricks Runtime 10.4 to 15.4 one of our Spark jobs which uses broadcast hint started to fail with error:```ERROR Executor: Exception in task 2.0 in stage 371.0 (TID 16912)org.apache.spark.memory.S...
- 3362 Views
- 4 replies
- 0 kudos
- 0 kudos
I found similar issuehttps://kb.databricks.com/python/job-fails-with-not-enough-memory-to-build-the-hash-map-errorLooks like the reason of error is a bug in new Databricks feature which is called executor-side broadcast (ebj, executor broadcast join)...
- 0 kudos
- 1099 Views
- 1 replies
- 0 kudos
COPY INTO maintaining row order
I have a CSV file in S3 and loading the rows in the order they appear in the file is necessary for parsing it out later. When using COPY INTO will it maintain that order so the bronze layer is in exactly the same order as the source file?
- 1099 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey @wilsmith COPY INTO does not guarantee the order of rows because it processes files in parallel using Spark’s distributed architecture. This means that the ingestion engine reads different parts of the file simultaneously, potentially splitting a...
- 0 kudos
- 5163 Views
- 4 replies
- 0 kudos
Asset Bundle .py files being converted to notebooks when deployed to Databricks
Hi everyone, I'm finding a particularly frustrating issue whenever I try to run some python code in an asset bundle on my workspace. The code and notebooks deploy fine but once deployed, the code files get converted to notebooks and I'm no longer abl...
- 5163 Views
- 4 replies
- 0 kudos
- 0 kudos
I came here looking for a solution to the opposite problem: I was hoping my .py files to be available as a notebook (without adding extra headers). Unfortunately, this does not seem to be possible with DABs.@facebiranhari if you have not solved your ...
- 0 kudos
- 21045 Views
- 10 replies
- 2 kudos
Resolved! Performance enhancement while writing dataframes into Parquet tables
Hi,I am trying to write the contents of a dataframe into a parquet table using the command below.df.write.mode("overwrite").format("parquet").saveAsTable("sample_parquet_table")The dataframe contains an extract from one of our source systems, which h...
- 21045 Views
- 10 replies
- 2 kudos
- 2 kudos
I am Bob Clarke marketing manager of virtual assistants Pakistan and I help companies hire amazon virtual assistants who manage product listings order processing and inventory updates. Our trained staff improves efficiency and boosts sales. We suppor...
- 2 kudos
- 1447 Views
- 2 replies
- 0 kudos
reading XML file of mutiple row Tags
I have multiple xml files in a folder. i am reading into dataframe in a databricks cell. It has one rootTag and multiple rowTags. can i read into single spark dataframe (pyspark) for all the rowTags. Any reference for this or approach would greatly a...
- 1447 Views
- 2 replies
- 0 kudos
- 0 kudos
you are correct, but i have N number of different rowTags. how to read in a dataframe.
- 0 kudos
- 3306 Views
- 8 replies
- 2 kudos
Resolved! German Umlauts wrong via JDBC
Hi,I have the issue that German Umlauts are not getting retrieved correctly via the JDBC driver.It shows M�nchen instead of München.I load the driver in my java app via:<groupId>com.databricks</groupId><artifactId>databricks-jdbc</artifactId><version...
- 3306 Views
- 8 replies
- 2 kudos
- 2 kudos
ok, so it seems that it has something to do with the newly introduced native.encoding system property.So In Netbeans you have to provide -Dstdout.encoding=utf-8 to the vm if you are using JDK21.
- 2 kudos
- 3047 Views
- 3 replies
- 1 kudos
Deploying Metastore with Terraform
my goal is to be able to enable unity catalog on a clean Azure deployment of databricks with absolutely no history of databricks. I know I need to create a metastore for the Azure Region. And to do this I know I need Account Admin from the accounts p...
- 3047 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @JCooke ,The first assignment of the Databricks Account Admin role is a bit of a special case. There is always a manual step required to assign the first Account Admin in a new Databricks account on Azure. This step cannot be fully automated via T...
- 1 kudos
- 2318 Views
- 2 replies
- 1 kudos
Why is Merge with Deletion Vectors Slower Than Full File Rewrite on the Same Table?
I've run two MERGE INTO operations on the same Delta table—one with Deletion Vectors enabled (Case 1), and one without (Case 2).In Case 1 (with Deletion Vectors): executionTimeMs: 106,708 materializeSourceTimeMs: 24,344 numTargetRowsUpdated: 22 nu...
- 2318 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi Poojalets understand DV first - This avoid rewriting entire files by marking rows as deleted/updated via a bitmap (the deletion vector), which should, in theory, be faster for small updates.but DV introduces new overhead:1) Writing and updating t...
- 1 kudos
- 5604 Views
- 3 replies
- 3 kudos
Resolved! Unable to deploy Databricks Asset Bundle
Hi Team,I created a workflow/job and was able to deploy to Dev and Prod workspaces. But now I am unable to deploy job to "Dev" workspace and getting below error, [Also unable to see this job on Databricks UI]Deploying resources...Updating deployment ...
- 5604 Views
- 3 replies
- 3 kudos
- 2421 Views
- 1 replies
- 0 kudos
Is it possible to retain original deltatable data with Unity Catalog?
Hi everyone,I have a question regarding data retention in Unity Catalog. In the pre–Unity Catalog setup, I believe that even if we dropped an external table, the underlying data files remained intact.However, in the current best practices for Unity C...
- 2421 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Yuki, If you drop an external table, the underlying data remains accessible even now. Only the table definition is removed from the metastore, while the actual data is retained. The UNDROP command for an EXTERNAL table simply recreates the table...
- 0 kudos
- 1639 Views
- 2 replies
- 0 kudos
Delta table partition folder names is getting changed
I am facing an issue where the expected date partition folder should be named in format like "campaign_created_date=2024-01-17", but instead it is writing as "ad", "8B" looks like a random folder names.Usually it will be like below:Now it changed lik...
- 1639 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi Satish,This is due to the column mapping enabled on the tableFrom Databricks Docs:When you enable column mapping for a Delta table, random prefixes replace column names in partition directories for Hive-style partitioning. See Rename and drop colu...
- 0 kudos
- 3526 Views
- 6 replies
- 0 kudos
No option to create cluster
I don't see any option to create cluster inside compute .How to create cluster ? Please help me
- 3526 Views
- 6 replies
- 0 kudos
- 0 kudos
Yes if you are using legacy community version you will be able to create clusters but with free edition it is limited serverless compute
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
