- 1921 Views
- 2 replies
- 0 kudos
Range join hint does not help in faster execution of spark sql
Spark SQL execution did not complete even after 12 hours, i ran it on i3.xlarge with 4 worker nodes.only two worker nodes showed as running, with CPU at 100%what should i do differently? --SQLINSERT into attribute_results...SELECT /*+ BROADCAST(t) ...
- 1921 Views
- 2 replies
- 0 kudos
- 0 kudos
can you share the result of the below query ?select count(1) from transaction_attributes where analysis_start_date = '2025-05-01' and analysis_end_date = '2025-05-01' , If it has multiple entries , the join condition will lead to cross join and henc...
- 0 kudos
- 13068 Views
- 3 replies
- 1 kudos
How can I fix this error. ExecutionError: An error occurred while calling o392.mount: java.lang.NullPointerException
Hello, I'm trying to mount my Databricks to my Azure gen 2 data lake to read in data from the container, but I get an error when executing this line of code: dbutils.fs.mount( source = "abfss://resumes@choisysresume.dfs.core.windows.net/", mount_poin...
- 13068 Views
- 3 replies
- 1 kudos
- 1 kudos
I was using databricks scopes, to get the key which was used in the the config. I received a similar mount error while mounting with "wasbs" driver, "ExecutionError: An error occurred while calling o427.mount.", this was the issue because the scope ...
- 1 kudos
- 39552 Views
- 21 replies
- 13 kudos
Community Edition signup issues
I am trying to sign up for the community edition (https://databricks.com/try-databricks) for use with a databricks academy course. However, I am unable to signup and I receive the following error (image attached). On going to login page (link in ora...
- 39552 Views
- 21 replies
- 13 kudos
- 13 kudos
I am facing issue like you have reached maximum number of account associated with this databricks account and also you are not the member of any workspace and also i can't delete my existing account associated with my email.And also i can't open my c...
- 13 kudos
- 1797 Views
- 2 replies
- 0 kudos
How to check integrity on tables with PRIMARY KEY RELY optimization
Databricks can now use RELY to optimize some queries when using Photon-enabled compute.But what if one wanted to check the integrity of the table, actually not relying on the constraint. That's not an unreasonable ask I would think.Is there a way to ...
- 1797 Views
- 2 replies
- 0 kudos
- 0 kudos
Unfortunately, none of these suggestions had any effect.I seem to have been able (for now) to work around the optimization using EXECUTE IMMEDIATE sql INTO var, crafting a query string on the form "SELECT COUNT(*) - COUNT(DISTINCT id)".I suppose the ...
- 0 kudos
- 23523 Views
- 3 replies
- 0 kudos
performance issues using shared compute access mode in scala
I created on our dev environment a cluster using the shared access mode, for our devs to use (instead of separate single user clusters).What I notice is that the performance of this cluster is terrible. And I mean really terrible: notebook cells wit...
- 23523 Views
- 3 replies
- 0 kudos
- 0 kudos
I am experiencing a huge performance difference between shared and dedicated compute with spark.createDataFrame(pandas_df). Same code, same data, but it completes in 6 s on dedicated cluster and takes 6+ minutes on the shared cluster. >60 times diffe...
- 0 kudos
- 1651 Views
- 2 replies
- 2 kudos
Is it ok to Run ANALYZE TABLE COMPUTE DELTA STATISTICS While data is loading into a Delta Table?
Hi all,I have a doubt regarding the best practices for running ANALYZE TABLE table_name COMPUTE DELTA STATISTICS on a Delta table. Is it recommended to execute this command while data is being loaded into the table, or should it be run afterward? Ad...
- 1651 Views
- 2 replies
- 2 kudos
- 2 kudos
ANALYZE TABLE is a read-only operation. It reads the data to compute statistics but does not modify the data. Running ANALYZE TABLE COMPUTE DELTA STATISTICS while data is still being loaded into a Delta table is generally not recommended. The ANALYZE...
- 2 kudos
- 2098 Views
- 2 replies
- 1 kudos
Resolved! How to change streaming table/column description in DLT
Hi folks,How to change streaming table/column description in DLT during run time like we do for delta tables because ALTER STREAMING table isn't working.eg:COMMENT ON COLUMN ops_catalog_gld_dev.schema_silver.table_name.property_sid IS 'The key of the...
- 2098 Views
- 2 replies
- 1 kudos
- 1 kudos
Modifying streaming table column descriptions should be done via pipeline configuration instead of runtime SQL commands, as DLT does not retain such runtime alterations during pipeline refreshes.
- 1 kudos
- 4505 Views
- 3 replies
- 2 kudos
Resolved! updates on Bring Your Own Lineage (BYOL)?
One of the most exciting things in recent roadmap discussions was the idea of BYOL, so we could import external lineage into Unity Catalog and make it really useful for understanding where our data flows. We're planning some investments for the next ...
- 4505 Views
- 3 replies
- 2 kudos
- 1307 Views
- 1 replies
- 0 kudos
Cleanup databricks logon
I have too many Accounts to logon at - https://login.databricks.com/How do I clean up unwanted credentials and delete accounts ?
- 1307 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Thayal! To remove unwanted accounts, you can refer to this post: https://community.databricks.com/t5/administration-architecture/delete-databricks-account/td-p/87187It clearly outlines the steps to delete accounts.
- 0 kudos
- 3238 Views
- 8 replies
- 1 kudos
Databricks notebooks error
I want to read a csv file using pandas library in python in Databricks Notebooks and I uploaded my csv file (employee_data) on adfs but it still shows no such file exists can anyone help me on this?
- 3238 Views
- 8 replies
- 1 kudos
- 1 kudos
Load it using PySpark and create a pandas data frame. Here is how you do it after uploading the datafile_path = "/FileStore/tables/your_file_name.csv"# Load CSV as Spark DataFramedf_spark = spark.read.option("header", "true").option("inferSchema", "t...
- 1 kudos
- 2912 Views
- 1 replies
- 1 kudos
Resolved! CHAR/VARCHAR fields sometimes show as STRING in a view
We've found an interesting behavior where `char` and `varchar` fields in a table show as the `string` type in a view. Consider the following table and view:create or replace table thirty_day_tables.kit_varchar_string ( str1 string, str2 char(10),...
- 2912 Views
- 1 replies
- 1 kudos
- 1 kudos
In Spark SQLstring is the canonical type for all textual data.char(n) and varchar(n) are parsed and stored as metadata, but internally treated as string.When you create a view, Spark does not preserve the original char(n) or varchar(n) types — it nor...
- 1 kudos
- 15091 Views
- 7 replies
- 5 kudos
Resolved! Delta Live table
Hi All,I'm working on a databricks delta live table(DLT) pipe line where we receive daily fully sanshot csv files in azure cloud storage .These files contain HR data (eg.employee file) and i'm using autoloader to ingest them into bronze layer DLT tab...
- 15091 Views
- 7 replies
- 5 kudos
- 5 kudos
Hi @Ranga_naik1180, There is no need to create an intermediate view in SQL. You can directly read the change data feed from silver into the gold table. You can use the code something like below: CREATE STREAMING LIVE TABLE gold_table AS SELECT * FRO...
- 5 kudos
- 919 Views
- 1 replies
- 0 kudos
Showing masked column when they should not
In my organization we mask some column because they are PII, I have a dlt pipeline, I am masking these columns like this:CASE WHEN is_account_group_member("BDAIM-{environment.upper()}-PII_Unmask") THEN Personshopper.firstName ELSE mask(Personshopper....
- 919 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @carlos_tasayco, this is supported for materialized views. Please check the document to confirm if you're using the right syntax: https://docs.databricks.com/aws/en/dlt-ref/dlt-sql-ref-create-materialized-view#examples Please let me know if you ...
- 0 kudos
- 4489 Views
- 2 replies
- 0 kudos
Very Slow UDF Execution on One Cluster Compared to Another with Similar Config
Hi all,I’m experiencing a significant slowdown behavior in Python UDF execution times on a particular cluster. The same code runs much faster on another cluster with very similar hardware and policy settings.This cell takes 2–3 minutes on the problem...
- 4489 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @alexbarev ,The slowdown is likely due to using Python UDFs on a Shared (Standard) access mode cluster with Unity Catalog, which adds extra security and isolation overhead. Using a Dedicated access mode cluster removes the extra isolation overhead...
- 0 kudos
- 5528 Views
- 1 replies
- 0 kudos
Resolved! Diffrence b/w community edition and Free edition Databrick platform
Recently there is a news from DATA +AI summit mentioning Free edition of Databricks platform, how its different from community edition?Follow up question.Is there any limitation on compute resources.Will it support other cloud services provider apart...
- 5528 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello Pavankumar,Regarding your questions:The difference between free edition and CE: "Free Edition has been designed and extended to include full access to the Data Intelligence Platform. It provides an easy-to-use environment where you can build AI...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
