- 2368 Views
- 4 replies
- 5 kudos
Relative Path Reading Ambiguity in running nested run commands
Hello All,I came across an unusual error while using the %run & dbutils.notebook.run() functionalities of the notebook in tandem and the particular scenarios are listed below -I have below directory structure(simplified) where all 3 notebooks are loc...
- 2368 Views
- 4 replies
- 5 kudos
- 5 kudos
I'm going to run an experiment in my workspace and let you know if I see the same thing. I'm not sure if I have seen this, but also not sure if my use of relative pathing previously had notebooks in different directories as you have listed. General...
- 5 kudos
- 1871 Views
- 2 replies
- 0 kudos
Range join hint does not help in faster execution of spark sql
Spark SQL execution did not complete even after 12 hours, i ran it on i3.xlarge with 4 worker nodes.only two worker nodes showed as running, with CPU at 100%what should i do differently? --SQLINSERT into attribute_results...SELECT /*+ BROADCAST(t) ...
- 1871 Views
- 2 replies
- 0 kudos
- 0 kudos
can you share the result of the below query ?select count(1) from transaction_attributes where analysis_start_date = '2025-05-01' and analysis_end_date = '2025-05-01' , If it has multiple entries , the join condition will lead to cross join and henc...
- 0 kudos
- 12974 Views
- 3 replies
- 1 kudos
How can I fix this error. ExecutionError: An error occurred while calling o392.mount: java.lang.NullPointerException
Hello, I'm trying to mount my Databricks to my Azure gen 2 data lake to read in data from the container, but I get an error when executing this line of code: dbutils.fs.mount( source = "abfss://resumes@choisysresume.dfs.core.windows.net/", mount_poin...
- 12974 Views
- 3 replies
- 1 kudos
- 1 kudos
I was using databricks scopes, to get the key which was used in the the config. I received a similar mount error while mounting with "wasbs" driver, "ExecutionError: An error occurred while calling o427.mount.", this was the issue because the scope ...
- 1 kudos
- 38956 Views
- 21 replies
- 13 kudos
Community Edition signup issues
I am trying to sign up for the community edition (https://databricks.com/try-databricks) for use with a databricks academy course. However, I am unable to signup and I receive the following error (image attached). On going to login page (link in ora...
- 38956 Views
- 21 replies
- 13 kudos
- 13 kudos
I am facing issue like you have reached maximum number of account associated with this databricks account and also you are not the member of any workspace and also i can't delete my existing account associated with my email.And also i can't open my c...
- 13 kudos
- 1743 Views
- 2 replies
- 0 kudos
How to check integrity on tables with PRIMARY KEY RELY optimization
Databricks can now use RELY to optimize some queries when using Photon-enabled compute.But what if one wanted to check the integrity of the table, actually not relying on the constraint. That's not an unreasonable ask I would think.Is there a way to ...
- 1743 Views
- 2 replies
- 0 kudos
- 0 kudos
Unfortunately, none of these suggestions had any effect.I seem to have been able (for now) to work around the optimization using EXECUTE IMMEDIATE sql INTO var, crafting a query string on the form "SELECT COUNT(*) - COUNT(DISTINCT id)".I suppose the ...
- 0 kudos
- 23414 Views
- 3 replies
- 0 kudos
performance issues using shared compute access mode in scala
I created on our dev environment a cluster using the shared access mode, for our devs to use (instead of separate single user clusters).What I notice is that the performance of this cluster is terrible. And I mean really terrible: notebook cells wit...
- 23414 Views
- 3 replies
- 0 kudos
- 0 kudos
I am experiencing a huge performance difference between shared and dedicated compute with spark.createDataFrame(pandas_df). Same code, same data, but it completes in 6 s on dedicated cluster and takes 6+ minutes on the shared cluster. >60 times diffe...
- 0 kudos
- 1608 Views
- 2 replies
- 2 kudos
Is it ok to Run ANALYZE TABLE COMPUTE DELTA STATISTICS While data is loading into a Delta Table?
Hi all,I have a doubt regarding the best practices for running ANALYZE TABLE table_name COMPUTE DELTA STATISTICS on a Delta table. Is it recommended to execute this command while data is being loaded into the table, or should it be run afterward? Ad...
- 1608 Views
- 2 replies
- 2 kudos
- 2 kudos
ANALYZE TABLE is a read-only operation. It reads the data to compute statistics but does not modify the data. Running ANALYZE TABLE COMPUTE DELTA STATISTICS while data is still being loaded into a Delta table is generally not recommended. The ANALYZE...
- 2 kudos
- 2041 Views
- 2 replies
- 1 kudos
Resolved! How to change streaming table/column description in DLT
Hi folks,How to change streaming table/column description in DLT during run time like we do for delta tables because ALTER STREAMING table isn't working.eg:COMMENT ON COLUMN ops_catalog_gld_dev.schema_silver.table_name.property_sid IS 'The key of the...
- 2041 Views
- 2 replies
- 1 kudos
- 1 kudos
Modifying streaming table column descriptions should be done via pipeline configuration instead of runtime SQL commands, as DLT does not retain such runtime alterations during pipeline refreshes.
- 1 kudos
- 4392 Views
- 3 replies
- 2 kudos
Resolved! updates on Bring Your Own Lineage (BYOL)?
One of the most exciting things in recent roadmap discussions was the idea of BYOL, so we could import external lineage into Unity Catalog and make it really useful for understanding where our data flows. We're planning some investments for the next ...
- 4392 Views
- 3 replies
- 2 kudos
- 1220 Views
- 1 replies
- 0 kudos
Cleanup databricks logon
I have too many Accounts to logon at - https://login.databricks.com/How do I clean up unwanted credentials and delete accounts ?
- 1220 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Thayal! To remove unwanted accounts, you can refer to this post: https://community.databricks.com/t5/administration-architecture/delete-databricks-account/td-p/87187It clearly outlines the steps to delete accounts.
- 0 kudos
- 3060 Views
- 8 replies
- 1 kudos
Databricks notebooks error
I want to read a csv file using pandas library in python in Databricks Notebooks and I uploaded my csv file (employee_data) on adfs but it still shows no such file exists can anyone help me on this?
- 3060 Views
- 8 replies
- 1 kudos
- 1 kudos
Load it using PySpark and create a pandas data frame. Here is how you do it after uploading the datafile_path = "/FileStore/tables/your_file_name.csv"# Load CSV as Spark DataFramedf_spark = spark.read.option("header", "true").option("inferSchema", "t...
- 1 kudos
- 2822 Views
- 1 replies
- 1 kudos
Resolved! CHAR/VARCHAR fields sometimes show as STRING in a view
We've found an interesting behavior where `char` and `varchar` fields in a table show as the `string` type in a view. Consider the following table and view:create or replace table thirty_day_tables.kit_varchar_string ( str1 string, str2 char(10),...
- 2822 Views
- 1 replies
- 1 kudos
- 1 kudos
In Spark SQLstring is the canonical type for all textual data.char(n) and varchar(n) are parsed and stored as metadata, but internally treated as string.When you create a view, Spark does not preserve the original char(n) or varchar(n) types — it nor...
- 1 kudos
- 14774 Views
- 7 replies
- 5 kudos
Resolved! Delta Live table
Hi All,I'm working on a databricks delta live table(DLT) pipe line where we receive daily fully sanshot csv files in azure cloud storage .These files contain HR data (eg.employee file) and i'm using autoloader to ingest them into bronze layer DLT tab...
- 14774 Views
- 7 replies
- 5 kudos
- 5 kudos
Hi @Ranga_naik1180, There is no need to create an intermediate view in SQL. You can directly read the change data feed from silver into the gold table. You can use the code something like below: CREATE STREAMING LIVE TABLE gold_table AS SELECT * FRO...
- 5 kudos
- 857 Views
- 1 replies
- 0 kudos
Showing masked column when they should not
In my organization we mask some column because they are PII, I have a dlt pipeline, I am masking these columns like this:CASE WHEN is_account_group_member("BDAIM-{environment.upper()}-PII_Unmask") THEN Personshopper.firstName ELSE mask(Personshopper....
- 857 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @carlos_tasayco, this is supported for materialized views. Please check the document to confirm if you're using the right syntax: https://docs.databricks.com/aws/en/dlt-ref/dlt-sql-ref-create-materialized-view#examples Please let me know if you ...
- 0 kudos
- 4380 Views
- 2 replies
- 0 kudos
Very Slow UDF Execution on One Cluster Compared to Another with Similar Config
Hi all,I’m experiencing a significant slowdown behavior in Python UDF execution times on a particular cluster. The same code runs much faster on another cluster with very similar hardware and policy settings.This cell takes 2–3 minutes on the problem...
- 4380 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @alexbarev ,The slowdown is likely due to using Python UDFs on a Shared (Standard) access mode cluster with Unity Catalog, which adds extra security and isolation overhead. Using a Dedicated access mode cluster removes the extra isolation overhead...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
