- 1906 Views
- 2 replies
- 1 kudos
Resolved! How to change streaming table/column description in DLT
Hi folks,How to change streaming table/column description in DLT during run time like we do for delta tables because ALTER STREAMING table isn't working.eg:COMMENT ON COLUMN ops_catalog_gld_dev.schema_silver.table_name.property_sid IS 'The key of the...
- 1906 Views
- 2 replies
- 1 kudos
- 1 kudos
Modifying streaming table column descriptions should be done via pipeline configuration instead of runtime SQL commands, as DLT does not retain such runtime alterations during pipeline refreshes.
- 1 kudos
- 4183 Views
- 3 replies
- 2 kudos
Resolved! updates on Bring Your Own Lineage (BYOL)?
One of the most exciting things in recent roadmap discussions was the idea of BYOL, so we could import external lineage into Unity Catalog and make it really useful for understanding where our data flows. We're planning some investments for the next ...
- 4183 Views
- 3 replies
- 2 kudos
- 1102 Views
- 1 replies
- 0 kudos
Cleanup databricks logon
I have too many Accounts to logon at - https://login.databricks.com/How do I clean up unwanted credentials and delete accounts ?
- 1102 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Thayal! To remove unwanted accounts, you can refer to this post: https://community.databricks.com/t5/administration-architecture/delete-databricks-account/td-p/87187It clearly outlines the steps to delete accounts.
- 0 kudos
- 2780 Views
- 8 replies
- 1 kudos
Databricks notebooks error
I want to read a csv file using pandas library in python in Databricks Notebooks and I uploaded my csv file (employee_data) on adfs but it still shows no such file exists can anyone help me on this?
- 2780 Views
- 8 replies
- 1 kudos
- 1 kudos
Load it using PySpark and create a pandas data frame. Here is how you do it after uploading the datafile_path = "/FileStore/tables/your_file_name.csv"# Load CSV as Spark DataFramedf_spark = spark.read.option("header", "true").option("inferSchema", "t...
- 1 kudos
- 2624 Views
- 1 replies
- 1 kudos
Resolved! CHAR/VARCHAR fields sometimes show as STRING in a view
We've found an interesting behavior where `char` and `varchar` fields in a table show as the `string` type in a view. Consider the following table and view:create or replace table thirty_day_tables.kit_varchar_string ( str1 string, str2 char(10),...
- 2624 Views
- 1 replies
- 1 kudos
- 1 kudos
In Spark SQLstring is the canonical type for all textual data.char(n) and varchar(n) are parsed and stored as metadata, but internally treated as string.When you create a view, Spark does not preserve the original char(n) or varchar(n) types — it nor...
- 1 kudos
- 14048 Views
- 7 replies
- 5 kudos
Resolved! Delta Live table
Hi All,I'm working on a databricks delta live table(DLT) pipe line where we receive daily fully sanshot csv files in azure cloud storage .These files contain HR data (eg.employee file) and i'm using autoloader to ingest them into bronze layer DLT tab...
- 14048 Views
- 7 replies
- 5 kudos
- 5 kudos
Hi @Ranga_naik1180, There is no need to create an intermediate view in SQL. You can directly read the change data feed from silver into the gold table. You can use the code something like below: CREATE STREAMING LIVE TABLE gold_table AS SELECT * FRO...
- 5 kudos
- 775 Views
- 1 replies
- 0 kudos
Showing masked column when they should not
In my organization we mask some column because they are PII, I have a dlt pipeline, I am masking these columns like this:CASE WHEN is_account_group_member("BDAIM-{environment.upper()}-PII_Unmask") THEN Personshopper.firstName ELSE mask(Personshopper....
- 775 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @carlos_tasayco, this is supported for materialized views. Please check the document to confirm if you're using the right syntax: https://docs.databricks.com/aws/en/dlt-ref/dlt-sql-ref-create-materialized-view#examples Please let me know if you ...
- 0 kudos
- 4168 Views
- 2 replies
- 0 kudos
Very Slow UDF Execution on One Cluster Compared to Another with Similar Config
Hi all,I’m experiencing a significant slowdown behavior in Python UDF execution times on a particular cluster. The same code runs much faster on another cluster with very similar hardware and policy settings.This cell takes 2–3 minutes on the problem...
- 4168 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @alexbarev ,The slowdown is likely due to using Python UDFs on a Shared (Standard) access mode cluster with Unity Catalog, which adds extra security and isolation overhead. Using a Dedicated access mode cluster removes the extra isolation overhead...
- 0 kudos
- 5022 Views
- 1 replies
- 0 kudos
Resolved! Diffrence b/w community edition and Free edition Databrick platform
Recently there is a news from DATA +AI summit mentioning Free edition of Databricks platform, how its different from community edition?Follow up question.Is there any limitation on compute resources.Will it support other cloud services provider apart...
- 5022 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello Pavankumar,Regarding your questions:The difference between free edition and CE: "Free Edition has been designed and extended to include full access to the Data Intelligence Platform. It provides an easy-to-use environment where you can build AI...
- 0 kudos
- 1707 Views
- 6 replies
- 0 kudos
Experiencing sorting problems with bigint columns
Experiencing sorting problems with bigint columns across tables tested. Example:In the table the projectid (bigint as per schema - which is correct) yet when sorted in SQL is sorting as per a string value 1000903 is returned as the minimum projectid ...
- 1707 Views
- 6 replies
- 0 kudos
- 0 kudos
Try running explain command what's happening in the background, also if table is small try to write in another table and check
- 0 kudos
- 2935 Views
- 2 replies
- 1 kudos
Resolved! Batch Python UDFs in Unity Catalog and Spark SQL
Hello datanauts 六,I'm encountering a conceptual challenge regarding Batch Python UDFs within Spark SQL in Databricks. My primary question is: can Batch Python UDFs be used directly via Spark SQL? As a Databricks beginner, I'm seeking to understand ...
- 2935 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @stefan-vulpe Looking at your code and the behavior you're describing, I can identify the core issue and provide some insights about Batch Python UDFs in Databricks.The Core ProblemThe issue you're encountering is related to session isolation and ...
- 1 kudos
- 1269 Views
- 1 replies
- 1 kudos
Is it possible to reprocess only a portion of a streaming table data using DLT?
Hi all,Currently I have a standard notebook that it takes 2 dates as parameters, start date and end date it goes to the source then it pull only that portion of data then on target table deletes if necessary (if data within those ranges exists) the u...
- 1269 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @MauricioS Yes, you can achieve similar reprocessing functionality with DLT streaming tables,but it requires a different approach than your current batch process. Here are the main strategies:1. CDC Pattern with Tombstone RecordsThe most common ap...
- 1 kudos
- 2976 Views
- 1 replies
- 0 kudos
Resolved! Connection with Azure service principal
I have configured a data bricks connection with Azure using a service principal who is configured in both Azure system and data bricks. I am getting an issue calling this API since early June. https://$URI$/api/2.0/secrets/scopes/list Headers = @{ ...
- 2976 Views
- 1 replies
- 0 kudos
- 0 kudos
For anyone interested, this particular case was due migrating Azure Powershell to a new version. This is the link to follow just in case https://learn.microsoft.com/en-us/powershell/azure/migrate-az-14.0.0?view=azps-14.1.0
- 0 kudos
- 1827 Views
- 2 replies
- 1 kudos
Resolved! Specifing Output mode and Path when using For Each Batch
Since .foreachBatch() is "hijacking" the stream and executing arbitrary code in it, do I need to specify Output mode and Path:(df.writeStream .format("delta") .trigger(availableNow = True) .option("checkpointLocation", "check_point_location") .forea...
- 1827 Views
- 2 replies
- 1 kudos
- 4841 Views
- 1 replies
- 0 kudos
Parquet file for delta streaming live table with pipeline
I am having an issue with parquet files: I'm getting Illegal Parquet type: INT64 (TIMESTAMP(NANOS,false)) error while trying to read a parquet file (generated outside of DataBricks). I am using a Delta streaming live table with a pipeline. If I r...
- 4841 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @ClarkElliott Good day!! Cause Databricks Runtime versions 11.3 LTS and above do not support the TIMESTAMP_NANOS type in open source Apache Spark and Databricks Runtime. If a Parquet file contains fields with the TIMESTAMP_NANOS type, attempts to...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 580 | |
| 349 | |
| 287 |
