- 5095 Views
- 1 replies
- 0 kudos
Readying a partitioned Table in Spark Structured Streaming
Does the pre-partitioning of a Delta Table has an influence on the number of "default" Partition of a Dataframe when readying the data?Put differently, using spark structured streaming, when readying from a delta table, is the number of Dataframe par...
- 5095 Views
- 1 replies
- 0 kudos
- 0 kudos
Pre-partitioning of a Delta Table does not strictly determine the number of "default" DataFrame partitions when reading data with Spark Structured Streaming. Unlike Kafka, where each DataFrame partition maps one-to-one to a Kafka partition, Delta Lak...
- 0 kudos
- 4894 Views
- 1 replies
- 0 kudos
APPLY INTO Highdate instead of NULL for __END_AT
I really like the APPLY INTO function to keep track of changes and historize them in SCD2.However, I am a bit confused that current records get an __END_AT of NULL. Typically, __END_AT should be a highgate (i.e. 9999-12-31) or similar, so that a poin...
- 4894 Views
- 1 replies
- 0 kudos
- 0 kudos
The APPLY INTO function for SCD2 historization typically sets the __END_AT field of current records to NULL rather than a "highgate" like 9999-12-31. This is by design and reflects that the record is still current and has no defined end date yet. Cur...
- 0 kudos
- 6661 Views
- 1 replies
- 0 kudos
Pyspark - alias is not applied in pivot if only one aggregation
This is not making it consistent when we perform aggregation on multiple columns and thus it is hindering metadata driven transformation because of inconsistency.How can we request Databricks/pyspark to include this ? and is there any known work arou...
- 6661 Views
- 1 replies
- 0 kudos
- 0 kudos
When using PySpark or Databricks to perform a pivot operation with only a single aggregation, you may notice that the alias is not applied as expected, leading to inconsistencies, especially when trying to automate or apply metadata-driven frameworks...
- 0 kudos
- 4828 Views
- 1 replies
- 0 kudos
Timeout for dbutils.jobs.taskValues.set(key, value)
I have a job that call notebook with dbutils.jobs.taskValues.set(key, value) method and assigns around 20 parameters.When I run it - it works.But when I try to call 2 or more copies of a job with different parameters - it fails with error on differen...
- 4828 Views
- 1 replies
- 0 kudos
- 0 kudos
The error you are encountering when running multiple simultaneous Databricks jobs using dbutils.jobs.taskValues.set(key, value) indicates a connection timeout issue to the Databricks backend API (connect timed out at ...us-central1.gcp.databricks.com...
- 0 kudos
- 5935 Views
- 1 replies
- 0 kudos
How to implement Lifecycle of Data When Use ADLS
Hello everyone, nice to greet you. I have a question about the data lifecycle in ADLS. I know ADLS has its own rules, but they aren't working properly because I have two ADLS accounts: one for hot data and another for cool storage where the informati...
- 5935 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes, you can move data from your HOT ADLS account to a COOL ADLS account while handling Delta Lake log issues, but this requires special techniques due to the nature of Delta Lake’s transaction log. The problem stems from Delta tables’ dependency on ...
- 0 kudos
- 5058 Views
- 1 replies
- 0 kudos
Workspace Client dbutils issue
host = "https://adb-xxxxxx.xx.azuredatabricks.net"token = "dapxxxxxxx"we are using databricksconnect from databricks.sdk import WorkspaceClientdbutil = WorkspaceClient(host=host,token=token).dbutilsfiles = dbutil.fs.ls("abfss://container-name@storag...
- 5058 Views
- 1 replies
- 0 kudos
- 0 kudos
The error where files and directories can be read at the root ADLS level but not at the blob/subdirectory level, combined with a "No file or directory exists on path" message, is frequently due to permission configuration, incorrect path usage, or ne...
- 0 kudos
- 5142 Views
- 1 replies
- 0 kudos
Executing Stored Procedures/update in Federated SQL Server
I have federated Azure SQL DB in my DBX workspace, but I am not able to run update commands or execute a stored procedure, is this still not supported?
- 5142 Views
- 1 replies
- 0 kudos
- 0 kudos
Federated connections from Azure Databricks to Azure SQL DB via Lakehouse Federation currently only support read-only queries—meaning running update commands or executing stored procedures directly through the federated Unity Catalog interface is not...
- 0 kudos
- 3051 Views
- 1 replies
- 0 kudos
System tables for DLT Expectations Quality Metrics
Hi Everyone,I’m working with Delta Live Tables (DLT) and using Expectations to track data quality, but I’m having trouble finding where the expectation quality metrics are stored in the DLT system tables.My questions are:Which specific system table(s...
- 3051 Views
- 1 replies
- 0 kudos
- 0 kudos
@vamsi_simbus DLT captures data quality metrics in specialized system tables known as “event” and “metrics” tables. Specifically, look in the following table:LIVE.DLT_EVENT_LOG or LIVE.DLT_METRICS: These tables contain granular event logs and metric...
- 0 kudos
- 1247 Views
- 1 replies
- 0 kudos
Resolved! What are best practices for designing a large-scale data engineering pipeline on Databricks for real
How do you design a scalable, reliable pipeline that handles both fast/continuous data and slower bulk data in the same system?
- 1247 Views
- 1 replies
- 0 kudos
- 0 kudos
Very generic question Here are general rules and best practices related to Databricks well-architected framework: https://docs.databricks.com/aws/en/lakehouse-architecture/well-architected Take a deeper look on operational excellence, reliability an...
- 0 kudos
- 36303 Views
- 23 replies
- 8 kudos
vocareum lab access
Hi I am doing a data engineering course in databricks(Partner labs) and would like to have access to vocareum workspace to practice using the demo sessions.can you please help me to get the access to this workspace?regards,Aravind
- 36303 Views
- 23 replies
- 8 kudos
- 8 kudos
You can log into databricks, search for "Canada Sales" in the Marketplace and find "Simulated Canada Sales and Opportunities Data". Get free instant access, wait a few seconds for the warehouse to be built for you et voila: the tables for building th...
- 8 kudos
- 7169 Views
- 1 replies
- 0 kudos
Naming jobs in the Spark UI in Databricks Runtime 15.4
I am asking almost the same question as: https://community.databricks.com/t5/data-engineering/how-to-improve-spark-ui-job-description-for-pyspark/td-p/48959 . I would like to know how to improve the readability of the Spark UI by naming jobs. I am...
- 7169 Views
- 1 replies
- 0 kudos
- 0 kudos
You are correct—on Databricks Runtime 15.4 and with shared clusters (or clusters enabled with Unity Catalog), you will see the [JVM_ATTRIBUTE_NOT_SUPPORTED] error when trying to directly access sparkContext attributes that are only available in singl...
- 0 kudos
- 4704 Views
- 1 replies
- 0 kudos
Can we develop a connector that integrates nintex and Databricks community version
Can we develop a connector that integrates nintex and Databricks with community version
- 4704 Views
- 1 replies
- 0 kudos
- 0 kudos
It is technically possible to develop a connector that integrates Nintex with Databricks, but there are important limitations when trying to achieve this with Databricks Community Edition. Connector Integration Overview Nintex can be integrated with...
- 0 kudos
- 5888 Views
- 1 replies
- 0 kudos
Conditional Execution in DLT Pipeline based on the output
Hello ,I'm working on a Delta Live Tables (DLT) pipeline where I need to implement a conditional step that only triggers under specific conditions. Here's the challenge I'm facing:I have a function that checks if the data meets certain thresholds. If...
- 5888 Views
- 1 replies
- 0 kudos
- 0 kudos
In Delta Live Tables (DLT), native conditional or branch-based control flow is limited; all table/stream definitions declared in your pipeline will execute, and dependencies are handled via @Dlt.table or @Dlt.view decorators. You can’t dynamically sk...
- 0 kudos
- 6315 Views
- 1 replies
- 0 kudos
Efficient Parallel JSON Extraction from Elasticsearch in Azure Databricks
Hi Databricks community,I'm facing a challenge extracting JSON data from Elasticsearch in Azure Databricks efficiently, maintaining header information.Previously, I had to use RDDs for parallel extraction, but they're no longer supported in Databrick...
- 6315 Views
- 1 replies
- 0 kudos
- 0 kudos
To efficiently extract JSON data from Elasticsearch in Azure Databricks—while maintaining header information and without reverting to legacy RDD-based parallelization—a few modern Spark-based strategies can be used. Spark DataFrames and Spark SQL sup...
- 0 kudos
- 1752 Views
- 2 replies
- 1 kudos
Resolved! DLT behaving differently when used with python syntax vs when used with sql syntax to read CDF
I was trying t read CDF data of a table as a DLT materialized view.It works fine with sql syntax reading all the columns of the source table along with the 3 CDF columns : _change_type,_commit_timestamp,_commit_version:@dlt.table()def change_table():...
- 1752 Views
- 2 replies
- 1 kudos
- 1 kudos
When accessing Change Data Feed (CDF) data in Delta Live Tables (DLT), the behavior between SQL and Python APIs differs notably regarding CDF metadata columns—_change_type, _commit_timestamp, and _commit_version. SQL Approach (using table_changes):T...
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
