- 6477 Views
- 1 replies
- 0 kudos
Pyspark - alias is not applied in pivot if only one aggregation
This is not making it consistent when we perform aggregation on multiple columns and thus it is hindering metadata driven transformation because of inconsistency.How can we request Databricks/pyspark to include this ? and is there any known work arou...
- 6477 Views
- 1 replies
- 0 kudos
- 0 kudos
When using PySpark or Databricks to perform a pivot operation with only a single aggregation, you may notice that the alias is not applied as expected, leading to inconsistencies, especially when trying to automate or apply metadata-driven frameworks...
- 0 kudos
- 4710 Views
- 1 replies
- 0 kudos
Timeout for dbutils.jobs.taskValues.set(key, value)
I have a job that call notebook with dbutils.jobs.taskValues.set(key, value) method and assigns around 20 parameters.When I run it - it works.But when I try to call 2 or more copies of a job with different parameters - it fails with error on differen...
- 4710 Views
- 1 replies
- 0 kudos
- 0 kudos
The error you are encountering when running multiple simultaneous Databricks jobs using dbutils.jobs.taskValues.set(key, value) indicates a connection timeout issue to the Databricks backend API (connect timed out at ...us-central1.gcp.databricks.com...
- 0 kudos
- 5663 Views
- 1 replies
- 0 kudos
How to implement Lifecycle of Data When Use ADLS
Hello everyone, nice to greet you. I have a question about the data lifecycle in ADLS. I know ADLS has its own rules, but they aren't working properly because I have two ADLS accounts: one for hot data and another for cool storage where the informati...
- 5663 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes, you can move data from your HOT ADLS account to a COOL ADLS account while handling Delta Lake log issues, but this requires special techniques due to the nature of Delta Lake’s transaction log. The problem stems from Delta tables’ dependency on ...
- 0 kudos
- 4912 Views
- 1 replies
- 0 kudos
Workspace Client dbutils issue
host = "https://adb-xxxxxx.xx.azuredatabricks.net"token = "dapxxxxxxx"we are using databricksconnect from databricks.sdk import WorkspaceClientdbutil = WorkspaceClient(host=host,token=token).dbutilsfiles = dbutil.fs.ls("abfss://container-name@storag...
- 4912 Views
- 1 replies
- 0 kudos
- 0 kudos
The error where files and directories can be read at the root ADLS level but not at the blob/subdirectory level, combined with a "No file or directory exists on path" message, is frequently due to permission configuration, incorrect path usage, or ne...
- 0 kudos
- 5005 Views
- 1 replies
- 0 kudos
Executing Stored Procedures/update in Federated SQL Server
I have federated Azure SQL DB in my DBX workspace, but I am not able to run update commands or execute a stored procedure, is this still not supported?
- 5005 Views
- 1 replies
- 0 kudos
- 0 kudos
Federated connections from Azure Databricks to Azure SQL DB via Lakehouse Federation currently only support read-only queries—meaning running update commands or executing stored procedures directly through the federated Unity Catalog interface is not...
- 0 kudos
- 2842 Views
- 1 replies
- 0 kudos
System tables for DLT Expectations Quality Metrics
Hi Everyone,I’m working with Delta Live Tables (DLT) and using Expectations to track data quality, but I’m having trouble finding where the expectation quality metrics are stored in the DLT system tables.My questions are:Which specific system table(s...
- 2842 Views
- 1 replies
- 0 kudos
- 0 kudos
@vamsi_simbus DLT captures data quality metrics in specialized system tables known as “event” and “metrics” tables. Specifically, look in the following table:LIVE.DLT_EVENT_LOG or LIVE.DLT_METRICS: These tables contain granular event logs and metric...
- 0 kudos
- 1097 Views
- 1 replies
- 0 kudos
Resolved! What are best practices for designing a large-scale data engineering pipeline on Databricks for real
How do you design a scalable, reliable pipeline that handles both fast/continuous data and slower bulk data in the same system?
- 1097 Views
- 1 replies
- 0 kudos
- 0 kudos
Very generic question Here are general rules and best practices related to Databricks well-architected framework: https://docs.databricks.com/aws/en/lakehouse-architecture/well-architected Take a deeper look on operational excellence, reliability an...
- 0 kudos
- 33588 Views
- 23 replies
- 6 kudos
vocareum lab access
Hi I am doing a data engineering course in databricks(Partner labs) and would like to have access to vocareum workspace to practice using the demo sessions.can you please help me to get the access to this workspace?regards,Aravind
- 33588 Views
- 23 replies
- 6 kudos
- 6 kudos
You can log into databricks, search for "Canada Sales" in the Marketplace and find "Simulated Canada Sales and Opportunities Data". Get free instant access, wait a few seconds for the warehouse to be built for you et voila: the tables for building th...
- 6 kudos
- 6953 Views
- 1 replies
- 0 kudos
Naming jobs in the Spark UI in Databricks Runtime 15.4
I am asking almost the same question as: https://community.databricks.com/t5/data-engineering/how-to-improve-spark-ui-job-description-for-pyspark/td-p/48959 . I would like to know how to improve the readability of the Spark UI by naming jobs. I am...
- 6953 Views
- 1 replies
- 0 kudos
- 0 kudos
You are correct—on Databricks Runtime 15.4 and with shared clusters (or clusters enabled with Unity Catalog), you will see the [JVM_ATTRIBUTE_NOT_SUPPORTED] error when trying to directly access sparkContext attributes that are only available in singl...
- 0 kudos
- 4577 Views
- 1 replies
- 0 kudos
Can we develop a connector that integrates nintex and Databricks community version
Can we develop a connector that integrates nintex and Databricks with community version
- 4577 Views
- 1 replies
- 0 kudos
- 0 kudos
It is technically possible to develop a connector that integrates Nintex with Databricks, but there are important limitations when trying to achieve this with Databricks Community Edition. Connector Integration Overview Nintex can be integrated with...
- 0 kudos
- 5693 Views
- 1 replies
- 0 kudos
Conditional Execution in DLT Pipeline based on the output
Hello ,I'm working on a Delta Live Tables (DLT) pipeline where I need to implement a conditional step that only triggers under specific conditions. Here's the challenge I'm facing:I have a function that checks if the data meets certain thresholds. If...
- 5693 Views
- 1 replies
- 0 kudos
- 0 kudos
In Delta Live Tables (DLT), native conditional or branch-based control flow is limited; all table/stream definitions declared in your pipeline will execute, and dependencies are handled via @Dlt.table or @Dlt.view decorators. You can’t dynamically sk...
- 0 kudos
- 6153 Views
- 1 replies
- 0 kudos
Efficient Parallel JSON Extraction from Elasticsearch in Azure Databricks
Hi Databricks community,I'm facing a challenge extracting JSON data from Elasticsearch in Azure Databricks efficiently, maintaining header information.Previously, I had to use RDDs for parallel extraction, but they're no longer supported in Databrick...
- 6153 Views
- 1 replies
- 0 kudos
- 0 kudos
To efficiently extract JSON data from Elasticsearch in Azure Databricks—while maintaining header information and without reverting to legacy RDD-based parallelization—a few modern Spark-based strategies can be used. Spark DataFrames and Spark SQL sup...
- 0 kudos
- 1533 Views
- 2 replies
- 1 kudos
Resolved! DLT behaving differently when used with python syntax vs when used with sql syntax to read CDF
I was trying t read CDF data of a table as a DLT materialized view.It works fine with sql syntax reading all the columns of the source table along with the 3 CDF columns : _change_type,_commit_timestamp,_commit_version:@dlt.table()def change_table():...
- 1533 Views
- 2 replies
- 1 kudos
- 1 kudos
When accessing Change Data Feed (CDF) data in Delta Live Tables (DLT), the behavior between SQL and Python APIs differs notably regarding CDF metadata columns—_change_type, _commit_timestamp, and _commit_version. SQL Approach (using table_changes):T...
- 1 kudos
- 2803 Views
- 1 replies
- 1 kudos
JSON Medallion Best Practices
I'm looking at ingesting JSON files from an API, pulling a list of orders. Each JSON file has header information and then a nested array of items - I want to flatten this into a table with 1 row/item and the header repeated for every item.What is the...
- 2803 Views
- 1 replies
- 1 kudos
- 1 kudos
I would need to know a little more about your scenario but it makes me remember a similar case I faced. My approach was to use silver layer to create a delta table with enforced schema, standard field names and types, etc. to perform typical actions ...
- 1 kudos
- 2137 Views
- 8 replies
- 5 kudos
Resolved! No rows returned when calling Databricks procedure via .NET API and Simba ODBC driver
I created a simple Databricks procedure that should return a single value."SELECT 1 AS result;"When I call this procedure from my .NET API using ExecuteReader, ExecuteAdapter, or ExecuteScalar, the call completes without any errors, but no rows are r...
- 2137 Views
- 8 replies
- 5 kudos
- 5 kudos
Thank you @mark_ott and @Coffee77 for your help .This has saved me a great deal of time. I now understand that I need to use procedures, functions, or direct SQL queries as a workaround to retrieve data in the .NET API. I will also keep an eye out...
- 5 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
