- 13868 Views
- 12 replies
- 4 kudos
Resolved! ImportError: cannot import name 'AnalyzeArgument' from 'pyspark.sql.udtf'
Hello community,I installed databricks extension on my vscode ide. How to fix this error? I created the environment to run locally my notebooks and selected the available remote cluster to execute my notebook, what else?I Have this error: ImportError...
- 13868 Views
- 12 replies
- 4 kudos
- 4 kudos
@unj1m yes, as Alberto said you don't need to install pyspark, it is included in your cluster configuration.
- 4 kudos
- 456 Views
- 2 replies
- 2 kudos
Choosing between a materialized view and a job-maintained Delta table for an expensive SQL query
I am relatively new to Databricks and do not yet have much experience with materialized views, Delta tables, incremental processing, or the computational costs associated with refreshing persisted results.I currently have a Databricks SQL view that c...
- 456 Views
- 2 replies
- 2 kudos
- 2 kudos
Adding to the above, can do one small validation before deciding between MV and a custom Delta/MERGE solution.Databricks now lets you check whether the MV query is structurally eligible for incremental refresh:EXPLAIN CREATE MATERIALIZED VIEW mv_orde...
- 2 kudos
- 464 Views
- 2 replies
- 0 kudos
Are attached-but-idle serverless notebooks billed? Idle timeout? 32 min job, 86 min billed
Hi all, hope everyone is doing well. Two questions I am hoping someone can help with: is there a way to set an idle timeout for serverless notebooks, and is an attached-but-idle serverless notebook billed while it sits there?Some background on why I ...
- 464 Views
- 2 replies
- 0 kudos
- 0 kudos
@Louis_Frolio Thanks, this clarifies the idle billing behaviour.I also observed a very similar pattern in another Azure workspace using the EU West serverless all-purpose SKU. The last visible notebook execution was around 09:56 UTC, while system.bil...
- 0 kudos
- 557 Views
- 2 replies
- 0 kudos
Getting started with Delta Lake basics
Hi all — I am working through the Delta Lake intro and wanted to share a short note on using OPTIMIZE after merge. Happy to hear better patterns.
- 557 Views
- 2 replies
- 0 kudos
- 0 kudos
@rachelnguyen MERGE operation is inherently expensive as it searches the entire target table to find matches, then shuffles data multiple times to write updates/inserts. It generates many small files - especially on partitioned tables where each shuf...
- 0 kudos
- 348 Views
- 0 replies
- 1 kudos
Databricks architecture drawn step by step: control plane, compute plane, Delta, Unity Catalog
I kept getting the same interview question, "explain the Databricks architecture", and most answers are a wall of text. So I drew it.The crisp answer: Databricks manages the control plane (web app, job scheduler, notebook metadata). Your compute runs...
- 348 Views
- 0 replies
- 1 kudos
- 785 Views
- 2 replies
- 2 kudos
Resolved! ConcurrentAppendException on parallel MERGE into the same Delta table, despite disjoint keys
Hi everyone, Hoping someone has been through this, because I am running out of ideas. Setup. DBR 15.4 LTS, Unity Catalog managed Delta table, roughly 900 million rows. Six upstream sources each write into the same target table through their own job, ...
- 785 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Islam_hoti This is a classic "Optimistic Concurrency Control" bottleneck in Delta Lake. Even with partitions, the MERGE operation requires a metadata-level lock on the table's transaction log, which is where your contention is happening.Here is a...
- 2 kudos
- 573 Views
- 2 replies
- 4 kudos
Resolved! AUTO CDC FROM SNAPSHOT schema evolution in history tracking columns
Does the create_auto_cdc_from_snapshot_flow() allow for expansion of included columns in track_history_column_list or track_history_except_column_list without doing a full refresh? My intended flow is: Bronze - Managed ingestion of table (scd 1) usin...
- 573 Views
- 2 replies
- 4 kudos
- 4 kudos
Adding a column and changing the history-tracking configuration need separate checks. Databricks describes adding columns as generally safe without a full refresh:"Plan your schema to avoid changes that require a full refresh. Adding columns is gener...
- 4 kudos
- 611 Views
- 4 replies
- 0 kudos
AI/BI Dashboards
Hi AllMy client is looking to create a few dashboards in DataBricks instead of Power BI. We already have a template in Power BI (header for each page, information about out BI Page) in each of our power bi report which has colors, fonts, images etc.H...
- 611 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @km1837, Databricks AI/BI Dashboards have solid theming and organisational capabilities that should cover most of what you are used to in Power BI. AI/BI Dashboards don't have a "template file" concept the way Power BI does with .pbit files, but y...
- 0 kudos
- 881 Views
- 4 replies
- 1 kudos
Resolved! Auto Loader stream fails on RocksDB checkpoint after enabling managed file events
Hello Community.We recently enabled the following option on several existing Auto Loader streams:"cloudFiles.useManagedFileEvents": "true"Most streams continued working, but one now fails while restoring its existing checkpoint, referencing a file si...
- 881 Views
- 4 replies
- 1 kudos
- 1 kudos
After further investigation into it, have understood the root cause and the fix for it.After enabling Auto Loader managed file events across the platform: .option("cloudFiles.useManagedFileEvents", "true")the failed stream started discovering RocksDB...
- 1 kudos
- 968 Views
- 3 replies
- 2 kudos
Resolved! Databricks Apps - Deployment process - Need help !!
Hi all,Do we have any guidelines or a CI/CD approach for deploying a Databricks app from one environment (e.g., DEV) to PROD?We are currently facing the following challenge:We created a Databricks app in the DEV environment through the UI. At runtime...
- 968 Views
- 3 replies
- 2 kudos
- 2 kudos
Thank you for the details. Yes, I agree we should adopt the DABs for the deployment and define the resources while creating the app ( which can be consider as as solution). However if the permission module is managed separately, lets say via terrafor...
- 2 kudos
- 968 Views
- 2 replies
- 3 kudos
Resolved! Best Practice for Handling Schema Evolution with Auto Loader in Production?
Hi Databricks Community,I’d like to hear how other Data Engineers are handling schema evolution with Auto Loader in production environments.Consider the following scenario:We have a continuously running ingestion pipeline using Auto Loader that proce...
- 968 Views
- 2 replies
- 3 kudos
- 3 kudos
You can absolutely leverage Databricks Genie and the new Genie Ontology to handle downstream changes caused by addNewColumnsWithTypeWidening, and handle the changes using Genie ontology, but it requires understanding the division of labor between you...
- 3 kudos
- 484 Views
- 1 replies
- 0 kudos
OAuth M2M (client-credentials) - getting error with github run
I'm setting up OAuth M2M (client-credentials) authentication for a service principal, as the long-term replacement for PAT-based auth in a GitHub Actions CI/CD workflow. The token exchange itself succeeds and returns a valid, well-formed access token...
- 484 Views
- 1 replies
- 0 kudos
- 0 kudos
further refined this problem with below if anyone has any idea how I could address? thanks. Later in troubleshooting, this same should_change_password error started appearing on a personal-account PAT as well (not just the SP's OAuth token) — so what...
- 0 kudos
- 379 Views
- 2 replies
- 0 kudos
How to create monotonic function to incrementally add obj_id for datasource in pyspark
I have used monotonically_increasing_id() to add unique to datasource .However , this does not add unique id in increasing number ..Its kind of random.How to perform this in pyspark.
- 379 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey So monotonically_increasing_id() only promises the numbers go up and are unique, not that they're neat like 1, 2, 3. Under the hood it bakes the partition number into the ID, so each partition jumps ahead by billions.If you let me know what you'...
- 0 kudos
- 667 Views
- 2 replies
- 4 kudos
Resolved! Managing Service Principal Permissions at Scale: External Locations vs. Managed Volumes
Hi community,I’m currently architecting a multi-workspace environment under Unity Catalog, and I’m looking for the most sustainable way to manage Service Principal access to raw data landing zones.We are debating between using External Locations (sco...
- 667 Views
- 2 replies
- 4 kudos
- 4 kudos
Hi,On question one, I would reframe the choice rather than answer it directly. Managed volumes sit in Databricks controlled storage, which means an external producer cannot write into them through Unity Catalog. If your landing zone is fed by systems...
- 4 kudos
- 719 Views
- 2 replies
- 3 kudos
Resolved! Migrating Large-Scale Z-Ordered Tables to Liquid Clustering: Strategies for Production Pipelines
Hi everyone,As we look to modernize our Delta tables, I’m evaluating the move from Z-Ordering to Liquid Clustering, particularly for our historical datasets that exceed 500TB.While the benefits of Liquid Clustering (avoiding over-partitioning and bet...
- 719 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi,The good news is that the migration is much cheaper than you are assuming, because the full rewrite is optional.When you run ALTER TABLE with CLUSTER BY, existing data is not rewritten. Subsequent writes and normal OPTIMIZE runs use the new cluste...
- 3 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
