- 4739 Views
- 5 replies
- 2 kudos
Exclude a job from bundle deployment in PROD
My question is regarding Databricks Asset Bundles. I have defined a databricks.yml file the following way: bundle: name: my_bundle_name include: - resources/jobs/*.yml targets: dev: mode: development default: true workspace: ...
- 4739 Views
- 5 replies
- 2 kudos
- 2 kudos
Me too. No clean solution yet. As workaround I implemented first an "extra" control in specific jobs that never should be run in PROD to block execution based on environment variable in all clusters (I don't really like much but it was effective). As...
- 2 kudos
- 4158 Views
- 10 replies
- 4 kudos
Resolved! High Level Design for Transfer Data from One Databricks account to Another databricks account
Hi,May someone please help me with only Points which should be part of High Level Design and Low Level Design when transfering Data from One Databricks account to Another databricks account using Unity Catalog. First time full data transfer and than ...
- 4158 Views
- 10 replies
- 4 kudos
- 4 kudos
Based on my previous reply, you can use DEEP CLONE to clone data incrementally between workspaces by including it in a scheduled job but this will not work in real time indeed.
- 4 kudos
- 6384 Views
- 11 replies
- 10 kudos
DLT - SCD 2 - detect deletes
Hello,I have a question related to APPLY AS DELETE WHEN...If the source table does not have a column that specifies whether a record was deleted, I am currently using a workaround by ingesting synthetic data with a soft_deletion flag. In the future, ...
- 6384 Views
- 11 replies
- 10 kudos
- 10 kudos
ok. In my case I am qualified for: Incremental without a delete flag (classic case)Generate synthetic tombstones via an anti-join between the current set of keys and the target’s active keys.I don't want to use Merge, that's why my question was for C...
- 10 kudos
- 6543 Views
- 2 replies
- 3 kudos
Resolved! Opinions/Thoughts: SQL Best Practices in Production .. DBT vs DLT ?
Hey everyone, I'd like to hear the experiences of the community on DLT (Lakeflow declarative pipelines) vs DBT. 1. Why would one choose one instead of the other? 2. How does picking one of these level up your SQL strategy?I am somebody who's well-ver...
- 6543 Views
- 2 replies
- 3 kudos
- 3 kudos
That's a cracking write up @szymon_dybczak. Thanks for that .That's certainly given me some food for thought. I think the safest option here, at least for me, is digging into both of them. I feel better informed moving forward with this. I'd love to ...
- 3 kudos
- 4082 Views
- 11 replies
- 5 kudos
Ensuring Row Order When Importing CSV with COPY INTO
Hi everyone,I have a CSV file stored in S3, and it's critical for my process that the rows are loaded in the exact order they appear in the file.Does the COPY INTO command preserve the original row order during the load? I need to make sure the bronz...
- 4082 Views
- 11 replies
- 5 kudos
- 1468 Views
- 1 replies
- 0 kudos
Suggestion: allow existing_cluster_id override from run_job_task and for_each_task
I'm not sure if this feature exists in newer versions of databricks CLI (doubtful because this doesn't seem possible in the UI either but if it helps my team has been on 0.222.0 for a while because it has been stable enough for us) and maybe this is ...
- 1468 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @kenmyers-8451, I’m also not sure if this feature exists in newer versions. Since no one else has replied, I’d suggest raising a ticket with the Databricks Support Team — they’ll be able to provide clarity on this topic:http://help.databricks.com/...
- 0 kudos
- 1690 Views
- 4 replies
- 1 kudos
spark excel reading custom cell
Hi all,some one help me to read excel with custom format cells _(* #,##0_);_(* (#,##0);_(* "-"??_);_(@_) from databricks using spark.excel read
- 1690 Views
- 4 replies
- 1 kudos
- 1 kudos
I don't think this should be too hard to handle with a python library @SebastianRowan. I'm happy to take a look if you could provide an example of where it doesn't work .All the best,BS
- 1 kudos
- 6773 Views
- 4 replies
- 0 kudos
Resolved! Reference file name when using COPY INTO?
When using the COPY INTO statement is it possible to reference the current file name in the select staement? A generic example is shown below, hoping I can log the file name in the target table.COPY INTO my_table FROM (SELECT key, index, textData, ...
- 6773 Views
- 4 replies
- 0 kudos
- 0 kudos
In sql, with a subselect works just fine %sql COPY INTO tabele_copy_into_test from ( SELECT user_id, email, _metadata.file_name AS source_filefrom "/Volumes/dbacademy_ecommerce/v01/raw/users-historical/")FILEFORMAT = PARQUETCOPY_OPTIONS ('m...
- 0 kudos
- 2783 Views
- 5 replies
- 10 kudos
Databricks Hungary Meetup
Details:Free event, registration is required: https://datapao.com/databricks-hungary-meetup/SUMMARYJoin DATAPAO for the autumn edition of our Databricks Hungary Meetup. Join us and our exciting guests from Databricks for talks, product updates and ne...
- 2783 Views
- 5 replies
- 10 kudos
- 10 kudos
@WiliamRosa if you make any material available after the meeting, I'd be interested to take a look ! I hope you guys have a great time. All the best,BS
- 10 kudos
- 2465 Views
- 4 replies
- 4 kudos
Cobrix Library
I need cobrix package free in my account. How do I install it, as there is only 1 compute. Do I need to email admin? Also, I am not able open any support ticket for this
- 2465 Views
- 4 replies
- 4 kudos
- 4 kudos
HI @PriyankaM1 ,Assuming you're using Unity Catalog compute cluster with Dedicated access mode (formerly called single user), to install that library you need to:- grab maven coordinates -> za.co.absa.cobrix:spark-cobol_2.13:2.8.4- go to Compute sect...
- 4 kudos
- 2315 Views
- 5 replies
- 1 kudos
EXPLAIN PLAN parser
hello all,have you come across a freely available parser that does good job for parsing explain plans for SQL queries in databricks...
- 2315 Views
- 5 replies
- 1 kudos
- 1 kudos
got it. If execution is not an option, you can still extract columns without running the queries by parsing their AST.Use Spark’s internal SQL parser (no execution)You can parse to an unresolved logical plan and walk the tree for Filter, Join, and Ag...
- 1 kudos
- 8163 Views
- 6 replies
- 1 kudos
Migrate transformations from snowflake to databricks
I want to migrate my data as well as the transformations i have to convert my raw data into BI data. Is there a way i can move those transformation to databricks? Some of the transformations use native snowflake functions.Thanks in advance.
- 8163 Views
- 6 replies
- 1 kudos
- 1 kudos
Hi @thelogicplus, I’ve also used Travinto Technologies’ tool for this kind of migration.
- 1 kudos
- 2061 Views
- 1 replies
- 0 kudos
Clarification on how Streaming Backlog Duration & Records are calculated
Hi all,I’m working on preparing a dashboard for streaming observability and I’m trying to understand how some of the backlog metrics shown in Databricks are actually calculated.In particular, I’m looking at:Streaming Backlog (records): described as t...
- 2061 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @saicharandeepb,Here’s what I found after reproducing this in Databricks (Auto Loader and the rate source) and inspecting lastProgress.TL;DRStreaming Backlog (records) = how many input records are still discoverable but not yet committed by the co...
- 0 kudos
- 754 Views
- 1 replies
- 0 kudos
For each: nested JAR task can’t reference upstream taskValues
I’m creating a For each with a nested JAR task. init_job (notebook) sets dbutils.jobs.taskValues.set("my_value", ...), and the For each container depends on init_job.but the UI says: “Reference … can only be used in a task that depends on task 'init_...
- 754 Views
- 1 replies
- 0 kudos
- 0 kudos
Yep—this is a current Workflows limitation. Task value refs ({{tasks.<key>.values.<name>}}) only work if the consuming task directly depends on the producer. A For Each/group dependency doesn’t count—nested tasks don’t inherit those edges for interpo...
- 0 kudos
- 1466 Views
- 1 replies
- 1 kudos
Resolved! Databricks Badge Challenge
Hi Team,As part of badges challenge from Databricks for partners. The criteria is the badge from Sales/Pre-Sales not from Tech-Sales?
- 1466 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @Nidhig! Please raise a ticket with the Databricks Support Team, they’ll be able to provide clarity on this topic.
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
