- 3358 Views
- 4 replies
- 1 kudos
Resolved! DLT Pipelines with DABs - Support for tags field?
Hi all,I'm working with DABs and trying to define tags for DLT pipelines in the bundle YAML config. However, adding a `tags:` block under the pipeline results in the following warning: Warning: unknown field: tags This suggests that tags might not be...
- 3358 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @mikvaar, Yes, tags are not supported yet in DABs, but it is in the roadmap. The ETA for this is around first or second week of June.
- 1 kudos
- 7837 Views
- 7 replies
- 0 kudos
Resolved! ODBC data source to connect to a Databricks catalog.database via MS Access Not Working
When using an ODBC data source to connect to a Databricks catalog database via Microsoft Access, the tables are not listing/appearing in the MS Access database for selection.However, when using the same ODBC data source to connect to Microsoft Excel,...
- 7837 Views
- 7 replies
- 0 kudos
- 0 kudos
Why do «Databricks employee» keep answering with the same AI generated reply, when that obviously not is the solution? Has anyone been able to come up with a solution which actually works?
- 0 kudos
- 1403 Views
- 2 replies
- 0 kudos
Databricks job calling DBT - persist job name
Hello all,Is it possible to persist Databricks job name into the Brooklyn audit tables data model when when a Databricks job calls DBT model?Currently, my colleagues persist audit information into fact & dimensional tables of the Brooklyn data model....
- 1403 Views
- 2 replies
- 0 kudos
- 0 kudos
Yes, it’s possible to include the Databricks job name in your Brooklyn audit tables, but it won’t happen automatically. Right now, only the job run ID is being logged, so you’d need to extend your audit logic a bit. One common approach is to pass the...
- 0 kudos
- 2293 Views
- 3 replies
- 2 kudos
Resolved! Autoloader: Trigger batch vs micro-batch (as in .forEachBatch)
Hey everyone,I’m trying to clarify a confusion in AutoLoader regarding trigger batches and micro-batches when using .forEachBatch.Here’s what I understand so far:Trigger batch – Controlled by cloudFiles.maxFilesPerTrigger and cloudFiles.maxBytesPerTr...
- 2293 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @yit ,1. They are not quite the same. Trigger batch defines how many new files Auto Loader lists for ingestion per streaming trigger (this is controlled as you correctly pointed out by cloudFiles.maxFilesPerTrigger and cloudFiles.maxBytesPerTrigge...
- 2 kudos
- 884 Views
- 1 replies
- 0 kudos
Postgress Lakeflow connect
I want to get data from postgress using lakeflow connect for every 10 mins, how to set-up lakeflow connect, can you give step-by-step process, for creating lakeflow connect pipeline?
- 884 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @xavier_db ,Postgres lakeflow connector is currently in private preview according to below thread:Solved: Lakeflow Connect - Postgres connector - Databricks Community - 127633But the thing is I cannot see it in Workspace Preview and Account Previe...
- 0 kudos
- 1426 Views
- 3 replies
- 3 kudos
Advanced Technique
Reduced Monthly Databricks Bill from $47K to $12.7KThe Problem: We were scanning 2.3TB for queries needing only 8GB of data.Three Quick Wins1. Multi-dimensional Partitioning (30% savings)# Beforedf.write.partitionBy("date").parquet(path)# After-parti...
- 1426 Views
- 3 replies
- 3 kudos
- 3 kudos
@ck7007 no worries. I asked a question on the other thread: https://community.databricks.com/t5/data-engineering/cost/td-p/130078 , I'm not sure if you're classing this thread as the duplicate or the other one so I'll repost.I didn't see you mention ...
- 3 kudos
- 1500 Views
- 2 replies
- 2 kudos
Resolved! Read Files from Adobe and Push to Delta table ADLS Gen2
The Upstream is sending 2 files of different schema. The Storage Account has Private Endpoints. there is no public access.no public IP (NPIP) = yes.How to design using only Databricks :-1. Databricks API to read data file from Adobe and Push it to AD...
- 1500 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Pratikmsbsvm ,Okay, since you’re going to use Databricks compute for data extraction and you wrote that your workspace is deployed with the secure connectivity cluster (NPIP) option enabled, you first need to make sure that you have a stable egre...
- 2 kudos
- 5956 Views
- 5 replies
- 2 kudos
Resolved! Managing libraries in workflows with multiple tasks - need to configure a list of libs for all tasks
I have workflows with multiple tasks, each of which need 5 different libraries to run. When I have to update those libraries, I have to go in and make the update in each and every task. So for one workflow I have 20 different places where I have to g...
- 5956 Views
- 5 replies
- 2 kudos
- 2 kudos
Actually I think I found most of a solution here in one of the replies: https://community.databricks.com/t5/administration-architecture/installing-libraries-on-job-clusters/m-p/37365/highlight/true#M245It seems like I only have to define libs for the...
- 2 kudos
- 989 Views
- 1 replies
- 2 kudos
How to work with 300 billions rows and 5 columns?
Hi guys!I'm having a problem at work where I need to process a customer data dataset with 300 billion rows and 5 columns. The transformations I need to perform are "simple," like joins to assign characteristics to customers. And at the end of the pro...
- 989 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @guilhermecs001 ,Wow, that's massive amount of rows. Can you somehow preprocess first this huge CSV file? For example, read CSV, partition by some columns that makes sense (maybe country from which customer is coming from) and save that data as de...
- 2 kudos
- 932 Views
- 1 replies
- 2 kudos
Is Photon Acceleration Helpful for All Maintenance Tasks (OPTIMIZE, VACUUM, ANALYZE_COMPUTE_STATS)?
Hi everyone,We’re currently reviewing the performance impact of enabling Photon acceleration on our Databricks jobs, particularly those involving table maintenance tasks. Our job includes three main operations: OPTIMIZE, VACUUM, and ANALYZE_COMPUTE_S...
- 932 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Sainath368 ,I wouldn't use photon for this kind of task. You should use it primarly for ETL transformations where it shines.VACUUM and OPTIMIZE are more of maintenance tasks and using photon would be pricey overkill here.According to documentatio...
- 2 kudos
- 17627 Views
- 13 replies
- 7 kudos
Value array {{QUERY_RESULT_ROWS}} in Databricks SQL alerts custom template
Please include in documentation an example how to incorporate the `QUERY_RESULT_ROWS` variable in the custom template.
- 17627 Views
- 13 replies
- 7 kudos
- 7 kudos
Databricks confirmed this was an issue on their end and it should be resolved now. It is working for me.
- 7 kudos
- 1632 Views
- 2 replies
- 1 kudos
Resolved! cosmosdb metadata integration with unity catalog
Hi Team,How can we integrate Cosmos DB Meta data with Unity Catalog, can you please provide some insights on this?Regards,Phani
- 1632 Views
- 2 replies
- 1 kudos
- 1 kudos
Hello @Phani1 Good day:I have found a whole document on your requirementshttps://community.databricks.com/t5/technical-blog/optimising-data-integration-and-serving-patterns-with-cosmos-db/ba-p/91977 It has a project with it as well.
- 1 kudos
- 1060 Views
- 1 replies
- 0 kudos
Resolved! How to build Data Pipeline to consume data from Adobe Campaign to Azure Databricks
May Techie please help me design the pipeline with Databricks.I don't have any control over Adobe.How to set up a data pipeline that moves csv files from Adobe to ADLS Gen2 via a cron job, using Databricks.where this cron job will execute ? how ADLS ...
- 1060 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Datalight Good day!Can I please know what do you mean by "you dont have any control over adobe"? I found similar case study over here: https://learn.microsoft.com/en-us/answers/questions/5533633/data-pipeline-to-push-files-from-external-system...
- 0 kudos
- 2900 Views
- 8 replies
- 5 kudos
Resolved! Cannot deploy DAB with the Job branch using a feature branch in Workspace UI
Hi, I tried to deploy DAB on Workspace UI with a feature branch (sf-trans-seq) targeted to Dev. After deploying successfully, the Job branch is, however, using the master branch (see the screenshot below).Is there any option to force the Job branch t...
- 2900 Views
- 8 replies
- 5 kudos
- 5 kudos
I agree.Can you mark your (or someone else´s) answer as solved? Because I think you won´t be the only one with this issue/feature.
- 5 kudos
- 1726 Views
- 1 replies
- 1 kudos
Resolved! Mongodb connection in GCP Databricks
I am trying to connect with Mongodb from databricks which is UC enabled, and both the mongodb and databricks are in same VPC, I am using the below code, df = ( spark.read.format("mongodb") .option( "connection.uri", f'''mongodb://{username}:{password...
- 1726 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @xavier_db ,Standard access mode has more limitations compared to dedicate access mode. For example, look at the limitations list of standard access mode:Standard compute requirements and limitations | Databricks on AWSNow, compare it to dedicated...
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 585 | |
| 349 | |
| 287 |
