- 1975 Views
- 7 replies
- 0 kudos
Cannot Remove Table from Delta Share
Hello! We have some production data pipelines using delta shares that have previously worked but broken today and I can't find a new solution. I create a delta share: CREATE SHARE IF NOT EXISTS `local_new_2022` Then, like normal, I add a table to th...
- 1975 Views
- 7 replies
- 0 kudos
- 0 kudos
We are using the databricks_sql python sdk: ```pythonfrom databricks import sql as databricks_sqlconnection = databricks_sql.connect( server_hostname=server_hostname, http_path=http_path, access_token=access_token)with connection.cursor() as...
- 0 kudos
- 882 Views
- 2 replies
- 3 kudos
Unity Catalog not detected until I manually restarted the cluster
I just spent a lot of time setting up Unity Catalog with external locations, access connectors, and all the right permissions. My cluster had Dedicated access mode, Unity Catalog has been assigned properly, and everything was configured correctly. Bu...
- 882 Views
- 2 replies
- 3 kudos
- 3 kudos
Appreciate you posting this, @Dharshan15! It’s something that can easily be overlooked. After enabling Unity Catalog, the cluster needs to be restarted for the changes to take effect. You can also share this feedback directly with the Databricks team...
- 3 kudos
- 547 Views
- 1 replies
- 0 kudos
How to get a programmatically get he non dlt pipeline logs.
We are triggering the jobs which notebooks. Notebook is consist of python and sql queries. We need to read the error messages when it is triggered using an job thanksprasanna
- 547 Views
- 1 replies
- 0 kudos
- 0 kudos
Use some logging to send the error logs to a delta table or file. Can you please share the code. I wanted to see what you are trying to do.
- 0 kudos
- 3060 Views
- 5 replies
- 3 kudos
Resolved! Problems and questions with deploying Lakeflow Declarative Pipeline using Databricks Bundles
Hi all,I met some problems and have some questions about deploying Lakeflow Declarative Pipeline using Databricks Bundles. Could anyone kindly help?Below is my current bundle resource file for the pipeline: resources: pipelines: dbr_d365_crm_p...
- 3060 Views
- 5 replies
- 3 kudos
- 3 kudos
Hi @AlbertWang ,I think some of those issues could be related to your databricks assets bundle version. For example the glob thing is in Beta. It could be available in UI, but not in your version of databricks cli.The same applies for root_path: As o...
- 3 kudos
- 1145 Views
- 1 replies
- 3 kudos
Data Engineering Associate Exam Course Content Not Aligned With Exam Guide
Recently Databricks has changed the exam guide content for Data Engineering Associate Certification exam. The sections under Exam Outline list Data Asset Bundles too which is neither part of the e-learning course material and nor the ongoing instruct...
- 1145 Views
- 1 replies
- 3 kudos
- 3 kudos
Thanks for bringing this to light, @junaid-databrix!Could you please raise a ticket with the Databricks Support team? That will help ensure this reaches the appropriate team for review. While creating the ticket, kindly mention which certification ex...
- 3 kudos
- 1890 Views
- 2 replies
- 0 kudos
Delta sharing json predicate doesn't work
I'm trying to push predicates via python delta_sharing pkg: https://github.com/delta-io/delta-sharing/tree/main/python/delta_sharingand delta sharing protocol: https://github.com/delta-io/delta-sharing/blob/main/PROTOCOL.md#json-predicates-for-filter...
- 1890 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @drag7ter ,PredicateHints are just hints, not enforced filters. They do not guarantee that the returned data will be filtered.Check below thread for detailed discussion. Also check if your delta server has evaluate predicate hints flag set to true...
- 0 kudos
- 1536 Views
- 3 replies
- 0 kudos
Resolved! flattening json in dlt pipeline
Hi,I have in my bronze schema json files, I am flattening them in a dataframe after that I am creating materialized views in a dlt pipeline, however, in production is taking a lot of time (over 3 hours) is not even a lot of data the biggest materiali...
- 1536 Views
- 3 replies
- 0 kudos
- 0 kudos
Hello @carlos_tasayco As I mentioned whether you used any join, just wanted to ask, were this cross join were joining two large tables?
- 0 kudos
- 1531 Views
- 3 replies
- 0 kudos
Databricks Workflow Orchestration for Pipeline
Hello,I am using Databricks first time. May someone please help me how to do orchestration for the pipeline shown below.Kindly share the steps how to implement Orchestration , what all steps we have to consider.Thanks a lot
- 1531 Views
- 3 replies
- 0 kudos
- 0 kudos
The diagram you have shared is bit confusing: From Azure there is a data pull to Bronze layer, and from the same data source data is being pulled into Silver layer. However, following the Medallion architecture typically the raw data is ingested into...
- 0 kudos
- 4591 Views
- 7 replies
- 5 kudos
Resolved! Are there any student discounts or coupons for the Databricks Data Engineer Associate certification
I’m planning to take the Databricks Data Engineer Associate certification exam, and I wanted to ask if there are any official discounts, coupons, or student offers available to help reduce the exam fee.I’m a student right now, so any discount or prom...
- 4591 Views
- 7 replies
- 5 kudos
- 3742 Views
- 3 replies
- 4 kudos
Auto Loader Schema Hint Behavior: Addressing Nested Field Errors
Hello,I'm using the auto loader to stream a table of data and have added schema hints to specify field values.I've observed that when my initial data file is missing fields specified in the schema hint,the auto loader correctly identifies this and ad...
- 3742 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi, we are having similar issues with schema hints formulated in fully qualified DDL, e.g. "a STRUCT<b INT>" etc. Did you find a solution? Also, did you specify the schema hint using the dot-notation, e.g. "a.b INT" before ingesting any data or after...
- 4 kudos
- 967 Views
- 1 replies
- 0 kudos
Performance Behavior of MERGE with Partitioned Table: Impact of ZORDER and Deletion Vectors
Hi Databricks Community,I’m analyzing the performance of Delta Lake MERGE operations on a partitioned table, and I observed unexpected behavior across 3 test cases.I wanted to share my findings to better understand:Why ZORDER or Deletion Vectors help...
- 967 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @pooja_bhumandla Thanks for such a nice and detailed description of Your case, that really helps to understand the scenario Regarding Your questions:1)Overall operation could become more complex due to:a) deletion vector creation and maintenance,b...
- 0 kudos
- 1957 Views
- 1 replies
- 2 kudos
Delta Lake File Sizes - optimize maxfilesize, tunefilesizesforrewrites
hello allwhile reading content that provides guidance on delta lake file size, i realized tuneFileSizesForRewrites behind the scenes targets for 256 MB file size.and optimize.maxFileSize will target for 1 GB file ((reference : https://docs.databricks...
- 1957 Views
- 1 replies
- 2 kudos
- 2 kudos
hello @noorbasha534 That's very interesting topic regarding fine-tuning file sizes under delta table.Answering Your questions:1)I use spark.databricks.delta.optimize.maxFileSize and set maximum file size for optimize command. Its working for me just ...
- 2 kudos
- 1109 Views
- 2 replies
- 3 kudos
Resolved! Bug Report: SQL Editor, Run Now button run query from different query file/tab
When I click Run Now button or press (Command + Enter), the editor somehow executed query from another query file/tab. I can only use Run Selected for now. Clear cache and re-login didn't solve the problem.Please see the screen recording herehttps://...
- 1109 Views
- 2 replies
- 3 kudos
- 3 kudos
Hello @Bank_Kirati! Thanks for sharing the screen recording. I’m unable to reproduce the issue on my end. Could you try toggling the New SQL Editor off and on, and see if that makes a difference? Also, please check if this happens only with specific ...
- 3 kudos
- 789 Views
- 1 replies
- 1 kudos
Partner Academy Login Issue
Hi Team, I mam facing the issue with the partner academy login.
- 789 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @Nidhig! From the screenshot, it appears you're not currently authorized to access the Partner Academy. In this case please raise a ticket with the Databricks Support Team. They’ll be able to investigate further and assist you with gaining acce...
- 1 kudos
- 1768 Views
- 3 replies
- 1 kudos
Resolved! Error Logging and Orchastration In Databricks
Hello,May someone please Help me designing the Error Logging and How to do orchestration for Pipeline below.I am pulling data from Bronze layer and pushing it to silver layer after transformation.1. How to do Error Logging and where to store2. How to...
- 1768 Views
- 3 replies
- 1 kudos
- 1 kudos
@Brahmareddy : Thanks a lot. do you have any page which shows real implementation, if handy. Kindly share.
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 554 | |
| 349 | |
| 287 |
