- 8549 Views
- 11 replies
- 41 kudos
From Associate to Professional: My Learning Plan to ace all Databricks Data Engineer Certifications
In today’s data-driven world, the role of a data engineer is critical in designing and maintaining the infrastructure that allows for the efficient collection, storage, and analysis of large volumes of data. Databricks certifications holds significan...
- 8549 Views
- 11 replies
- 41 kudos
- 41 kudos
@SumitSingh this is getting put in the favourites. Thanks a bunch for this All the best,BS
- 41 kudos
- 8253 Views
- 8 replies
- 7 kudos
My Journey with Schema Management in Databricks
When I first started handling schema management in Databricks, I realized that a little bit of planning could save me a lot of headaches down the road. Here’s what I’ve learned and some simple tips that helped me manage schema changes effectively. On...
- 8253 Views
- 8 replies
- 7 kudos
- 7 kudos
Haha, glad it made sense! Joao.Try it out, and if you run into any issues, just let me know. Always happy to help! And best friends? You got it!
- 7 kudos
- 1174 Views
- 2 replies
- 6 kudos
🔐 How Do I Prevent Users from Accidentally Deleting Tables in Unity Catalog? 🔐
Question:I have a role called dev-dataengineer with the following privileges on the catalog dap_catalog_dev:APPLY TAGCREATE FUNCTIONCREATE MATERIALIZED VIEWCREATE TABLECREATE VOLUMEEXECUTEREAD VOLUMEREFRESHSELECTUSE SCHEMAWRITE VOLUMEDespite this, u...
- 1174 Views
- 2 replies
- 6 kudos
- 6 kudos
Managing assets in UC is always a overhead maintenance. We have this access controls in terraform codes and it is always hard to see what level of access is given to different personas in the org. We are building an audit dashboard for it.
- 6 kudos
- 1170 Views
- 1 replies
- 1 kudos
Databricks Optimization Tips – What’s Your Secret?
When I first started working with Databricks, I was genuinely impressed by its potential. The seamless integration with Delta Lake, the power of PySpark, and the ability to process massive datasets at incredible speeds—it was truly impactful.Over tim...
- 1170 Views
- 1 replies
- 1 kudos
- 1 kudos
1. Try to remove cache() and persist() in the dataframe operations in the code base.2. Fully avoid driver operations like collect() and take() - the information from the executors are brought back to driver, which is highly network i/o overhead.3. Av...
- 1 kudos
- 644 Views
- 0 replies
- 0 kudos
Request for a guest post
Hi, I hope you're doing well. My name is Prasanna. C, Digital Marketing Strategist at Express Analytics, a company that understands consumer behavior and provides analytics solutions and services to businesses. Express Analytics primarily offers...
- 644 Views
- 0 replies
- 0 kudos
- 1260 Views
- 2 replies
- 1 kudos
Automatic Liquid Clustering and PO
I spent some time to understand how to use automatic liquid clustering with dlt pipelines. Hope this can help you as well.Enable Predictive Optimization Use this code:# Enabling Automatic Liquid Clustering on a new table @dlt.table(cluster_by_auto=Tr...
- 1260 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Addy0_, thanks for sharing how to set it for existing table. Unfortunately, I think ALTER cannot be used with materialized view and streaming tables defined in dlt pipelines.I was looking for something similar to @dlt.table(cluster_by_auto=True, ...
- 1 kudos
- 651 Views
- 0 replies
- 1 kudos
Databricks Data Classification
I encourage you to try out a new beta feature in Databricks called : Data Classification. It automatically classifies your catalog data and tag it with tags. Docs: https://docs.databricks.com/aws/en/lakehouse-monitoring/data-classification
- 651 Views
- 0 replies
- 1 kudos
- 728 Views
- 0 replies
- 1 kudos
Strong Databricks Fundamental - Gen Z
Why Databricks is the Future of Data Analytics for Gen ZIn the fast-paced world of data analytics, staying ahead of the curve is crucial. For Gen Z, who are digital natives and always on the lookout for the latest tech trends, understanding the diffe...
- 728 Views
- 0 replies
- 1 kudos
- 3296 Views
- 3 replies
- 0 kudos
Financial Crime detection with the help of Apache Spark, Data Mesh and Data Lake
For those interested in Data Mesh and Data Lakes for FinCrime detection:Data mesh is a relatively new architectural concept for data management that emphasizes domain-driven data ownership and self-service data availability. It promotes the decentral...
- 3296 Views
- 3 replies
- 0 kudos
- 0 kudos
It's great that you're focusing on financial crime detection with advanced technologies like Apache Spark, Data Mesh, and Data Lake. For those looking to dive deeper into criminal records and related data, tools like KY criminal lookup can provide es...
- 0 kudos
- 3438 Views
- 1 replies
- 1 kudos
Post: Lakehouse Federation - Databricks
Lakehouse Federation - Databricks In the world of data, innovation is constant. And the most recent revolution comes with Lakehouse Federation, a fusion between data lakes and data warehouses, taking data manipulation to a new level. This advancement...
- 3438 Views
- 1 replies
- 1 kudos
- 1 kudos
Hey Quick Question, Can we use it for the production version ? We have application server as SQL server, we are planning to use lakehouse federation so we can bypass creating and maintaining 100 of workflows. as we a small dataset I am not too sure o...
- 1 kudos
- 730 Views
- 0 replies
- 1 kudos
Hub Star Modeling 2.0 for Medalion Architecture
Excited to share my latest publication on arXiv!“Hub Star Modeling 2.0 for Medallion Architecture” https://arxiv.org/abs/2504.08788This new version builds on the original Hub Star Modeling approach, published last year, and now tailored for the Meda...
- 730 Views
- 0 replies
- 1 kudos
- 2088 Views
- 1 replies
- 6 kudos
Handling Complex Nested JSON in Databricks Using schemaHints
When I first got into managing schemas in Databricks, it took me a while to realize that putting in a little planning up front could save me a ton of headaches later on.I was working with these deeply nested, constantly changing JSON files. At first,...
- 2088 Views
- 1 replies
- 6 kudos
- 6 kudos
Great tip @genevive_mdonça! schemaHints help avoid issues with evolving JSON data, making data processing more reliable and easier to maintain. Thanks for sharing.
- 6 kudos
- 1442 Views
- 1 replies
- 0 kudos
Understanding Coalesce, Skewed Joins, and Why AQE Doesn't Always Intervene
In Spark, data skew can be the silent killer of performance. One wide partition pulling in 90% of the data?But even with AQE (Adaptive Query Execution) turned on in Databricks, skewness isn't always automatically identified— and here’s why.What Is co...
- 1442 Views
- 1 replies
- 0 kudos
- 0 kudos
@mark_ott , this question seems right up your alley. Care to comment?
- 0 kudos
- 1696 Views
- 0 replies
- 1 kudos
One of the solution of [FAILED_READ_FILE.NO_HINT] Error while reading file, when display() or SELECT
One of the solution of [FAILED_READ_FILE.NO_HINT] Error while reading file, when display() or SELECTI got stuck with the above error when using `spark.read.table().display()` or directly query the table using %sql.While the display method is just one...
- 1696 Views
- 0 replies
- 1 kudos
- 1602 Views
- 0 replies
- 1 kudos
Power BI to Databricks Semantic Layer Generator (DAX → SQL/PySpark)
Hi everyone!I’ve just released an open-source tool that generates a semantic layer in Databricks notebooks from a Power BI dataset using the Power BI REST API. Im not an expert yet, but it gets job done and instead of using AtScale/dbt/or the PBI Sem...
- 1602 Views
- 0 replies
- 1 kudos
