Have you earned a Databricks certification or thought about getting one? Share your success story and inspire others! Got questions or challenges about certifications? Let’s help each other out. Whether you’re certified or just curious, your journey matters—let’s discuss! ... View more

We’re curious to hear how you’re using Databricks in your projects. Whether it’s an innovative use case or just an example of everyday work, share your story and inspire others. Let us know in the comments below! ... View more

As we kick off the new year, let’s set the tone for an exciting 2025! Drop an emoji in the comments to share your vibe: 🔥 = Ready to build cutting-edge data pipelines! 📚 = Excited to master new Databricks features! 💡 = Innovating with data and AI solutions! 🌍 = Connecting and collaborating with the global Databricks community! What’s your 2025 Databricks vibe? Let’s hear it below! ... View more

In most Lakehouse implementations, teams (mine included) have traditionally used Schemas to segregate tables for the Medallion Architecture—Bronze, Silver, and Gold. However, a major challenge was that when declaring a catalog and schema in Delta Live Tables (DLT), all tables and views had to be registered in the same schema. This meant either: Keeping all tables from Bronze to Gold in one schema—resulting in messy, unorganized data. Creating multiple separate pipelines just to maintain proper segregation Inability to tack pipeline progress and dependencies via the graph But now, with this new Databricks release, users can register tables to different catalogs and schemas in Unity Catalog!  Key Highlights: Automatic Enablement for New Pipelines: Any new pipelines created through the UI will have this functionality enabled by default (if the preview is enabled). However, existing pipelines cannot be converted to utilize this feature. To migrate, you must create a new pipeline and re-ingest data from the source(s). Fully Qualified Table Names for Streaming Tables: When creating streaming tables or views, you must register them using the fully qualified path (catalog.schema.table). This ensures that subsequent processes can accurately read the data. Schema Configuration: Because you have to use the full qualified path, the schema where the pipeline is defined cannot be the same as the schema where the tables are registered. Attempting to register tables in the same schema will result in an error. Pipeline Creation via API: If you're creating the pipeline using the API, to register it as a Unity Catalog Pipeline, you need to use the schema object in your JSON configuration instead of the previous target parameter. Using target will configure the pipeline in the old mode. Why This Matters: Maintain a clean Medallion Architecture without the need for multiple pipelines. Organize tables into separate schemas for enhanced governance. Simplify access control and multi-environment management. How do you plan to leverage this update? Let's discuss how this can enhance data governance and streamline your pipeline designs! ... View more

What's Your Databricks Aha! Moment? 🌟 Whether you’re new to Databricks or a seasoned pro, there’s always that one moment when everything just clicked. Maybe it was when you nailed a tricky notebook, optimized a query, solved a big data challenge, or realized how collaboration can change the game. We’d love to hear your Databricks Aha Moment! Share your story in the comments below and inspire others! Let’s celebrate those breakthroughs together. 🚀 ... View more