- 2371 Views
- 6 replies
- 8 kudos
From Associate to Professional: My Learning Plan to ace all Databricks Data Engineer Certifications
In today’s data-driven world, the role of a data engineer is critical in designing and maintaining the infrastructure that allows for the efficient collection, storage, and analysis of large volumes of data. Databricks certifications holds significan...
- 2371 Views
- 6 replies
- 8 kudos
- 8 kudos
This is great! I have worked with Databricks for almost three years and has decided to pursue the Databricks Engineer Professional certification. This will certainly help setting up an effective plan.
- 8 kudos
- 715 Views
- 0 replies
- 2 kudos
Code patterns / best practices in SparkSQL, pyspark and Scala
Hi all,I am looking for collections of Code patterns for:SparkSQLpySparkScalaI'm sure there are at least a few repos in Github with snippets and will share them as I find them in this thread. If you come across any good collections, please post your...
- 715 Views
- 0 replies
- 2 kudos
- 1453 Views
- 0 replies
- 2 kudos
Cross-filtering for AI/BI dashboards
AI/BI dashboards now support cross-filtering, which allows you to click on an element in one chart to filter and update related data in other charts.Cross-filtering allows users to interactively explore relationships and patterns across multiple visu...
- 1453 Views
- 0 replies
- 2 kudos
- 483 Views
- 0 replies
- 2 kudos
SHOW CREATE TABLE for materialized views and streaming tables
Added support for SHOW CREATE TABLE for materialized views and streaming tables. This will show the complete CREATE command used at creation time, including properties and schedulesCatalog Explorer "Overview" tab now shows the full CREATE command for...
- 483 Views
- 0 replies
- 2 kudos
- 1472 Views
- 0 replies
- 2 kudos
My Journey with Schema Management in Databricks
When I first started handling schema management in Databricks, I realized that a little bit of planning could save me a lot of headaches down the road. Here’s what I’ve learned and some simple tips that helped me manage schema changes effectively. On...
- 1472 Views
- 0 replies
- 2 kudos
- 2289 Views
- 0 replies
- 2 kudos
LakeFlow Connect
LakeFlow Connect introduces simple ingestion connectors for databases, enterprise applications, and file sources. Set up efficient, low-maintenance pipelines in just a few clicks or via an API. Current sources include Salesforce, Workday, and SQL Ser...
- 2289 Views
- 0 replies
- 2 kudos
- 2106 Views
- 2 replies
- 3 kudos
Resolved! Timeout handling with JDBC connection to SQL Warehouse
We have tried to build a connection test logic to our software to try out the reachability of the SQL Warehouse, yet the connection parameters do not seem to function in expected manner.When the SQL Warehouse is running, the connection test functions...
- 2106 Views
- 2 replies
- 3 kudos
- 3 kudos
Hello, To create the connection you would need an endpoint, I would suggest you to give Serverless warehouse a try so that you don not have to wait, and for the suggestion on the product you may also submit a feedback and share the details of use cas...
- 3 kudos
- 776 Views
- 0 replies
- 1 kudos
Databricks java.util.NoSuchElementException: None.get Error on "SHOW TABLES IN" command
This is not a question, this is just the solution to a problem we encountered in case someone from the community finds it useful.Recently we encountered an issue, where our users' jobs started failing out of nowhere on the following command, with the...
- 776 Views
- 0 replies
- 1 kudos
- 3083 Views
- 0 replies
- 2 kudos
- 3083 Views
- 0 replies
- 2 kudos
- 515 Views
- 0 replies
- 1 kudos
Terraform support for AI/BI dashboards
AI/BI dashboards can now be managed through Terraform.Dashboard using serialized_dashboard attribute: data "databricks_sql_warehouse" "starter" { name = "Starter Warehouse" } resource "databricks_dashboard" "dashboard" { display_name = "...
- 515 Views
- 0 replies
- 1 kudos
- 2764 Views
- 3 replies
- 4 kudos
Introducing Databricks LakeFlow: A New Solution for Data Engineering
Hello Everyone!!!I hope everyone knows about the exciting new launch from Databricks: LakeFlow. This innovative tool makes data engineering simpler and more efficient. Let’s understand what LakeFlow offers and how it can help data teams. Let’s unders...
- 2764 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @nihalataskeen Thankyou and I have signed up in the waitlist. Any idea when can we might get access to this??
- 4 kudos
- 723 Views
- 0 replies
- 0 kudos
Databricks User interface updates
AI/BI dashboards:Improvements:The assistant can now help you edit chart axis titles and control the visibility of chart data labels.You can now assign multiple measures to pivot tables.Added more relative datetime options, such as Last 10 seconds and...
- 723 Views
- 0 replies
- 0 kudos
- 3649 Views
- 2 replies
- 0 kudos
Resolved! DevOps CICD for Databricks Schema
Does anyone know how to do DevOps CICD pipeline to deploy changes to a Databricks Catalog? I've added a couple tables in DEV and need to deploy in Prod. Could'nt find any info on this anywhere.
- 3649 Views
- 2 replies
- 0 kudos
- 0 kudos
Thanks @Retired_mod , the links refer mostly deployments of workspace objects rather than changes in database objects, eg. a new schema, table, or new column added to a table etc. I'll keep looking...
- 0 kudos
- 1177 Views
- 1 replies
- 1 kudos
Databricks Terraform Authentication Methods
Databricks, a unified data analytics platform, is widely recognized for its ability to process big data and run complex algorithms. When it comes to infrastructure as code, Terraform stands out as a tool that can manage Databricks resources efficient...
- 1177 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Ajay-Pandey , Thank you for sharing this comprehensive guide on managing Databricks resources with Terraform. The detailed explanation of the different authentication methods and examples is very helpful. I appreciate you taking the time to put t...
- 1 kudos
- 3011 Views
- 0 replies
- 0 kudos
Reading modes in spark
Hello Everyone, 3 type of reading mode into pyspark:1.Permissive Mode : its a default mode. if spark is not able to parse it due to dattype mismatch then make it as null without impacting other result, i.e only issue record are null others are fine.2...
- 3011 Views
- 0 replies
- 0 kudos
- 668 Views
- 0 replies
- 0 kudos
Lakehouse Monitoring GA: Profile, diagnose, and enforce data quality
Lakehouse Monitoring provides automated profiling and a dashboard that visualizes trends and anomalies over time. Track key metrics such as data volume, percent nulls, numerical distribution changes, and categorical distribution. For inference tables...
- 668 Views
- 0 replies
- 0 kudos
