The need for anomaly detection across industries This article is part of a series on time series analysis in partnership with the University of Koblenz. See our other articles on data quality and fore...
Introduction Announced at the Data + AI Summit in June 2023, Lakehouse Federation in Databricks is a groundbreaking new capability that allows you to query data across external data sources - includin...
Authors: Liping Huang (@Liphuan) and Marius Panga (@mariuspc)In recent years, a tidal wave of real-time data has inundated us from individuals, machines, and organizations across various channels—oper...
Introduction Databricks provides a powerful platform for building and running big data analytics and AI workloads in the cloud. However, as with any complex system, issues can arise. Effective monito...
Introduction What is RAG and its use cases Many conversation and assistance AI technologies now use the advanced natural language prowess of large language models (LLMs). However, these models only h...
Introduction Modern data pipelines can be complex, especially when dealing with massive volumes of data from diverse sources. Managing the processing of this data is not too dissimilar to the respons...
Introduction Google Pub/Sub is a fully managed messaging service with pub/sub semantics that is widely used by customers on Google Cloud Platform (GCP), often as an alternative to Kafka. Structured S...
Authors: Andrey Mirskiy (@AndreyMirskiy) and Marco Scagliola (@MarcoScagliola) Introduction With this blog post we start our series on Databricks SQL Serverless, using tangible examples to explain w...
Transitioning to Unity Catalog in the Databricks ecosystem is a critical move for better data governance and operational efficiency. Unity Catalog streamlines data management, ensuring a safe and orga...
What is Overwatch? Overwatch is an observability tool which helps you to monitor spending on your clouds and track usage in various dimensions. It works by collecting job and audit log data, then join...
