Databricks Community

balajij8

High Quality Data is no longer enough for AI based decisions. Governance, Lineage and Observability helped creation of Trusted Data Products to allow consumers to use the trusted information delivered by the Data Platforms.Generative AI and Agentic A...

balajij8

Data Engineering 1.0 was largely about integrating, moving, consolidating massive data to support enterprise decisions. Enterprises built ETL pipelines, data warehouses, data lakes and orchestration platforms to bring data together reliably. Data Eng...

balajij8 · 06-30-2026

Real time capabilities in healthcare is a critical factor in care outcomes and operational efficiency. From streaming continuous vitals from IoT-enabled care monitors to routing instantaneous telemetry from wearable medical devices, modern healthcare...

balajij8 · 06-25-2026

Modern Enterprise Healthcare Lake bases have fundamentally transformed care data operations by seamlessly unifying high concurrency transactional workloads such as electronic records (EMR) syncing, streaming care vitals and persistent memory for gene...

balajij8 · 06-18-2026

The friction between modern operational edge applications and core analytical data systems is a massive architectural bottleneck in healthcare organizations. Building patient-facing mobile applications, synchronizing remote patient monitoring (RPM) w...

balajij8

@dsay96 You can use interactive compute or try to replace the private CA cert with a public CA cert on the IBM server. If the server presents a certificate from a recognized public CA (DigiCert etc), the serverless runtime's default ca certs truststo...

balajij8

@rachelnguyen MERGE operation is inherently expensive as it searches the entire target table to find matches, then shuffles data multiple times to write updates/inserts. It generates many small files - especially on partitioned tables where each shuf...

balajij8

@DineshOjha The issue here is likely tied to the Spark or Python REPL hanging during its initialization phase on boot. When custom libraries are installed - the packaging Hive clients, custom Hadoop configurations or conflicting JARs -dependency conf...

balajij8

@faruk Handling Oracle NUMBER in the ingestion pipelineThe Oracle JDBC driver maps an NUMBER (no precision, no scale) to DecimalType(38,10) as it must pick a fixed scale for a type system that has no equivalent to Oracle's unconstrained decimal. You ...

balajij8

@keshavmonga22 You can also check by writing logs to /tmp then upload using the SDK's Files API (w.files.upload() or w.files.upload_from()) as it generally bypasses FUSE entirely - uploads happen via REST calls directly to the object storage backend ...

Databricks Community

User Stats

User Activity

From Data Quality to Context Quality

Data Engineering 3.0 - From Trusted Data Products to Context Layers

Real Time Healthcare Analytics with Databricks Zerobus and Lakehouse RT via Reyden

Fortifying Enterprise Healthcare Databricks Lakebase with the Security Triad

Zero Code REST Integration for Modern HealthCare Vitals via Databricks Lakebase Data API

Re: using remote_query with SQL Serverless Warehouse

Re: Getting started with Delta Lake basics

Re: Notebook commands hang after restarting a cluster with custom library installed

Re: Oracle NUMBER → DecimalType(38,10) on ingestion, and silver layer best practices

Re: shutil.copy from /local_disk0 to Unity Catalog Volume hangs for hours — recommended pattern for