Databricks Community

ManojkMohan

AskCan we get a UC catalog (like prod or genie) in free edition of data bricks ?Problem i am solving:Structuring Data in Databricks before sending customer, account data to salesforceissue:cannot see workspace-local tables (workspace.default.structur...

ManojkMohan

Use Case:Converting unstructured data from PDF to structured format before sending to SalesforceAsk:Best practices to structure my table better before sending to a system like salesforceOutput in structured format looks like: My code:Extract Tables f...

ManojkMohan

Use Case:Kafka real time steaming network telemetry logsIn a real use case approx. 40 TB of data can be real time streamed in a dayArchitecture:issue encountered: when i try to simulate kakfa like streaming in databricks itself , as this is a free e...

ManojkMohan

Problem i am solving:Upload PDF → available in /Volumes/<catalog>/<schema>/<volume>/.Extract text with pdfplumber (or OCR if scanned).Store in Delta table for governance.Parse intelligently using:ai_query() with Databricks LLMs for flexible JSON outp...

ManojkMohan

Problem i am trying to solve:Bronze is the landing zone for immutable, raw data.At this stage, i am trying to sse a columnar format (Parquet or ORC) → good compression, efficient scans. and then apply lightweight compression (e.g., Snappy) → balances...

ManojkMohan

Solution:can you try belowChange logic in creation mode:Only use .mode("append") or .mode("overwrite") when the table exists.Use .mode("ignore") or .mode("errorIfExists") (default) appropriately.Avoid .mode("overwrite" in streaming foreachBatch to p...

ManojkMohan

Root Cause:The table or view s_test already exists in the catalog.The code tries a CREATE TABLE without IF NOT EXISTS or without dropping the existing table first.The underlying Spark or SQL engine enforces uniqueness of table names and raises an err...

ManojkMohan

can you share the below logs here if possibleLog SourceHow to Enable/UseUnity Catalog audit logsEnable in workspace settings, configure external log destinationSpark/Delta engine logsSet log level DEBUG, configure cluster log collectionQuery logs as ...

ManojkMohan

For production teams already relying heavily on streaming tables within DLT, introducing a split pipeline with downstream all-purpose clusters can indeed increase orchestration complexity and operational overhead. https://www.databricks.com/blog/2025...

ManojkMohan

Root Cause:Required Kafka SCRAM login module JAR is missing from the managed DLT cluster environment. This typically happens after Databricks platform updates or cluster environment changes, which may remove previously available libraries from the ru...

Databricks Community

User Stats

User Activity

Databricks ro Salesforce | Unity Catalog Query

Parsing from PDF to a Structured Table | Looking for best practies

Simulating Real Time Streaming in Databricks free edition

Extracting PDFs and using AI queries | best practices

Ingesting 100 TB raw CSV data into the Bronze layer in Parquet + Snappy

Re: append using foreach batch autoloader

Re: append using foreach batch autoloader

Re: Create external tables with properties set in delta log and no collation

Re: DLT Pipeline Failed to create new KafkaAdminClient SQLSTATE: XXKST:

Re: DLT Pipeline Failed to create new KafkaAdminClient SQLSTATE: XXKST: