rss.livelink.threads-in-node

eMudhra CA key leak: Databricks LTS runtimes still trust the affected root.

Kirankumarbs — Thu, 29 Oct 2026 10:44:13 GMT

Background. In October 2026, eight eMudhra intermediate CA keys turned out to be recoverable from their public certificates. Browsers blocked them within days. Non-browser clients that carry their own trust store keep trusting the chain until that store is updated. On Databricks that means the Python and pip inside the runtime, not just the OS.

(from the release notes):

Runtime	certifi	pip	OS
17.3 LTS	2025.1.31	25.0.1	Ubuntu 24.04.3
14.3 LTS	2022.12.7	n/a	Ubuntu 22.04.3
12.2 LTS (end of support)	2021.10.8	n/a	Ubuntu 20.04.5

certifi 2025.1.31 lists four emSign roots, including `emSign ECC Root CA - G3`. The fixed bundle is 2026.10.19 or later, which keeps only `emSign Root CA - G1`. The newest 17.3 maintenance update I could find was dated 29 Sep, before the disclosure.

CPython 3.11.18, 3.12.16 and 3.13.18 reject the compromised intermediates in `ssl`, but DBR uses the OS Python, so you can't choose the patch release. And pip below 26.2.2 always loads its own vendored certifi on top of the system store, so `PIP_CERT` or `REQUESTS_CA_BUNDLE` only *add* certificates. They don't remove the bad ones.

Fix 1: certifi, per job: Either put it in your wheel's dependencies:

# requirements.txt / install_requires certifi>=2026.10.19

or add it as a job library in your bundle:

libraries: - pypi: package: "certifi>=2026.10.19"

Gotcha: a dependency in your wheel only reaches jobs pinned to the new wheel version. Jobs still pinned to an older release keep the runtime's certifi until you bump their version in the bundle.

Fix 2: pip, before libraries install:

pip can't safely upgrade itself through the pip that has the problem. A cluster init script runs before library installation and can bootstrap a hash-verified wheel, so the download is checked whatever pip trusts. Test it on one cluster first, and check your workspace's init-script allowlist for your access mode like

#!/bin/bash set -euo pipefail /databricks/python/bin/python -m pip install \ "pip @ https://files.pythonhosted.org/packages/3f/06/3e4420302363fbb71be3c7f35ce042f54bfca330e3e66b3a2e3c4cca579f/pip-26.2.2-py3-none-any.whl#sha256=fd7f4dfa8190e014013b9f08f3e973f4e4c26ffd179a204895533a7254813a70"

Check what a cluster actually trusts(notebook on a job cluster):

import certifi, pip, subprocess print("certifi", certifi.__version__, "| pip", pip.__version__) print("G3 in certifi bundle:", "emSign ECC Root CA - G3" in open(certifi.where()).read()) print(subprocess.run("ls /etc/ssl/certs | grep -i emsign", shell=True, capture_output=True, text=True).stdout) print(subprocess.run("keytool -list -cacerts -storepass changeit | grep -ci emsign", shell=True, capture_output=True, text=True).stdout)

The OS store and the JVM `cacerts` belong to the runtime image (17.3 runs Zulu 17). Raise a support ticket and ask when an LTS maintenance update covers it. Runtimes past end of support, like 12.2, won't get one, so move off them.

Exploiting this needs a forged certificate *and* a position on the network path (DNS, proxy, routing). The highest-value targets are clients that send credentials or install code: package installs at cluster start, token endpoints, API clients.

My Exam got suspended before launching

ramesh4 — Thu, 29 Oct 2026 08:16:49 GMT

Hi @Cert-Team MY exam got suspended before even launch the exam, please help in taking exam again as this is important to me.

Unity Catalog external location access error after metastore attachment

Khasim_1 — Thu, 29 Oct 2026 07:46:02 GMT

Hi everyone,

I am running into a permission issue when trying to query data from an external S3 storage location managed through Unity Catalog.

Here are the symptoms:

The Setup: Storage credentials and an external location were successfully created, and the metastore is properly linked to the workspace.
The Problem: When running a query or trying to list files in the external location, the job fails with a permission denied error, even though the assigned IAM role appears correct on the cloud side.
What I'm looking for: Has anyone faced a similar authorization gap between AWS IAM and Unity Catalog storage credentials? What troubleshooting steps or logs should I check next?

Any guidance from the community would be greatly appreciated!

Building an AI Search Visibility Tracker With Databricks, Delta Lake, and Python

brainwavesindia — Thu, 29 Oct 2026 05:38:53 GMT

AI-powered search is changing how people discover companies, products, and professional services. Instead of browsing several links, users may receive a generated response containing recommendations, explanations, and references to selected websites. For businesses, appearing in these answers can create another opportunity to reach potential customers.

However, measuring this presence consistently presents a challenge. Checking individual responses manually offers limited insight because answers can differ between platforms, queries, and testing sessions. A dedicated tracking system can turn these scattered observations into organised records, helping teams understand where their websites appear and which subjects require further attention.

Creating a Reliable Tracking Framework

A useful tracker begins with a carefully prepared question set. These queries should represent genuine customer interests, including product comparisons, service-related questions, industry problems, and purchasing decisions. Grouping them by topic makes the resulting analysis more meaningful than treating every query as an unrelated search.

Python can support data collection through approved APIs and permitted sources. Each observation should include the question, platform, timestamp, generated response, brand mentions, and available citations. Recording the collection method alongside these details helps maintain consistency. Repeating selected queries over time also makes it possible to distinguish persistent patterns from occasional appearances.

Using Databricks and Delta Lake for Storage

Databricks provides an environment for processing datasets, running analytical workloads, and coordinating repeatable data operations. Delta Lake adds transactional reliability, schema enforcement, and versioning capabilities that help maintain dependable records as information accumulates.

A practical architecture can organise information into three stages. The initial layer preserves original observations, the processing layer standardises fields and removes duplicates, and the final layer prepares aggregated results for reporting. This separation makes errors easier to investigate and allows processed records to be regenerated when analytical requirements change.

For example, PySpark can standardise domain names, extract citation URLs, and group observations by query category. SQL can then calculate summary statistics without repeatedly processing the entire collection. Keeping these operations within the same analytical environment reduces unnecessary data movement and simplifies maintenance.

Measuring Mentions and Citation Patterns

Once the dataset is ready, analysts can calculate indicators that explain how frequently a business appears in tested AI responses. A mention rate measures the share of sampled answers containing a particular brand, while citation frequency shows how often its website is referenced. Comparing these indicators across subject areas can reveal differences between general awareness and source attribution.

These measurements require careful interpretation. A business mentioned in generated text may not receive a direct citation, and a cited webpage does not necessarily produce referral visits. Therefore, AI response observations should be compared with website analytics and conventional search performance whenever suitable data is available. Clear definitions also prevent teams from treating different measurements as interchangeable.

Automating Analysis and Reporting

Scheduled workflows can refresh collected records at consistent intervals, while validation checks identify missing timestamps, malformed URLs, and unexpected duplicate entries. Databricks notebooks provide space for exploratory analysis, whereas dashboards can present trends through charts, filters, and summary tables. Analysts can investigate which questions generate citations, where competitors appear, and whether observed visibility changes persist across multiple collection periods.

Responsible implementation matters as well. Collection should follow platform rules, and sensitive information should be excluded or protected through suitable access controls. Documenting sampling methods and limitations makes findings easier for other practitioners to reproduce and evaluate.

Converting Findings Into Practical Improvements

A tracking system becomes valuable when its findings inform specific website improvements. Repeatedly unanswered questions may indicate missing explanations, unclear service information, or insufficient evidence supporting important claims. Reviewing those pages can help teams improve factual accuracy, topical coverage, and overall usefulness without assuming that any individual adjustment guarantees an AI citation.

For professionals investigating how to appear in ai search result, a Databricks-based tracker provides a structured way to examine visibility, identify content weaknesses, and evaluate changes through recorded evidence rather than assumptions.

LEFT JOIN WITH FILTER vs LEFT JOIN and WHERE FILTER

RGSLCA — Wed, 28 Oct 2026 23:17:06 GMT

Hi,

I may need to investigate further but this is my observation,

I had a SQL query working fine until today,

Query-A SELECT ta.id,tb.id FROM TableA ta left join TableB tb on ta.id = tb.id and ta.dtx = to_date(<date string>) where ta.some_other_col = true Query-B SELECT ta.id,tb.id FROM TableA ta left join TableB tb on ta.id = tb.id where ta.dtx = to_date(<date string>) and ta.some_other_col = true

Option-A Query started not filtering rows and sending different dates without considering the date filter

I had to modify the SQL to be like Option-B to make the filter working.

I have tested on Oracle and Other platforms this seems to work fine. Can someone tell me what am I doing wrong or is there a reason for this ? it looked like a sudden change but I cant believe that as a reason.

Note: The original query was a CTE with multiple steps like above.

How to enable DeepSeek V4.1 Flash (1017) on a new AWS workspace?

sillytail_dev — Wed, 28 Oct 2026 15:41:15 GMT

Databricks Support referred me to the Community for help with account verification and model access.

I have a newly created Databricks on AWS workspace. The account console shows Premium with AWS Marketplace billing configured. I authenticate using the official Databricks CLI as a workspace administrator.

The exact model system.ai.deepseek-v4-1-flash-1017 is listed in Unity Gateway model services, and my administrator already inherits EXECUTE on system.ai.

A minimal synthetic request to POST /ai-gateway/mlflow/v1/chat/completions still returns HTTP 403 PERMISSION_DENIED:

"Access to this model is disabled for your workspace. Your Databricks account may require verification or additional model-access approval. Contact Databricks Support or your Databricks account team for assistance."

Could a community engineer clarify the official verification and approval process for this exact model on AWS, including any region or trial-account restrictions? Can existing trial credits be used once access is approved?

Help Center SSO succeeds, but the resulting page says this account has no Help Center access. Which official private channel or team can verify the account and enable model access? I can provide account and workspace identifiers privately to Databricks staff.

Thank you.

Genie One PDF Download Cut Off

nhiltonADPMKP — Wed, 28 Oct 2026 15:22:30 GMT

Hello,

Today when we downloaded the pdf file from Genie One and the results are off center to the left. We've tried different browsers but the results are cutoff and shifted to the left. Is anyone else experiencing this or know of a solution to correct this?

Thank you!

Is a NAT Gateway required for a VNet-injected authentication workspace ?

Dia75 — Wed, 28 Oct 2026 12:01:28 GMT

Hello Community,

We are deploying Azure Databricks in France Central with VNet injection, Secure Cluster Connectivity (SCC) and Private Link.

We have a dedicated workspace for the browser_authentication private endpoint, as recommended in the Databricks documentation. ( This workspace will never run any compute)

Does this workspace need explicit outbound connectivity, such as a NAT Gateway?

The VNet injection documentation says that new workspaces require explicit outbound connectivity after March 31, 2026. https://learn.microsoft.com/en-us/azure/databricks/security/network/classic/vnet-inject.
The secure-cluster-connectivity documentation mentions that outbound connectivity is required to ensure proper cluster functionality. https://learn.microsoft.com/en-us/azure/databricks/security/network/classic/secure-cluster-connectivity

But it's not clear if this also applies to authentication-only workspaces that never run compute.

Can we deploy this workspace without any explicit outbound connectivity, and would this configuration still be officially supported?

Thanks for your help

Best practices and optimization tips for Delta Lake tables

Khasim_1 — Wed, 28 Oct 2026 11:16:39 GMT

Hi everyone,

I’m currently working with Delta Lake and wanted to start a discussion around performance tuning, maintenance, and best practices.

I’m interested in hearing how others in the community approach a few key areas:

Maintenance Operations: How often do you run OPTIMIZE (compaction) and VACUUM on your large production Delta tables?

Partitioning vs. Z-Ordering: What guidelines do you follow when choosing between partitioning a table and using Z-Ordering?

Would love to hear your experiences, lessons learned, or tips for keeping Delta tables running efficiently at scale!

Google Shopping Feed Optimization for Shopify Stores

UnfoldCRO2 — Wed, 28 Oct 2026 10:31:11 GMT

our Google Shopping feed, not your bids, is the real growth lever most Shopify stores ignore. Here’s how to optimize product data for Merchant Center, Shopping and Performance Max.

Most Shopify stores spend their Google Ads effort on bids, budgets and creative, while the product feed goes to Google exactly as the store exported it. That’s backwards. Shopping and Performance Max campaigns have no keyword list. Google reads your feed to decide when a product shows, who sees it, and how the listing is described. If the feed is weak, bid strategy can only do so much. If the feed is clean, every rupee of spend has better material to work with.

How your product data reaches Merchant Center

On Shopify, product data reaches Google Merchant Center in one of two ways:

The native Google & YouTube app. It syncs your catalog automatically, mapping Shopify fields to Google attributes on a schedule.
A dedicated feed app or a direct API integration. Larger or more complex catalogs often use one of these for finer control over field mapping, exclusions and extra data.

Either way, Merchant Center sits in the middle. It validates every item, flags problems, and passes approved products to your campaigns.

The key thing to understand is that Shopify fields were built for your storefront, not for Google. Your titles, descriptions, tags and variants were written for people browsing your site. Feed optimization means reshaping that data into what Google’s matching and ad surfaces actually reward.

Write titles that match how people search

The title is the highest-impact field in the feed, because Google matches search queries against it heavily. A storefront title like “Aria Midi Dress” might look clean on your site, but it misses most of the searches that should trigger it.

Build titles from the terms shoppers actually type, most important first. A pattern that works for most catalogs is Brand + Product Type + Key Attribute + Color + Size or Material. For apparel, that could be “Zoya Cotton Midi Dress, Floral Print, Navy, Size M”. Google allows up to 150 characters, but listings show far fewer, so put the words that decide the click within roughly the first 70.

Keep descriptions factual

Descriptions carry less weight than titles, but they still feed into matching. Make the first sentence cover what the product is, what it’s for, and its standout attributes, then add detail.

Leave out store policies, shipping terms and promotional language. They add nothing to matching and can trigger policy flags.

Images that pass review

Google expects a clean primary image, usually on a white or plain background, with no promotional overlays, watermarks or added text. Overlays such as “SALE” badges are one of the most common causes of disapproval.

Use your highest-resolution product shot as the main image, and supply additional images so Google has options to rotate. Many of the image SEO basics from organic search carry straight over to the feed: file quality, sizing and naming.

Identifiers: GTIN, MPN and identifier_exists

Unique product identifiers tell Google exactly what you sell. They’re required for most branded and manufactured goods.

gtin: supply the barcode number wherever you have one.
mpn: for products with a manufacturer part number, supply it together with brand.
identifier_exists: for custom or handmade products with no barcode, set this to no. Don’t leave the fields blank, and never invent a value.

Correct identifiers improve matching. Some listing features are only available to products that have them.

Two category fields, two different jobs

google_product_category maps each item to Google’s official taxonomy. Set it as deep as the taxonomy allows. Stopping at a broad top-level node gives Google less to work with.

product_type is your own category string. Mirror your real site structure here, for example “Home > Kitchen > Cookware > Non-stick Pans”. It becomes useful when you split campaigns, listing groups and reports by category.

Keep price and availability in sync

Google regularly checks the price and availability in your feed against what it crawls on the landing page. A mismatch leads to disapproval. Common causes are:

A sale price that hasn’t reached the feed
The wrong currency
An out-of-stock variant still marked as in stock

Check that the sync pushes sale_price correctly and that availability reflects real inventory. For stores selling in India, confirm that the currency and the tax-inclusive price in the feed match what the live product page shows.

Variants and custom labels

Submit each variant as its own item, and give variants of the same product a shared item_group_id. Google can then group colors and sizes together while still advertising the exact option a shopper searched for. Each variant needs its own identifier, image and availability.

Custom labels are never shown to shoppers. They exist to help you structure campaigns. You get five fields, custom_label_0 to custom_label_4. Use them to tag products by:

Margin
Bestseller status
Season
Price band
Clearance

Then build campaigns and budgets around those tags. This is how you stop treating every SKU the same and move spend toward the products that actually make money.

Disapprovals and supplemental feeds

Check the Diagnostics section in Merchant Center regularly. A disapproved item can’t serve at all. The usual causes are:

Images: overlays, low resolution, or a missing primary image
Price and availability: mismatches between the feed and the landing page
Identifiers: missing or invalid values
Policy: violations in titles or descriptions
Landing pages: pages that fail review or don’t load correctly

Sometimes you can’t, or don’t want to, change the data at source in Shopify. In that case, use a supplemental feed to override or add specific fields, such as better titles, custom labels or corrected categories.

A supplemental feed matches rows to your primary feed by product ID. With the Google & YouTube app, those IDs follow the app’s own format, which combines the country code, the product ID and the variant ID. Export the IDs from Merchant Center before building the sheet, so every row matches. This lets you layer optimization on top of the automated sync without fighting the app’s field mapping.

Why the feed drives campaign performance

Shopping and Performance Max both pull their inventory, relevance signals and product copy straight from the feed. Google’s automation can only work with the data you give it:

Clear titles
Correct identifiers
Accurate categories
Useful custom labels

With that data in place, the automation has better inputs, and better inputs usually mean more relevant impressions at a lower cost. Strong organic collection pages help as well, since paid and organic listings for the same products reinforce each other.

How UnfoldCRO helps

We audit your Shopify-to-Merchant-Center sync, rewrite titles and categories at scale, fix disapprovals, add identifiers and custom labels, and build supplemental feeds for Indian D2C and ecommerce brands running Shopping and Performance Max. Feed work isn’t glamorous, but it’s where most of the easy wins are.

Needs verification

API name. Your draft said the Google & YouTube app syncs “over the Content API”. Google has been moving merchants from the Content API for Shopping to the newer Merchant API, so I left the API name out rather than risk naming a deprecated one. Confirm which applies before you add it back.
Missing links. Your draft mentioned the image SEO guide and the Shopify collection page SEO guide but had no URLs, so I kept them as plain mentions. Send me the URLs if you want them linked.
New detail. The note on supplemental feed IDs (country code, product ID and variant ID) is standard behaviour for the Google & YouTube app. It wasn’t in your original, so remove it if you’d rather not add new technical detail.

Not received my Databricks certified Data Engineer Professional certification

soumya_kumari — Wed, 28 Oct 2026 07:31:04 GMT

Hello Support Team,

I successfully completed and passed my Databricks certification exam on 15th October 2026.

However, as of 28th October 2026, I have not yet received my certificate, though it is visible in my Webassessor/Credentials portal.

It's already been more than 48hr.

Exam Status: Pass

Exam Date:15th October 2026

Could you please check the status of my certificate and advise if any action is required from my side?

I can provide a screenshot of the passing result and any additional details if needed.

Thank you for your assistance.

Request id: #01057320

@cert-ops @Cert-Team @Cert-Bricks @Cert-TeamOPS

system.ai model names return 404 on /serving-endpoints with Unity Gateway enforcement off

rajavemuri — Wed, 28 Oct 2026 06:25:51 GMT

Environment: one Azure Databricks workspace, pay-per-token foundation models, Enforce Unity Gateway off, requests sent with curl and a CLI-issued token.

Symptom. A request naming a Unity Catalog model service, system.ai.claude-haiku-4-5, sent to https://<workspace-host>/serving-endpoints/chat/completions returns:

HTTP 404
{"error_code":"ENDPOINT_NOT_FOUND","message":"The given endpoint does not exist, please retry after checking the specified model and version deployment exists."}

The same request to https://<workspace-host>/ai-gateway/mlflow/v1/chat/completions returns 200. The legacy endpoint name databricks-claude-haiku-4-5 on /serving-endpoints/chat/completions also returns 200. Embeddings split the same way. system.ai.gte-large-en is a 404 on /serving-endpoints/embeddings and a 200 with 1024 dimensions on /ai-gateway/mlflow/v1/embeddings.

Why it matters. The migration guide (https://learn.microsoft.com/azure/databricks/ai-gateway/migrate-to-unity-gateway) says to change the base URL from /serving-endpoints to /ai-gateway/mlflow/v1 and the model from the endpoint name to the fully qualified model service name, and that legacy configurations stay unchanged until Enforce Unity Gateway is turned on. Our legacy name answering is the sign enforcement was off. The 404 for the catalog name came anyway, so a client that lists system.ai names and sends them down the old path fails before any enforcement decision.

Reproduce it. Replace $HOST and $TOKEN:

BODY='{"model":"MODEL","messages":[{"role":"user","content":"Reply with OK"}],"max_tokens":8}'
for MODEL in system.ai.claude-haiku-4-5 databricks-claude-haiku-4-5; do
  for PATH_ in serving-endpoints ai-gateway/mlflow/v1; do
    printf '%s on /%s: ' "$MODEL" "$PATH_"
    curl -s -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
      -d "${BODY/MODEL/$MODEL}" "$HOST/$PATH_/chat/completions"
  done
done

Expected: the system.ai name is 404 on the first path and 200 on the second; the databricks- name is 200 on the first. We did not test the databricks- name on the gateway path or an enforcement-on workspace.

What we changed. Our product stores one URL per Databricks connection, usually the bare workspace host. Switching every bare host to the gateway path was proposed and rejected, because our workspace's custom-named serving endpoint is addressed under /serving-endpoints, the migration guide keeps existing provisioned throughput endpoints in place with a model service referencing them, and model discovery reads GET /api/2.0/serving-endpoints, which has no gateway counterpart. Instead the model name picks the path. For a stored URL with no path or exactly /serving-endpoints, a model id matching ^[^./]+\.[^./]+\.[^./]+$ goes to <origin>/ai-gateway/mlflow/v1 and any other id goes to <origin>/serving-endpoints; a stored URL with any other path is used as typed. Discovery keeps calling the serving-endpoints API with no model name.

Limits. One workspace, one run, enforcement off. A service principal with query rights on a serving endpoint and nothing on the model service may get PERMISSION_DENIED on the gateway path for a three-part name; those calls were 404 before. The embeddings docs page shows the Python SDK passing system.ai.gte-large-en to serving_endpoints.query, which we did not run; our REST call with that name on /serving-endpoints is the one above.

Longer write-up with the routing function: https://www.techfabric.com/blog/unity-catalog-model-names-return-404-on-serving-endpoints

Question regarding Databricks Apps deployment

Khasim_1 — Wed, 28 Oct 2026 06:04:57 GMT

Hello Databricks Community,

I am currently working on setting up a Databricks App and ran into a few questions.

Here are the details of what I am working with:

Target Framework: [e.g., Streamlit / Dash]
Goal: [Briefly describe what you are trying to achieve]
Current Challenge: [Briefly describe the blocker, e.g., authentication, Unity Catalog connection, etc.]

Has anyone tackled a similar setup?

Any pointers, code snippets, or best practices would be greatly appreciated!

Best practices for processing HVR CDC data in Databricks when handling out-of-order updates, delete

krishgarikipati — Wed, 28 Oct 2026 05:24:30 GMT

Hi Databricks Community,

I am working on a data lakehouse architecture on AWS, where HVR captures changes from OLTP databases such as Oracle and PostgreSQL, AWS S3 is used for storage, and Databricks handles data processing across Bronze, Silver, and Gold layers.

We need to maintain accurate downstream Delta tables while processing inserts, updates, and deletes from source systems.

I would like to understand the recommended architectural approach for the following scenarios:

Out-of-order events: If multiple updates for the same primary key arrive out of order, how can we ensure that an older event does not overwrite the latest record?
Duplicate events and retries: How should we handle duplicate CDC records or replayed files without creating inconsistent target data?
Delete propagation: What is the recommended approach to propagate source deletes from Bronze to Silver and Gold while retaining audit history where required?
Incremental processing: Would you recommend implementing custom Delta MERGE logic or using Lakeflow Declarative Pipelines AUTO CDC APIs for this type of architecture?
Performance: What are the best practices for handling high-volume CDC data, small files, and frequent MERGE operations on Delta tables stored in S3?

Our current architecture uses HVR for source ingestion and Databricks for downstream processing. I am particularly interested in a production-ready solution that balances data correctness, performance, maintainability, and cost.

If anyone has implemented a similar architecture, I would appreciate suggestions, design patterns, or code examples.

Thank you!

Possible Incorrect Answer Key: Spark ML Estimators — OneHotEncoder, Imputer, and StandardScaler

amittian — Tue, 27 Oct 2026 22:56:11 GMT

Hello Databricks Community,

While preparing for the Databricks Machine Learning Associate certification, I came across a question in a practice assessment that I believe may have an incorrect or incomplete answer key.

Question:

"In a Spark ML Pipeline, which two of the following are Estimators — meaning they learn parameters from the training data during fit()? (Select two)"

Options:

OneHotEncoder
VectorAssembler
Imputer
StandardScaler

My selected answers:

OneHotEncoder
StandardScaler

Assessment result: Partially correct — 2.5 out of 5 marks.

Answers provided by the assessment:

Imputer
StandardScaler

My understanding:

According to the Apache Spark ML API, three of the listed components are Estimators:

OneHotEncoder: An Estimator that learns category sizes during fit() and produces a OneHotEncoderModel.
Imputer: An Estimator that learns imputation statistics and produces an ImputerModel.
StandardScaler: An Estimator that learns feature statistics and produces a StandardScalerModel.

VectorAssembler is a Transformer because it combines feature columns without learning parameters through fit().

Therefore, technically, the question appears to have three correct answers rather than two.

Could someone from the Databricks training or certification team clarify whether the question or its answer key needs correction?

I would appreciate clarification so that learners can align their understanding with the actual Spark ML API.

Thank you!

English Is Becoming a Powerful Programming Language in Databricks!

Brahmareddy — Tue, 27 Oct 2026 19:51:40 GMT

I have been working with Python, PySpark, and SQL for years. But recently, while experimenting with Databricks Genie Code, I had a moment where I thought, "Are we entering a world where simple English becomes one of the most powerful programming languages?"

I wanted to test how far Genie Code could go, not just by asking it to write SQL or explain some code, but by building an entire data engineering project from scratch.

And I was genuinely amazed by the experience.

I started with simple English instructions. I explained what I wanted to build, and Genie Code helped me create the project step by step.

First, I asked it to create a catalog and the required schemas.

Then I asked it to create tables and generate sample raw data.

Next, I wanted to follow the Medallion Architecture, so I asked it to build the Bronze, Silver, and Gold layers.

It helped me prepare the transformations and organize the data through those layers.

Then came the interesting part.

I asked it to schedule the pipeline as a Databricks job. After that, I connected the curated data to Genie so I could explore it using natural language.

From creating a catalog to asking business questions through Genie, I was able to complete the project in minutes using simple English instructions.

No need to manually write every SQL statement, PySpark transformation, or job configuration.

Of course, I still needed to understand what was being created, review the logic, and make sure the results were correct.

And that is an important distinction.

I don't think Python, PySpark, or SQL are going away. They remain essential, especially when we need to troubleshoot, optimize performance, handle complex transformations, or build reliable production pipelines.

But I do think the way we interact with these technologies is changing.

Instead of spending most of our time telling the platform exactly how to perform every step, we can increasingly explain what we want to achieve and let AI help with the implementation.

For someone who has spent years building data pipelines, this is a meaningful change.

What impressed me most was not just the speed. It was the ability to move across different parts of the Databricks platform through one conversational experience.

From data setup to transformations, orchestration, and analytics.

All connected.

I believe this could make data engineering more accessible while helping experienced engineers spend more time on architecture, data quality, business logic, and solving actual problems.

My biggest takeaway: English may not replace programming languages, but it is becoming a powerful new way to work with them.

Really impressed with the direction Databricks is taking with Genie Code. It is one of those capabilities that becomes much more interesting when you actually try building something end to end.

Curious to hear from the community.

Have you tried building an entire data engineering workflow using Genie Code? How was your experience?

Why I started evaluating the pipeline instead of choosing the best model at each stage

FrankAzzollini — Tue, 27 Oct 2026 19:42:25 GMT

I recently worked on a multi-label classification pipeline on Databricks, using embeddings, scikit-learn classifiers, Delta Lake, and MLflow to track experiments.

One result caught my attention: the embedding model that ranked fourth in my initial screening ended up in the best-performing combination when evaluated with the actual classifier.

The experiments also surfaced a few less obvious issues:

The best decision threshold varied significantly across model combinations.
Rebuilding the label vocabulary at inference time could silently associate predictions with the wrong labels.
Differences between training and inference inputs could go unnoticed if you only evaluate individual stages.

I documented the experiments, results, code, and lessons learned here:

My Medium Article

I'm curious how others approach this in Databricks: when you have multiple interacting components, do you run a joint search over the pipeline, or narrow down candidates using stage-level metrics first?

𝗘𝗱𝗶𝘁𝗶𝗻𝗴 𝗙𝗶𝗹𝗲𝘀 𝗶𝗻 𝗨𝗻𝗶𝘁𝘆 𝗖𝗮𝘁𝗮𝗹𝗼𝗴 𝗩𝗼𝗹𝘂𝗺𝗲𝘀 𝗜𝘀 𝗡𝗼𝘄 𝗚𝗔

sridharplv — Tue, 27 Oct 2026 18:45:05 GMT

Databricks now allows you to edit supported files directly in the workspace file editor.

🔹𝗪𝗵𝗮𝘁’𝘀 𝗡𝗲𝘄?
You can open and edit files stored in Unity Catalog volumes directly from:
✅Catalog Explorer
✅The schema browser in notebooks
✅The schema browser in the SQL editor
Changes are automatically saved back to the volume.

🔹𝗛𝗼𝘄 𝗗𝗼𝗲𝘀 𝗜𝘁 𝗪𝗼𝗿𝗸?
1️⃣Navigate to Catalog Explorer and browse to your Unity Catalog volume.
2️⃣Open the file’s menu and select 𝗢𝗽𝗲𝗻 𝗶𝗻 𝗘𝗱𝗶𝘁𝗼𝗿. You can also open the file preview and select the editor icon.
3️⃣Make your changes. The editor automatically saves them to the volume.
You can also open files through the schema browser in a notebook or SQL editor.

🔹𝗪𝗵𝗶𝗰𝗵 𝗙𝗶𝗹𝗲 𝗧𝘆𝗽𝗲𝘀 𝗔𝗿𝗲 𝗦𝘂𝗽𝗽𝗼𝗿𝘁𝗲𝗱?
The editor supports several commonly used formats:
• 𝗖𝗼𝗱𝗲: Python, Scala, SQL, R, Shell
• 𝗧𝗲𝘅𝘁: TXT, LOG, Markdown
• 𝗗𝗮𝘁𝗮: CSV, JSON, NDJSON, JSONL, TSV, TAB
• 𝗖𝗼𝗻𝗳𝗶𝗴𝘂𝗿𝗮𝘁𝗶𝗼𝗻: YAML, TOML, INI, CONF, PROPERTIES, ENV

🔹𝗥𝗲𝗮𝗹-𝗪𝗼𝗿𝗹𝗱 𝗨𝘀𝗲 𝗖𝗮𝘀𝗲𝘀:

💡𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴: Update SQL scripts and pipeline configuration files without leaving the Databricks workspace.
💡𝗠𝗮𝗰𝗵𝗶𝗻𝗲 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 & 𝗔𝗜: Maintain supported configuration files, prompts stored in text files, and lightweight project artifacts.
💡𝗗𝗲𝘃𝗢𝗽𝘀: Modify environment configurations and shell scripts stored in governed volumes.
💡𝗧𝗲𝗮𝗺 𝗖𝗼𝗹𝗹𝗮𝗯𝗼𝗿𝗮𝘁𝗶𝗼𝗻: Simplify small file updates while keeping files within Unity Catalog-managed access controls.

🔹𝗜𝗺𝗽𝗼𝗿𝘁𝗮𝗻𝘁 𝗖𝗼𝗻𝘀𝗶𝗱𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝘀
⚠️You need READ VOLUME, USE SCHEMA, and USE CATALOG to open files, plus WRITE VOLUME to save changes.
⚠️Files larger than 10 MB cannot be opened in the editor.
⚠️Images, PDFs, Parquet files, and other unsupported extensions cannot be edited in this editor.
⚠️If a file changes externally while it is open, Databricks detects a conflict and lets you fetch the latest version or overwrite it with your version.
⚠️The file editor is not available in AWS GovCloud regions.

🔹𝗪𝗵𝘆 𝗗𝗼𝗲𝘀 𝗧𝗵𝗶𝘀 𝗠𝗮𝘁𝘁𝗲𝗿?
Unity Catalog volumes are more than storage locations for non-tabular data. They provide a governed way to manage files used by data engineering, analytics, machine learning, and AI workloads.
With in-place editing, Databricks reduces the friction involved in maintaining these files and helps developers stay within their existing workspace workflow.

📚𝗢𝗳𝗳𝗶𝗰𝗶𝗮𝗹 𝗱𝗼𝗰𝘂𝗺𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻: https://docs.databricks.com/aws/en/volumes/volume-files#edit-files

#storage

How to retire Genie Agents and dashboards nobody uses (so you can say yes to more of them)

Valeria_Koz_DBX — Tue, 27 Oct 2026 17:16:00 GMT

"We want more Genie Agents and dashboards. We just don't want to end up with 400 of them."

The business wants more self-service. The platform team is worried about two things: sprawl (nobody can find the right one, and three versions of "revenue" appear) and the cost of running it all (things refreshing that nobody looks at). But slowing creation down with approvals and tickets kills the self-service you wanted.

What worked for us, in a UK enterprise Finance function: make creating easy, and make retiring routine.

One myth first: an unused dashboard on its own costs very little. The cost hides in three places: schedules and subscriptions that still run, warehouses kept busy by them, and the time people waste working out which version to trust. Retiring is as much about trust as it is about money.

Step-by-step

Step 1: Agree the rules first. Write a one-page lifecycle policy: what counts as unused (for example, no views or questions in 60 days), who gets warned, how long they have to reply, and what happens next. Get the business sponsor to sign it off, so retiring isn't seen as IT taking things away.
Step 2: Give every asset an owner. Every dashboard and Genie Agent needs a named person or team. Use a naming convention, tags or a simple register table. No owner means first in line for review.
Step 3: Measure usage from the audit log.

The system.access.audit system table records when a dashboard is opened and when someone asks a Genie Agent a question. This query gives the last-used date and number of users per asset:

-- Last activity per dashboard and Genie Agent (last 90 days)

Anything missing from the result hasn't been used in 90 days.

4. Step 4: Don't decide on usage alone. A quarterly board pack is opened four times a year and is still critical. Combine usage with owner, whether it's certified or in production, and whether it has schedules. Then sort into three buckets: Keep · Review · Retire.

5. Step 5: Warn, then archive. Tell the owner the date. Then turn off schedules and unpublish the dashboard (the draft stays) or move it to Trash, where it can be restored for 30 days. Agents can be restored for 30 days. databricks lakeview unpublish <dashboard_id>

6. Step 6: Make it monthly. Run the review monthly, ideally with your champions or sponsors. Share the numbers: "We retired 18 assets this month, switched off 6 schedules, and nobody asked for one back." That last line is what builds trust in the process.

What good looks like

Every dashboard and Genie Agent has an owner and a last-used date.
Unused assets are archived in a predictable, announced way, so nobody is surprised.
Fewer duplicates, so people trust what's left.
The platform team says yes to new Genie Agents and dashboards faster, because the clean-up is built in.

Common errors

Retiring on usage alone: check purpose and owner too.
No warning: the fastest way to lose the business's trust.
Forgetting schedules: the "unused" dashboard still refreshes every morning.
Deleting instead of archiving. Unpublish or use Trash, so there's a way back.
Treating it as a one-off clean-up. Sprawl comes back within six months. Make it monthly.

The takeaway: you don't control sprawl by making things hard to create. You control it by making them easy, and safe, to retire.

Unable to Access Databricks Workspace After Changing Display Name

Karagaonkarm — Tue, 27 Oct 2026 15:58:31 GMT

Dear Databricks Support Team,

I am writing to request your assistance with an issue affecting my personal Databricks account.

I have been using Databricks for the past three months with my personal account, karagaonkarm@gmail.com, to learn Databricks and work on a data engineering project. I have successfully completed my project in this workspace.

Today, I attempted to change my display name from my existing account name to Mahesh Karagaonkar using Genie AI and the Databricks SDK. After this change, I became unable to access my workspace.

When I try to open it, I receive the following error message:

"Unable to view page – You do not have permission to access this page in workspace 7474660674554210."

The error message also states that I need the appropriate workspace access, Databricks SQL access, or Consumer access entitlement.

I am using this workspace for personal learning and project development, and I am concerned because I have already completed my project in it.

Could you please investigate this issue and help me restore access to my workspace? I would also appreciate your guidance on whether changing my display name using the Databricks SDK caused this permission issue and how I can safely resolve it without losing my project or data.

Account details:

- Registered email: karagaonkarm@gmail.com

- Requested display name: Mahesh Karagaonkar

- Workspace ID: 7474660674554210

- Account type: Personal account

- Issue: Unable to access the workspace after changing the display name using Genie AI and the Databricks SDK

I have attached a screenshot of the error message for your reference.

Thank you for your time and support. I look forward to your assistance in restoring access to my workspace and project.

Best regards,

Mahesh Karagaonkar