cancel
Showing results for 
Search instead for 
Did you mean: 
Discussions
Engage in dynamic conversations covering diverse topics within the Databricks Community. Explore discussions on data engineering, machine learning, and more. Join the conversation and expand your knowledge base with insights from experts and peers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Browse the Community

Community Discussions

Engage in vibrant discussions covering diverse learning topics within the Databricks Community. Expl...

4719 Posts

Activity in Discussions

ChristianRRL
by > Honored Contributor
  • 362 Views
  • 4 replies
  • 1 kudos

Lakeflow Community Connector Issue - ModuleNotFoundError: No module named 'databricks.labs'

Hi there, I have what should be a somewhat straightforward implementation of a Custom Community Connector. I've tried to Dry run a few times but am encountering PYTHON.MODULE_NOT_FOUND_ERRORBelow is some context about my setup:Source Relative path: l...

ChristianRRL_1-1784652953771.png ChristianRRL_0-1784652943308.png
  • 362 Views
  • 4 replies
  • 1 kudos
Latest Reply
iyashk-DB
Databricks Employee
  • 1 kudos

the sys.path approach won’t fix this for Lakeflow Connect pipelines. During a dry run, Databricks imports databricks.labs.community_connector before your ingest notebook runs, using only the pipeline’s Environment dependencies. Fix: Pipeline Settings...

  • 1 kudos
3 More Replies
ChristianRRL
by > Honored Contributor
  • 81 Views
  • 1 replies
  • 0 kudos

Community Connectors confusion with `connector_spec.yaml` usage (or lack of usage)

Hi there, I'm a bit confused about how connections for Community Connectors setup should be. I assumed that the `connector_spec.yaml` file should outline the specifics of what is required for a Community Connector, but in practice it doesn't seem lik...

ChristianRRL_0-1784673360714.png ChristianRRL_1-1784673396671.png ChristianRRL_2-1784673422995.png ChristianRRL_3-1784673445797.png
  • 81 Views
  • 1 replies
  • 0 kudos
Latest Reply
iyashk-DB
Databricks Employee
  • 0 kudos

The connector_spec.yaml isn't what controls the UI for custom connectors. When you use the "Add Custom Community Connector" path in the workspace UI, Databricks shows a generic connection form with all available auth types. That form is the same for ...

  • 0 kudos
zzepo
by > Visitor
  • 116 Views
  • 1 replies
  • 0 kudos

NCC private endpoint cannot be added

hi,Trying to set up serverless compute and can't create a private endpoint because "Private access endpoint service {endpoint name} is not supported because {endpoint name} is not supported for account {account name}". Following this thread but stuck...

  • 116 Views
  • 1 replies
  • 0 kudos
Latest Reply
iyashk-DB
Databricks Employee
  • 0 kudos

That error ("is not supported for account") means your Databricks account hasn't been enrolled in the private preview for NCC private endpoints. The feature is currently in private preview, so even if you follow the documentation steps correctly, the...

  • 0 kudos
Nimanita
by > New Contributor
  • 55 Views
  • 1 replies
  • 0 kudos

UI sends empty managed_identity_id, breaks storage credential creation with system-assigned Access

Environment:Cloud: AzureDatabricks Account Console (accounts.azuredatabricks.net)Access Connector: system-assigned managed identity (no user-assigned identity involved)Region: East USIssue:Creating a Unity Catalog storage credential via the Account C...

  • 55 Views
  • 1 replies
  • 0 kudos
Latest Reply
iyashk-DB
Databricks Employee
  • 0 kudos

Yes, this is a bug in the Account Console UI. You've already diagnosed it correctly: for a system-assigned managed identity, managed_identity_id should be omitted entirely from the request payload. An empty string "" is not the same as omitting the f...

  • 0 kudos
yassin
by > Visitor
  • 44 Views
  • 1 replies
  • 0 kudos

Enable Foundation Model APIs for AWS Marketplace trial workspace

Hello,I created a new Databricks account through the AWS Marketplace $400 trial using my business email. The workspace is in us-east-1, the AWS Marketplace payment method is connected, and the full $400 credit balance is active.Every Databricks-hoste...

  • 44 Views
  • 1 replies
  • 0 kudos
Latest Reply
iyashk-DB
Databricks Employee
  • 0 kudos

That 403 with "rate limit of 0" isn't something you accidentally configured, it's a Databricks-imposed gate that applies to newly created accounts. Even with active trial credit and a valid payment method on file, Foundation Model API access gets hel...

  • 0 kudos
ChristianRRL
by > Honored Contributor
  • 40 Views
  • 1 replies
  • 0 kudos

Issue: Lakeflow Connect Microsoft Teams Community Connector - No module named 'databricks.labs'

Hi there, I'm trying to run a pretty barebones test here with community connectors. I'm trying out the "vanilla" Microsoft Teams connector by:(A) going to the Data Ingestion tab in our workspace,(B) selecting the Microsoft Teams community connector,(...

ChristianRRL_1-1784735725394.png
  • 40 Views
  • 1 replies
  • 0 kudos
Latest Reply
iyashk-DB
Databricks Employee
  • 0 kudos

Both issues trace back to running on non-serverless compute. Community connectors were designed to run on serverless Spark Declarative Pipelines (SDP), and serverless is on by default in the CLI config for a reason. The No module named 'databricks.la...

  • 0 kudos
CURIOUS_DE
by > Valued Contributor
  • 36 Views
  • 1 replies
  • 0 kudos

Materialized View backing pipeline retains old Unity Catalog after catalog rename

Materialized View backing pipeline retains old Unity Catalog after catalog rename (dev_cat → dap_catalog_dev) resulting in NO_SUCH_CATALOG_EXCEPTIONThis issue blocks the creation and refresh of Materialized Views after a Unity Catalog rename.Workspac...

  • 36 Views
  • 1 replies
  • 0 kudos
Latest Reply
iyashk-DB
Databricks Employee
  • 0 kudos

What you're hitting is expected behavior, not something you did wrong. Every materialized view is backed by a managed serverless pipeline, and that pipeline pins its catalog, target schema, and internal storage location at the moment the M...

  • 0 kudos
ideal_knee
by > New Contributor III
  • 15535 Views
  • 8 replies
  • 9 kudos

Reading an Iceberg table with AWS Glue Data Catalog as metastore

I have created an Iceberg table using AWS Glue, however whenever I try to read it using a Databricks cluster, I get `java.lang.InstantiationException`. I have tried every combination of Spark configs for my Databricks compute cluster that I can think...

  • 15535 Views
  • 8 replies
  • 9 kudos
Latest Reply
nolwenn
Visitor
  • 9 kudos

Hello, I get the same issue "Failed to instantiate org.apache.hadoop.mapred.FileInputFormat" while trying to read Iceberg tables through Glue catalog federation. Provider appears as hive in Databricks while it is registered as an Iceberg table in Glu...

  • 9 kudos
7 More Replies
techarticle
by > New Contributor II
  • 768 Views
  • 1 replies
  • 0 kudos

Generative AI Development: What Does It Take to Move from PoC to Production?

Many organizations have successfully built Generative AI proofs of concept. The bigger challenge is deploying enterprise-grade AI systems that are secure, scalable, and deliver measurable business value.Key capabilities that make a difference include...

  • 768 Views
  • 1 replies
  • 0 kudos
Latest Reply
DoTA
New Contributor III
  • 0 kudos

3 biggest pieces:1. Reliabilty: How do you ensure this app will not breaks in production ?2. Evaluation: How do we evaluate at scale and whether the AI solution is yeilding better results than a human does it ?3. Value: As POC/demo(s) is now happenin...

  • 0 kudos
suryaprayaga
by > New Contributor
  • 185 Views
  • 2 replies
  • 3 kudos

Resolved! Snowflake OR Databricks

I am doing a case choice analysis on whether I should have my data in Snowflake to support my Customer workloads or have them migrated to Databricks to do the same.What has been your experience especially in handling large volumes of data especially ...

  • 185 Views
  • 2 replies
  • 3 kudos
Latest Reply
DoTA
New Contributor III
  • 3 kudos

Given the pace in the new AI era, which platform have the most inovations that brings value to customers clearly wins it. And we can see Databricks is releasing new features on weekly basis, they are clearly not behind.

  • 3 kudos
1 More Replies
Manjunath2684
by > Visitor
  • 51 Views
  • 1 replies
  • 0 kudos

comprehensive guide or usecases for data engineering hands-on

Hello members,I have access to my databricks environment and I see it already includes sample data sets (under "samples" database).I find the learning paths are very basic and not any close the real use cases. Are there any advanced hands-on use case...

  • 51 Views
  • 1 replies
  • 0 kudos
Latest Reply
Brahmareddy
Esteemed Contributor II
  • 0 kudos

Hi Manjunath  How are you doing today? I felt the same way starting out, so I know what you mean. The best structured resource is dbdemos.ai, maintained by the Databricks team. It installs full end to end demos into your workspace in one line (pip in...

  • 0 kudos
Bitrip007
by > New Contributor II
  • 184 Views
  • 1 replies
  • 0 kudos

New Agentic AI Ecosystem in Databricks

it seems Databricks is moving from traditional RAG pipelines toward agent-native architectures with Lakebase Search, Genie Agents, MCP integration, and semantic business context.For teams building enterprise GenAI applications today:Are you still bui...

  • 184 Views
  • 1 replies
  • 0 kudos
Latest Reply
DoTA
New Contributor III
  • 0 kudos

Given Databricks is now very mature in the Agentic/AI landscape comparing to 1 year ago, we are starting to leverage most of the capabilities from Databricks such as:1. Vector Search for Vector Database -> Transitioning to Lakebase Search.2. Lakebase...

  • 0 kudos
mk222
by > Visitor
  • 93 Views
  • 2 replies
  • 0 kudos

Unable to connect to Azure database from Databricks free edition

Hello Databricks community,I am trying to connect to Azure SQL DB from Databricks free edition, however ran into below error-[FAILED_JDBC.CONNECTION] Failed JDBC on the operation: Failed to connect to the database. SQLSTATE: HV000I have tried adding ...

  • 93 Views
  • 2 replies
  • 0 kudos
Latest Reply
Brahmareddy
Esteemed Contributor II
  • 0 kudos

Hi  ,How are you doing today? as per my understanding, this isn't an Azure firewall problem, so adding IP ranges won't help. That's also why opening 0.0.0.0 to 255.255.255.255 made no difference.The real reason is on the Databricks side. Free Edition...

  • 0 kudos
1 More Replies
dk_negrana
by > Visitor
  • 104 Views
  • 2 replies
  • 0 kudos

Databricks Cost for additional Workspace in Azure

I am planning to create an additional workpace in Databricks. I just want to know if there is any additional cost in terms of billing. thanks in advance for the reponse.

  • 104 Views
  • 2 replies
  • 0 kudos
Latest Reply
ajaygshah
New Contributor II
  • 0 kudos

There is no charge to create workspaces. Please go ahead and create one. While you create one, pay close attention to the networking tab. By default, it creates a virtual network and the associated costs for it, you will have to bear. You can simply ...

  • 0 kudos
1 More Replies
shubham_007
by > Contributor III
  • 12043 Views
  • 8 replies
  • 3 kudos

Resolved! What are powerfull data quality tools/libraries to build data quality framework in Databricks ?

Dear Community Experts,I need your expert advice and suggestions on development of data quality framework. What are powerfull data quality tools or libraries are good to go for development of data quality framework in Databricks ? Please guide team.R...

  • 12043 Views
  • 8 replies
  • 3 kudos
Latest Reply
ajaygshah
New Contributor II
  • 3 kudos

There are plenty of tools available but one that's closely integrated with Databricks is DQX. It is quite an awesome tool that really covers most of the rules you would need. It also has the ability to develop complex SQL queries wherein joins and ag...

  • 3 kudos
7 More Replies