Databricks Platform Discussions
Dive into comprehensive discussions covering various aspects of the Databricks platform. Join the co...
Dive into comprehensive discussions covering various aspects of the Databricks platform. Join the co...
Engage in vibrant discussions covering diverse learning topics within the Databricks Community. Expl...
Hi there, I have what should be a somewhat straightforward implementation of a Custom Community Connector. I've tried to Dry run a few times but am encountering PYTHON.MODULE_NOT_FOUND_ERRORBelow is some context about my setup:Source Relative path: l...
the sys.path approach won’t fix this for Lakeflow Connect pipelines. During a dry run, Databricks imports databricks.labs.community_connector before your ingest notebook runs, using only the pipeline’s Environment dependencies. Fix: Pipeline Settings...
Hi there, I'm a bit confused about how connections for Community Connectors setup should be. I assumed that the `connector_spec.yaml` file should outline the specifics of what is required for a Community Connector, but in practice it doesn't seem lik...
The connector_spec.yaml isn't what controls the UI for custom connectors. When you use the "Add Custom Community Connector" path in the workspace UI, Databricks shows a generic connection form with all available auth types. That form is the same for ...
hi,Trying to set up serverless compute and can't create a private endpoint because "Private access endpoint service {endpoint name} is not supported because {endpoint name} is not supported for account {account name}". Following this thread but stuck...
That error ("is not supported for account") means your Databricks account hasn't been enrolled in the private preview for NCC private endpoints. The feature is currently in private preview, so even if you follow the documentation steps correctly, the...
Environment:Cloud: AzureDatabricks Account Console (accounts.azuredatabricks.net)Access Connector: system-assigned managed identity (no user-assigned identity involved)Region: East USIssue:Creating a Unity Catalog storage credential via the Account C...
Yes, this is a bug in the Account Console UI. You've already diagnosed it correctly: for a system-assigned managed identity, managed_identity_id should be omitted entirely from the request payload. An empty string "" is not the same as omitting the f...
Hello,I created a new Databricks account through the AWS Marketplace $400 trial using my business email. The workspace is in us-east-1, the AWS Marketplace payment method is connected, and the full $400 credit balance is active.Every Databricks-hoste...
That 403 with "rate limit of 0" isn't something you accidentally configured, it's a Databricks-imposed gate that applies to newly created accounts. Even with active trial credit and a valid payment method on file, Foundation Model API access gets hel...
Hi there, I'm trying to run a pretty barebones test here with community connectors. I'm trying out the "vanilla" Microsoft Teams connector by:(A) going to the Data Ingestion tab in our workspace,(B) selecting the Microsoft Teams community connector,(...
Both issues trace back to running on non-serverless compute. Community connectors were designed to run on serverless Spark Declarative Pipelines (SDP), and serverless is on by default in the CLI config for a reason. The No module named 'databricks.la...
Materialized View backing pipeline retains old Unity Catalog after catalog rename (dev_cat → dap_catalog_dev) resulting in NO_SUCH_CATALOG_EXCEPTIONThis issue blocks the creation and refresh of Materialized Views after a Unity Catalog rename.Workspac...
What you're hitting is expected behavior, not something you did wrong. Every materialized view is backed by a managed serverless pipeline, and that pipeline pins its catalog, target schema, and internal storage location at the moment the M...
I have created an Iceberg table using AWS Glue, however whenever I try to read it using a Databricks cluster, I get `java.lang.InstantiationException`. I have tried every combination of Spark configs for my Databricks compute cluster that I can think...
Hello, I get the same issue "Failed to instantiate org.apache.hadoop.mapred.FileInputFormat" while trying to read Iceberg tables through Glue catalog federation. Provider appears as hive in Databricks while it is registered as an Iceberg table in Glu...
Many organizations have successfully built Generative AI proofs of concept. The bigger challenge is deploying enterprise-grade AI systems that are secure, scalable, and deliver measurable business value.Key capabilities that make a difference include...
3 biggest pieces:1. Reliabilty: How do you ensure this app will not breaks in production ?2. Evaluation: How do we evaluate at scale and whether the AI solution is yeilding better results than a human does it ?3. Value: As POC/demo(s) is now happenin...
I am doing a case choice analysis on whether I should have my data in Snowflake to support my Customer workloads or have them migrated to Databricks to do the same.What has been your experience especially in handling large volumes of data especially ...
Given the pace in the new AI era, which platform have the most inovations that brings value to customers clearly wins it. And we can see Databricks is releasing new features on weekly basis, they are clearly not behind.
Hello members,I have access to my databricks environment and I see it already includes sample data sets (under "samples" database).I find the learning paths are very basic and not any close the real use cases. Are there any advanced hands-on use case...
Hi Manjunath How are you doing today? I felt the same way starting out, so I know what you mean. The best structured resource is dbdemos.ai, maintained by the Databricks team. It installs full end to end demos into your workspace in one line (pip in...
it seems Databricks is moving from traditional RAG pipelines toward agent-native architectures with Lakebase Search, Genie Agents, MCP integration, and semantic business context.For teams building enterprise GenAI applications today:Are you still bui...
Given Databricks is now very mature in the Agentic/AI landscape comparing to 1 year ago, we are starting to leverage most of the capabilities from Databricks such as:1. Vector Search for Vector Database -> Transitioning to Lakebase Search.2. Lakebase...
Hello Databricks community,I am trying to connect to Azure SQL DB from Databricks free edition, however ran into below error-[FAILED_JDBC.CONNECTION] Failed JDBC on the operation: Failed to connect to the database. SQLSTATE: HV000I have tried adding ...
Hi ,How are you doing today? as per my understanding, this isn't an Azure firewall problem, so adding IP ranges won't help. That's also why opening 0.0.0.0 to 255.255.255.255 made no difference.The real reason is on the Databricks side. Free Edition...
I am planning to create an additional workpace in Databricks. I just want to know if there is any additional cost in terms of billing. thanks in advance for the reponse.
There is no charge to create workspaces. Please go ahead and create one. While you create one, pay close attention to the networking tab. By default, it creates a virtual network and the associated costs for it, you will have to bear. You can simply ...
Dear Community Experts,I need your expert advice and suggestions on development of data quality framework. What are powerfull data quality tools or libraries are good to go for development of data quality framework in Databricks ? Please guide team.R...
There are plenty of tools available but one that's closely integrated with Databricks is DQX. It is quite an awesome tool that really covers most of the rules you would need. It also has the ability to develop complex SQL queries wherein joins and ag...
| User | Count |
|---|---|
| 1899 | |
| 1035 | |
| 931 | |
| 480 | |
| 366 |