- 616 Views
- 1 replies
- 1 kudos
- 616 Views
- 1 replies
- 1 kudos
- 1 kudos
We are going to need a little more information to better help you. What is the scenario? Louis
- 1 kudos
- 2022 Views
- 1 replies
- 1 kudos
Why keep both Azure OpenAI and Databricks?
Hi everyone,I’m curious to hear your thoughts on the benefits of having both Azure OpenAI and Azure Databricks within the same ecosystem.From what I can see, Databricks provides a strong foundation for data engineering, governance, and model lifecycl...
- 2022 Views
- 1 replies
- 1 kudos
- 1 kudos
Two use case I can think of is RAG:Use Databricks for vector indexing (e.g., via Delta Lake or FAISS) and Azure OpenAI for inference.Example: A chatbot that queries Databricks-hosted documents and uses GPT-4 for response generation.Agentic Workflows:...
- 1 kudos
- 1040 Views
- 2 replies
- 1 kudos
Resolved! Ingesting data from APIs Like Shopify (for orders), Meta Ads, Google Ads etc
Hi,I am and trying to create some table by calling APIs of Shopify/Meta Ads/Google Ads and so on. Where will I make the API call ? Is making API calls in Notebooks considered standard way to ingest in these cases. I intend to make a daily call to ge...
- 1040 Views
- 2 replies
- 1 kudos
- 745 Views
- 0 replies
- 4 kudos
Databricks One
Databricks One is a user interface designed for business users, giving them a single, intuitive entry point to interact with data and AI in Azure Databricks, without needing to navigate technical concepts such as clusters, queries, models, or noteboo...
- 745 Views
- 0 replies
- 4 kudos
- 432 Views
- 0 replies
- 0 kudos
SQL warehouse: A materialized view is the simplest and cost-efficient way to transform your data
Materialized views running on SQL warehouse are super cost-efficient, and additionally, it is a really simple and powerful data engineering tool - just be sure that Enzyme updates it incrementally. Read more: - https://databrickster.medium.com/sql-wa...
- 432 Views
- 0 replies
- 0 kudos
- 568 Views
- 0 replies
- 2 kudos
The purpose of your All-Purpose Cluster
Small, hidden but useful cluster setting.You can set that no jobs are allowed on the all-purpose cluster.Or vice versa, you can set an all-purpose cluster that can be used only by jobs. read more: - https://databrickster.medium.com/purpose-for-your-...
- 568 Views
- 0 replies
- 2 kudos
- 723 Views
- 1 replies
- 1 kudos
How to Integrate Machine Learning Model Development with Databricks Workflows?
Hey everyone I’m currently exploring machine learning model development and I’m interested in understanding how to effectively integrate ML workflows within Databricks.Specifically, I’d like to hear from the community about:How do you structure ML pi...
- 723 Views
- 1 replies
- 1 kudos
- 1 kudos
You can integrate machine learning model development into Databricks Workflows pretty smoothly using the platform’s native tools. The main idea is to treat your ML lifecycle (data prep → training → evaluation → deployment) as a series of tasks within...
- 1 kudos
- 7369 Views
- 1 replies
- 0 kudos
Support for managed identity based authentication in python kafka client
We followed this document https://docs.databricks.com/aws/en/connect/streaming/kafka?language=Python#msk-aad to use Kafka client to read events from our event hub for a feature.As part of the SFI, the guidance is to move away from client secret and u...
- 7369 Views
- 1 replies
- 0 kudos
- 0 kudos
Currently, Databricks does not support using Managed Identities directly for Kafka client authentication (e.g., MSK IAM or Event Hubs Kafka endpoint) in Python Structured Streaming connections. However, there is a supported and secure alternative tha...
- 0 kudos
- 4753 Views
- 2 replies
- 1 kudos
Resolved! Data bricks is not mounting with storage account giving java lang exception error 480
Hi Everyone,I am currently facing an issue with in our Test Environment where Data bricks is not able to mount with the storage account and we are using the same mount in other environments those are Dev,Preprod and Prod and it works fine there witho...
- 4753 Views
- 2 replies
- 1 kudos
- 1 kudos
This issue in your Test environment, where Databricks fails to mount an Azure Storage account with the error java.lang.Exception: 480, is most likely related to expired credentials or cached authentication tokens, even though the same configuration w...
- 1 kudos
- 9574 Views
- 5 replies
- 1 kudos
Spatial Queries
Hi,I'm trying to execute the following code:%sqlSELECT LSOA21CD, ST_X(ST_GeomFromWKB(Geom_Varbinary)) AS STX, ST_Y(ST_GeomFromWKB(Geom_Varbinary)) AS STYFROM ordnance_survey_lsoas_december_2021_population_weighted_centroidsWHERE LSOA21CD ...
- 9574 Views
- 5 replies
- 1 kudos
- 1 kudos
@Corar You might want to enable that explicitly by setting 'spark.databricks.geo.st.enabled' configuration to value 'true'.
- 1 kudos
- 2779 Views
- 6 replies
- 0 kudos
Getting [08S01/500593] Can't connect to database - [Databricks][JDBCDriver](500593) Communication
I am getting below error connecting a databricks instance using JDBC driver .ERROR: [08S01/500593] Can't connect to database - [Databricks][JDBCDriver](500593) Communication link failure. Failed to connect to server. Reason: HTTP Response code: 401, ...
- 2779 Views
- 6 replies
- 0 kudos
- 0 kudos
I am trying to connect Databricks from Mainframe z/OS using JDBC driver and using below IBM Java version java version "11.0.26" 2025-01-21IBM Semeru Runtime Certified Edition for z/OS 11.0.26.0 (build 11.0.26+4)IBM J9 VM 11.0.26.0 (build z/OS-Release...
- 0 kudos
- 2820 Views
- 5 replies
- 1 kudos
Resolved! External MCP representing user data permissions
Hello Community!I am writing to you with a question and hope that you will help me to find the right approach.I am building AI Enterprise System and the organization store the data on Data Bricks. To access the given data, you have to raise a request...
- 2820 Views
- 5 replies
- 1 kudos
- 1 kudos
Ignore for now you have MCP Server.The problem you are trying to solve1) An AI Agent needs to access data inside Databricks 2) The agent need to operate at the user's permissionsThere are muliple paths1) Directly using OAuth/HTTPhttps://docs.databric...
- 1 kudos
- 2194 Views
- 1 replies
- 1 kudos
CREATE Community_User_Group [IF NOT EXISTS] IN MADRID(SPAIN)
Hi,I would like to get some support in creating a Community User Group in Madrid, Spain. It would be nice to host events/meetings/discussions ...Regards,Ángel
- 2194 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi Ángel,I see your post is from quite some time ago, but I wanted to say that I’d also love to see a Databricks User Group here in Madrid.Although I’m not new to Databricks, I haven’t really taken much advantage of the community so far due to lack o...
- 1 kudos
- 607 Views
- 1 replies
- 0 kudos
How to Optimize Spark Jobs in Databricks for Large-Scale Geospatial Data Processing?
I’m currently analyzing a large geospatial dataset focused on Michigan county boundaries and map data, and I’m using Apache Spark on Databricks to process and transform millions of records.Even though I’ve optimized basic things like repartitioning, ...
- 607 Views
- 1 replies
- 0 kudos
- 0 kudos
I do not have experience with geospatial data on databricks.But I do know that since a while, Sedona can be installed on Databricks.Sedona is created for large-scale geospatial data processing. Sounds like something for you no?https://sedona.apache....
- 0 kudos
- 18026 Views
- 4 replies
- 0 kudos
Resolved! Unable to add column comment on a View. Any way to update comments on multiple columns in bulk?
I noticed that unlike "Alter Table" there is no "Alter View" command to add comment on a column in the existing view. This is a regular view created on Tables (and not Materialized view). If the underlying table column has comment then the View inh...
- 18026 Views
- 4 replies
- 0 kudos
