Join us for the return of the Virtual Learning Festival! Mark your calendars from 9 April - 30 April 2025! Upskill across data engineering, data analysis, machine learning, and generative AI. Join the thousands who have elevated their career with...
- 15899 Views
- 49 replies
- 16 kudos
a month ago
Date & Time:8 April (Tuesday), 11 AM – 3 PM SGT Registration Page:Register Here: https://events.databricks.com/FY260408-SD-AIBIforSelf-ServiceAnalytics What You'll Learn:In this training session, you will learn how to self-serve business insights ...
- 1834 Views
- 1 replies
- 2 kudos
a week ago
Learn the foundational principles of the lakehouse architecture with three short self-paced videos The architecture of your data platform directly affects the reliability, performance and scalability of your data and AI initiatives. The lakehouse arc...
- 4527 Views
- 0 replies
- 2 kudos
4 weeks ago
Be part of a global movement! Connect with 22,000 data enthusiasts at 700+ sessions, keynotes, and hands-on training at this year’s Data + AI Summit. Whether you’re into data intelligence, governance, AI, or data warehousing, this is your chance to l...
- 5589 Views
- 1 replies
- 3 kudos
02-19-2025
Meet Takuya Omi (尾美拓哉), a passionate Data Engineer at NS Solutions and a valuable member of the Databricks Community. Takuya's career journey began in the marketing division of a financial institution, where his early work with SQL and machine learni...
- 2438 Views
- 4 replies
- 6 kudos
Monday
Community Activity
- 42 Views
- 1 replies
- 0 kudos
Can a SQL Warehouse Pro be shared across multiple workspaces
I'm currently using a SQL Warehouse Pro in one of my Databricks workspaces, and I’m trying to optimize costs. Since the Pro Warehouse can be quite expensive to run, I’d prefer not to spin up additional instances in each workspace.Is there any way to ...
- 42 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
hi @jfid A SQL Warehouse Pro instance cannot be shared directly across multiple Databricks workspaces. Each workspace requires its own SQL Warehouse instance, even if the compute and data access needs are similar. This is because compute resources li...
- 0 kudos
- 4 Views
- 0 replies
- 0 kudos
import openai | challenges
Hello World,I am facing some challenges running OpenAI on Databricks using Azure. I can easily pip install openai.However, problems start once I run the python line: import openai. After that, I can a series of errors and never ending error messages....
- 4 Views
- 0 replies
- 0 kudos
- 31 Views
- 0 replies
- 0 kudos
Databricks data engineer associate voucher
Is there any way where I can get voucher for Databricks Data engineer associate exam ?What are the opportunities through which I can get them ?
- 31 Views
- 0 replies
- 0 kudos
- 28 Views
- 0 replies
- 0 kudos
Delete files from databricks Volumes based on trigger
Hi,I noticed there's a file arrival trigger option in the workflow but cant see delete trigger option. However, let's say I want to delete files from the Databricks volume based on this trigger, and also remove the corresponding records from the bron...
- 28 Views
- 0 replies
- 0 kudos
- 44 Views
- 0 replies
- 0 kudos
How to set DLT pipeline warning alert?
Hi there,The example description of custom event hooks in the documentation is not clear enough, I do not know how to implement it inside python functions. event-hooks My Code: %python # Read the insertion of data raw_user_delta_streaming=spark.rea...
- 44 Views
- 0 replies
- 0 kudos
- 43 Views
- 0 replies
- 0 kudos
Wisconsin Databricks User Group Meetup
Wisconsin Databricks User Group Meetup - Tuesday April 22nd, 2025Join us for an exciting session diving deep into AI Agents in Databricks!! Whether you're a seasoned Databricks professional or just starting, this event has something for everyone. Dat...
- 43 Views
- 0 replies
- 0 kudos
- 1880 Views
- 5 replies
- 1 kudos
Possible to use `params` argument of `mlflow.pyfunc.PythonModel` deployed to Databricks endpoint?
I'm deploying a custom model using the `mlflow.pyfunc.PythonModel` class as described here. My model uses the `params` argument in the `predict` method to allow the user to choose some aspects of the model at inference time. For example:class CustomM...
- 1880 Views
- 5 replies
- 1 kudos
Latest Reply
- 1 kudos
I am able to send params now. We can send params as part of data_json request to the api. data_json = { 'dataframe_split': { 'index': data_index, 'columns': columns, 'data': data }, "client_request_id": "client_request_...
- 1 kudos
- 5983 Views
- 6 replies
- 1 kudos
Resolved! How to let Business Users edit tables in Databricks
Hi Community!I have the requirement that business users shall be able to edit/update tables in Unity Catalog, e.g. master data records, mapping tables. I also want thes actions to be logged for auditing/troubleshooting.Is there any simple solution to...
- 5983 Views
- 6 replies
- 1 kudos
Latest Reply
- 1 kudos
Just checking in to see if there are any new updates on this.
- 1 kudos
- 46 Views
- 0 replies
- 0 kudos
Trying to connect SFTP directly in Databricks
Hi, As a proof of concept, I have created an ADLS and enabled SFTP> Created Local User and SSH Private key. Now, I am trying to connect this SFTP connection directly in Databricks to create table or data frame. Below is the code snipper I have used a...
- 46 Views
- 0 replies
- 0 kudos
- 58 Views
- 0 replies
- 0 kudos
Databricks MCP Server
Hey, couldn't find a working Databricks MCP so I forked one and fixed it: https://github.com/markov-kernel/databricks-mcp This allows you to (using Cursor, Claude Code, Windsurf, Claude, etc.):* Manage Databricks Clusters: List, create, terminate, ...
- 58 Views
- 0 replies
- 0 kudos
- 11973 Views
- 2 replies
- 1 kudos
Unity Catalog permissions approach for strict controls when creating tables
Hi all,I'm deploying Unity Catalog into a large enterprise with quite strict controls. I want to give my users some autonomy over creating tables from upstream datasets they have select access on, but also restrict these controls. I've been through a...
- 11973 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
@ajbush did you workout your requirements in a way you wanted? appreciate if you can share insights here.
- 1 kudos
- 67 Views
- 0 replies
- 0 kudos
What's the best way to get from Python dict > JSON > PySpark and apply as a mapping to a dataframe?
I'm migrating code from Python Linux to Databricks PySpark. I have many mappings like this: { "main": { "honda": 1.0, "toyota": 2.9, "BMW": 5.77, "Fiat": 4.5, },}I exported using json.dump, saved to s3 and was able to import with sp...
- 67 Views
- 0 replies
- 0 kudos
- 687 Views
- 1 replies
- 0 kudos
Generating Multiple Excels for SQL Query
Hello,I am getting "OSError: Errno 95: Operation not supported for the code below. I have 'openpyxl 3.1.5' installed on the cluster and have imported all required modules. I am sure this is something small, but I can't put my finger on it why this is...
- 687 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @libpekin, is filepath pointing to a DBFS path? Because writing directly to DBFS paths using to_excel() is not supported due to DBFS limitations with certain file operations, especially random writes.As a workaround, first save the Excel file to a...
- 0 kudos
- 338 Views
- 4 replies
- 0 kudos
Capture data from a Specific SharePoint Site (List) in M365 into Azure DataBricks
Hello. We are using Azure Databricks and would like to ingest data from a specific M365 SharePoint Online Site/List. I was originally trying to use this recommendation, https://learn.microsoft.com/en-us/answers/questions/2116616/service-principal-a...
- 338 Views
- 4 replies
- 0 kudos
Latest Reply
- 0 kudos
Maybe, but at this point, we just need a method to get data from the specific SharePoint Online site reliability to pull into Databricks. So open to the easiest, most efficient method. Thanks.
- 0 kudos
- 81 Views
- 0 replies
- 0 kudos
Hydra configuration and job parameters of DABs
Hello Community,I'm trying to create a job pipeline in Databricks that runs a spark_python_task, which executes a Python script configured with Hydra. The script's configuration file defines parameters, such as id.How can I pass this parameter at the...
- 81 Views
- 0 replies
- 0 kudos
Announcements
| User | Count |
|---|---|
| 106 | |
| 43 | |
| 22 | |
| 17 | |
| 14 |
Latest from our Blog
How to Assess and Adopt New Technologies Using Data-Driven Proof of Concept
In this article, I share a #data-driven method for assessing and adopting new technologies and steps for creating or evaluating a strong proposal. This includes using statistical measures to evaluate ...
[PARTNER BLOG] Introducing Databricks Native Master Data Management (MDM) - Entity Resolution
LakeFusion’s MDM solution delivers a single source of truth by leveraging advanced entity resolution and deduplication algorithms. By unifying fragmented data across systems, we ensure accurate, consi...
💡 ML Training Tip Of The Week #5: Enable fine-grained data frequency in AutoML forecast
Customers with high-frequency time series data often need forecasts at a finer granularity based on their data sources. For example, an energy supply company may receive energy demand data every 15 mi...
💡 ML Training Tip Of The Week #4: Speed up your ML workload with one toggle
Machine learning workload can be time-consuming due to training data processing such as ETL and feature engineering and iterative model training steps. Recent ML Runtime releases have made both faster...
