Join tens of thousands of data leaders, engineers, scientists and architects from around the world at Moscone Center in San Francisco, June 10–13. Explore the latest advances in Apache Spark™, Delta Lake, MLflow, LangChain, PyTorch, dbt, Prest...
- 7646 Views
- 1 replies
- 4 kudos
02-12-2024
Get your first look at DBRX April 25, 2024 | 8 AM PT If you’re using off-the-shelf LLMs to build GenAI applications, you’re probably struggling with quality, privacy and governance issues. What you need is a way to cost-effectively build a custom LLM...
- 2304 Views
- 3 replies
- 2 kudos
2 weeks ago
AI has the power to address the data warehouse’s biggest challenges — performance, governance and usability — thanks to its deeper understanding of your data and how it’s used. This is data intelligence and it’s revolutionizing the way you query, man...
- 3022 Views
- 5 replies
- 1 kudos
2 weeks ago
Community Activity
- 6 Views
- 0 replies
- 0 kudos
Workflow file arrival trigger - does it apply to overwritten files?
I am exploring the use of the "file arrival" trigger for a workflow for a use case I am working on. I understand from the documentation that it checks every minute for new files in an external location, then initiates the workflow when it detects a ...
- 6 Views
- 0 replies
- 0 kudos
- 631 Views
- 3 replies
- 2 kudos
Resolved! Update regarding Community Reward Store
Hi Team,Is there any update on the Community Reward Store, as it's been discontinued from the old portal, and we still can't see the new portal for that.Is there any expected date when this will be available for community members?
- 631 Views
- 3 replies
- 2 kudos
- 150 Views
- 1 replies
- 0 kudos
Using SQL for Structured Streaming
Hi!I'm new to Databricks. I'm trying to create a data pipeline with structured streaming. A minimal example data pipeline would look like: read from upstream Kafka source, do some data transformation, then write to downstream Kafka sink. I want to do...
- 150 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Ok I figured out why I was getting an error on the usage of `read_kafka`. My default cluster was set up with the wrong Databricks runtime
- 0 kudos
- 494 Views
- 2 replies
- 0 kudos
Update DeltaTable on column type ArrayType(): add element to array
Hi all,I need to perform an Update on a Delta Table adding elements to a column of ArrayType(StringType()) which is initialized empty.Before UpdateCol_1 StringType()Col_2 StringType()Col_3 ArrayType()ValVal[ ]After UpdateCol_1 StringType()Col_2 Strin...
- 494 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @carlosancassani, It seems like you’re trying to append a string to an array column in a Delta table. The error you’re encountering is because you’re trying to assign a string value to an array column, which is not allowed due to type mismatch. To...
- 0 kudos
- 10 Views
- 0 replies
- 0 kudos
Nulls in Merge
If you are going to handle any null values in your MERGE condition, better watch out for your syntax #databricks
- 10 Views
- 0 replies
- 0 kudos
- 8 Views
- 0 replies
- 0 kudos
Installing CrowdStrike Falcon Sensor on Databricks Workers
Greetings,Does anyone here have experience deploying the CrowdStrike Falcon sensor on Databricks worker instances? For context, the cluster is deployed in AWS and we use a Databricks Ubuntu 20.04 AMI. Databricks allows adding a bootstrap/startup scri...
- 8 Views
- 0 replies
- 0 kudos
- 16 Views
- 3 replies
- 0 kudos
Databricks Points and Community Rewards Store
Hi! Oddly I wasnt able to find any information online about the following. Would expect there to be more documentation or a blog post laying it all out.1) Where the points accrued on the Databricks profile is listed?2) What website/link do we navigat...
- 16 Views
- 3 replies
- 0 kudos
Latest Reply
- 0 kudos
oh no! I was actually really excited when i heard about the reward system
- 0 kudos
- 1707 Views
- 8 replies
- 73 kudos
Trying to claim reward points but its not reflecting my points
Hi Team,Can anyone help me why my reward point still showing 0 balance. My databricks community points is not reflecting on reward claim portal.i was login first time . Also I was wait for 3 business days but its still not reflecting
- 1707 Views
- 8 replies
- 73 kudos
Latest Reply
- 73 kudos
Can you also share the link for the reward points redemption?
- 73 kudos
- 16 Views
- 0 replies
- 0 kudos
Command to display all computes available in your workspace
Hi Is there a command you could use to list all computes configured in your workspace (active and non-active). This would be really helpful for anyone managing the platfrom to pull all the meta data (tags ,etc) and quickly evaluate all the configura...
- 16 Views
- 0 replies
- 0 kudos
- 133 Views
- 4 replies
- 0 kudos
java.lang.ClassNotFoundException: com.johnsnowlabs.nlp.DocumentAssembler
I am trying to serve a pyspark model using an endpoint. I was able to load and register the model normally. I could also load that model and perform inference but while serving the model, I am getting the following error: [94fffqts54] ERROR StatusLog...
- 133 Views
- 4 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @Shreyash, It looks like your code is encountering a java.lang.ClassNotFoundException for the com.johnsnowlabs.nlp.DocumentAssembler class while serving your PySpark model. This error occurs when the required class is not found in the classpath. ...
- 0 kudos
- 193 Views
- 6 replies
- 0 kudos
Spark CSV file read option to read blank/empty value from file as empty value only instead Null
Hi,I am trying to read one file which having some blank value in column and we know spark convert blank value to null value during reading, how to read blank/empty value as empty value ?? tried DBR 13.2,14.3I have tried all possible way but its not w...
- 193 Views
- 6 replies
- 0 kudos
Latest Reply
- 0 kudos
dont quote something from stackoverflow because those are old version in spark tried.. have you tried the thing on your own to verify if this really working or not in spark3??
- 0 kudos
- 2950 Views
- 2 replies
- 0 kudos
�� Exciting Changes to the Databricks Community + Recognition System! We have some fantastic news to share with you! In the coming week, we wi...
Exciting Changes to the Databricks Community + Recognition System! We have some fantastic news to share with you! In the coming week, we will be launching a fully revamped Databricks Community, aiming to enhance your overall experience and introduce...
- 2950 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
@Sujitha is Databricks Community Rewards Store relaunched ??
- 0 kudos
- 102 Views
- 1 replies
- 0 kudos
Databricks Voucher
Hi Folks,I am looking for a free voucher to take up the "Data Analyst Associate certification". Could anyone guide please ?
- 102 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
@depp Apologies we currently don't have any promotion for discount voucher to offer. But here some event which might can help you https://www.databricks.com/lp/data-intelligence-days.
- 0 kudos
- 105 Views
- 1 replies
- 0 kudos
Getting client.session.cache.size warning in pyspark code using databricks connect
Hi Community, I have setup a jupyter notebook in a server and installed databricks connect in its kernel to leverage my databricks cluster compute in the notebook and write pyspark code. Whenever I run my code it gives me below warning: ```WARN Spark...
- 105 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
The warning indicates that the client cache (used to manage connections between your local environment and the Databricks cluster) has reached its maximum size (20 sessions). When this limit is reached, the oldest session is closed to make room for a...
- 0 kudos
- 75 Views
- 0 replies
- 0 kudos
Ephemeral storage how to create/mount.
Hi,I'm looking for information how to create/mount ephemeral storage to Databricks driver node in Azure Cloud. Does anyone have any experience working with ephemeral storage?Thanks,
- 75 Views
- 0 replies
- 0 kudos
Latest from our Blog
MLOps Gym - Beginners Guide to Cluster Configuration for MLOps
When setting up compute, there are many options and knobs to tweak and tune, and it can get quite overwhelming very quickly. To help you with optimally configuring your clusters, we have broken dow...
Why Serverless Databricks SQL is the best for BI workloads: Part IV - Remote Query Result Cache
Authors: Andrey Mirskiy (@AndreyMirskiy) and Marco Scagliola (@MarcoScagliola) Welcome to the fourth part (#4) of our blog series on “Why Databricks SQL Serverless is the best fit for BI workloads”. I...
Attributing Costs in Databricks Model Serving
Databricks Model Serving provides a scalable, low-latency hosting service for AI models. It supports models ranging from small custom models to best-in-class large language models (LLMs). In this blog...
MLOps Gym - Unity Catalog Setup for MLOps
Unity Catalog (UC) is Databricks unified governance solution for all data and AI assets on the Data Intelligence Platform. UC is central to implementing MLOps on Databricks as it is where all your as...
