cancel
Showing results for 
Search instead for 
Did you mean: 
Page Title

Welcome to the Databricks Community

Discover the latest insights, collaborate with peers, get help from experts and make meaningful connections

cancel
Showing results for 
Search instead for 
Did you mean: 
Building DBRX-class Custom LLMs with Mosaic AI Training

We recently introduced DBRX: an open, state-of-the-art, general-purpose LLM. DBRX was trained, fine-tuned, and evaluated using Mosaic AI Training, scaling training to 3072 NVIDIA H100s and processing more than 12 trillion tokens in the process. Train...

  • 503 Views
  • 0 replies
  • 0 kudos
Friday
Accurate, Safe and Governed: How to Move GenAI from POC to Production

In the realm of AI, achieving accuracy is paramount. The publication delves into techniques for refining models to ensure they reliably deliver precise outcomes in real-world scenarios. It covers methodologies such as continuous monitoring, data augm...

  • 841 Views
  • 0 replies
  • 1 kudos
Friday
Exciting Announcement: Introducing our Learning Library!

Dive into a world of knowledge with our brand-new Learning Library! Whether you prefer self-paced exploration or guided instruction, our extensive range of courses caters to all personas and learning styles. From beginners to experts, there's someth...

  • 1034 Views
  • 1 replies
  • 0 kudos
Tuesday
Databricks Community Social, May 2024 - Speaker session around Training offerings

Are you ready to enhance your socializing adventure? Our Monthly Community Social is here.   Date: May 23, 2024 Time: 8.30 PM IST | 8 AM PT Location: Virtual Event (Link to join) What's in Store for You?   Exciting Icebreaker Activities Engaging Disc...

  • 2186 Views
  • 1 replies
  • 1 kudos
2 weeks ago

Community Activity

standup1
by New Contributor III
  • 13 Views
  • 1 replies
  • 0 kudos

How to exclude/skip a file temporarily in DLT

Hi,Is there any way to exclude a file from the dlt pipeline (autoload) run temporarily? What I mean is that I want to be able to exclude a specific file until I decided to include it in the load? I can't control the files or the location where they a...

  • 13 Views
  • 1 replies
  • 0 kudos
Latest Reply
brockb
New Contributor III
  • 0 kudos

Hi, I'm not aware of default Autoloader functionality that does what you're looking to do given that Autoloader is designed to incrementally ingest data as it arrives in cloud storage. Can you describe more about: "...exclude a specific file until I ...

  • 0 kudos
radothede
by New Contributor II
  • 24 Views
  • 0 replies
  • 0 kudos

Databricks DBU pre-purchase

Hello there,are pre-purchased DBU still valid? Can we use it?https://learn.microsoft.com/en-us/azure/cost-management-billing/reservations/reservation-discount-databricksCan someone please explain how it works in practice, by example?What if I pre-puc...

Administration & Architecture
DBCU
optimize costs
Pre-purchase
reservation
  • 24 Views
  • 0 replies
  • 0 kudos
VGS777
by New Contributor III
  • 133 Views
  • 2 replies
  • 0 kudos

Resolved! Regarding Databricks Terraform to new user

Hey FolksI am new to terraform and databricksI have usecase I want to create new user or add them to databricks workspace. And assign role to this user. And also assign cluster to this new userAfter 12hrs I want to delete this new user and also these...

  • 133 Views
  • 2 replies
  • 0 kudos
Latest Reply
VGS777
New Contributor III
  • 0 kudos

Thanks for this information 

  • 0 kudos
1 More Replies
pjv
by New Contributor
  • 381 Views
  • 1 replies
  • 0 kudos

Asynchronous API calls from Databricks Workflow job

Hi all,I have many API calls to run on a python Databricks notebook which I then run regularly on a Databricks Workflow job. When I test the following code on an all purpose cluster locally i.e. not via a job, it runs perfectly fine. However, when I ...

  • 381 Views
  • 1 replies
  • 0 kudos
Latest Reply
mhiltner
New Contributor II
  • 0 kudos

Would you mind sharing the cluster setup for both cases? I'd make sure that databricks Runtime is the same for both and check the number of workers allocated in each cluster. 

  • 0 kudos
robbe
by New Contributor II
  • 29 Views
  • 1 replies
  • 1 kudos

Get job ID from Asset Bundles

When using Asset Bundles to deploy jobs, how does one get the job ID of the resources that are created?I would like to deploy some jobs through asset bundles, get the job IDs, and then trigger these jobs programmatically outside the CI/CD pipeline us...

  • 29 Views
  • 1 replies
  • 1 kudos
Latest Reply
mhiltner
New Contributor II
  • 1 kudos

Hey, not sure if this will do the trick, but i've thought about two workarounds: 1. Check if the "databricks bundle run my_job" suits your case. It accepts the name as the key to run here.  2. Would it be an option for you to use databricks jobs list...

  • 1 kudos
wyzer
by Contributor II
  • 4353 Views
  • 8 replies
  • 4 kudos

Resolved! How to pass parameters in SSRS/Power BI (report builder) ?

Hello,In SSRS/Power BI (report builder), how to query a table in Databricks with parameters please ?Because this code doesn't works :SELECT * FROM TempBase.Customers WHERE Name = {{ @P_Name }}Thanks.

  • 4353 Views
  • 8 replies
  • 4 kudos
Latest Reply
Nj11
New Contributor II
  • 4 kudos

Hi, I am not able to see the data in SSRS while I am using date parameters but with manual dates data is populating fine. The database is pointing to databricks. I am not sure what I am missing here. Please help me in this. ThanksI am trying with que...

  • 4 kudos
7 More Replies
nakaxa
by Visitor
  • 32 Views
  • 0 replies
  • 0 kudos

Fastest way to write a Spark Dataframe to a delta table

I read a huge array with several columns into memory, then I convert it into a spark dataframe,  when I want to write to a delta table it using the following command it takes forever (I have a driver with large memory and 32 workers) : df_exp.write.m...

  • 32 Views
  • 0 replies
  • 0 kudos
mh_db
by New Contributor III
  • 134 Views
  • 2 replies
  • 0 kudos

Unable to connect to oracle server from databricks notebook in AWS

I'm trying to connect to oracle server hosted in azure from AWS databricks notebook but seems the connection keeps timing out. I tested the connection IP using telnet <hostIP> 1521 command from another EC2 instance and that seems to reach the oracle ...

Data Engineering
AWS
oracle
TCP
  • 134 Views
  • 2 replies
  • 0 kudos
Latest Reply
Yeshwanth
Valued Contributor II
  • 0 kudos

@mh_db good day! Could you please confirm the Cluster type you used for testing? Was it a Shared Cluster, an Assigned/Single-User Cluster, or a No-Isolation cluster? Could you please try the same on the Assigned/Single User Cluster and No Isolation c...

  • 0 kudos
1 More Replies
marvin1
by New Contributor III
  • 129 Views
  • 3 replies
  • 0 kudos

Hostname redaction in delta table

I am ingesting job-cluster failure notifications that we send to OpsGenie into a delta table to automate the creation and tracking of Jira tickets.  The alert notification includes the job run url, which we use to quickly respond to job failures.  Ho...

  • 129 Views
  • 3 replies
  • 0 kudos
Latest Reply
marvin1
New Contributor III
  • 0 kudos

I tracked this issue down to running the notebook on a shared Unity-enabled cluster.  When scheduling the notebook to run on a job cluster with single user mode, this issue does not occur.

  • 0 kudos
2 More Replies
dbengineer516
by New Contributor
  • 296 Views
  • 3 replies
  • 1 kudos

/api/2.0/preview/sql/queries API only returning certain queries

Hello,When using /api/2.0/preview/sql/queries to list out all available queries, I noticed that certain queries were being shown while others were not. I did a small test on my home workspace, and it was able to recognize certain queries when I defin...

  • 296 Views
  • 3 replies
  • 1 kudos
Latest Reply
brockb
New Contributor III
  • 1 kudos

Hi,How many queries were returned in the API call in question? The List Queries documentation describes this endpoint as supporting pagination with a default page size of 25, is that how many you saw returned? Query parameters page_size integer <= 10...

  • 1 kudos
2 More Replies
thiagoawstest
by Visitor
  • 36 Views
  • 0 replies
  • 0 kudos

Migration Azure to AWS

Hello, today I use Azure Databricks, I want to migrate my wordspaces to AWS Databricks. What is the best practice, which path should I follow?, I didn't find anything in the documentation.thanks.

  • 36 Views
  • 0 replies
  • 0 kudos
MYB24
by New Contributor III
  • 3793 Views
  • 5 replies
  • 0 kudos

Resolved! Error: cannot create mws credentials: invalid Databricks Account configuration

Good Evening, I am configuring databricks_mws_credentials through Terraform on AWS.  I am getting the following error:Error: cannot create mws credentials: invalid Databricks Account configuration││ with module.databricks.databricks_mws_credentials.t...

Data Engineering
AWS
credentials
Databricks
Terraform
  • 3793 Views
  • 5 replies
  • 0 kudos
Latest Reply
TMD
New Contributor III
  • 0 kudos

just to add a context for probably the underlying issue requiring an account level service principal (with OAuth).I experienced the same issue while using username and password as in the case how TF provider was configured for existing workspaces cre...

  • 0 kudos
4 More Replies
Avinash_Narala
by New Contributor III
  • 60 Views
  • 1 replies
  • 0 kudos

Application Deployment in Marketplace

Hi,I want to deploy my flask application in Databricks Marketplace.How can I do it?Can you please share the details

  • 60 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @Avinash_Narala,  For more information, you can refer to the following resources: Tutorial: Deploy and query a feature serving endpoint - DatabricksPython Flask App migrate to Databricks - Microsoft Q&ADeploy custom models | Databricks on AWSHo...

  • 0 kudos
m208205
by New Contributor
  • 295 Views
  • 1 replies
  • 0 kudos

Difference in support for partitions between hive and Unity

The Unity migration guide (https://docs.databricks.com/en/data-governance/unity-catalog/migrate.html#before-you-begin) states the following:Unity Catalog manages partitions differently than Hive. Hive commands that directly manipulate partitions are ...

Data Engineering
Unity Catalog
  • 295 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @m208205, The Unity Catalog and Hive handle partitions differently, and this distinction is important when migrating existing partitioned Delta tables from Hive to Unity Catalog. Here are the key differences: Partition Management: Unity Catalo...

  • 0 kudos
hzh
by New Contributor
  • 273 Views
  • 1 replies
  • 0 kudos

Credential passthrough and Hive metastore table access controls are deprecated

Hello,Based on the recent platform release, the credential passthrough will be deprecated for runtime 15.0 and later. Our current setup involves using Databricks alongside AWS Glue and Athena, i.e. registering delta tables in AWS Glue and running oth...

Data Engineering
aws glue
Unity Catalog
  • 273 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @hzh,  This deprecation specifically affects automatic authentication to S3 buckets from Databricks clusters using the identity you use to log in to Databricks.It doesn’t prevent you from writing tables to the Hive metastore. You can still interac...

  • 0 kudos

Latest from our Blog

How to use System Tables with Overwatch

How to use System Tables with Overwatch Welcome to our blog post on integrating system tables with Overwatch! In this article, we'll delve into the exciting world of leveraging system tables to enhanc...

471Views 3kudos

Retrying dbt Runs in Databricks Workflows

Over the past few years, the variety of tools accessible to data teams has surged, with dbt emerging as a popular solution for data transformation. It empowers SQL-proficient users to craft flexible d...

480Views 0kudos