cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Community Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Shawn_Eary
by • New Contributor III
  • 6854 Views
  • 4 replies
  • 4 kudos

Resolved! Use SQL Server Management Studio to Connect to DataBricks?

The Notebook UI doesn't always provide the best experience for running exploratory SQL queries. Is there a way for me to use SQL Server Management Studio (SSMS) to connect to DataBricks? See Also:https://learn.microsoft.com/en-us/answers/questions/74...

Community Discussions
Reply
  • 6854 Views
  • 4 replies
  • 4 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 4 kudos

What you can do is define a SQL endpoint as a linked server.  Like that you can use SSMS and T-SQL.However, it has some drawbacks (no/bad query pushdown, no caching).Here is an excellent blog of Kyle Hale of databricks:Tutorial: Create a Databricks S...

  • 4 kudos
3 More Replies
Cloud_Architect
by • New Contributor III
  • 733 Views
  • 5 replies
  • 3 kudos

I am facing an issue while generating the DBU consumption report and need help.

I am trying to access the following system tables to generate a DBU consumption report, but I am not seeing this table in the system schema. Could you please help me how to access it?system.billing.inventory, system.billing.workspaces, system.billing...

Community Discussions
Reply
  • 733 Views
  • 5 replies
  • 3 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 3 kudos

Hi @Cloud_Architect, In Azure Databricks, system tables are an analytical store of your account’s operational data found in the system catalog1. These tables can be used for historical observability across your account1. To access system tables, y...

  • 3 kudos
4 More Replies
pshuk
by • New Contributor III
  • 572 Views
  • 1 replies
  • 2 kudos

ingest csv file on-prem to delta table on databricks

Hi,So I want to create a delta live table using a csv file that I create locally (on-prem). A little background: So I have a working ELT pipeline that finds newly generated files (since the last upload), and upload them to databricks volume and at th...

Community Discussions
Reply
  • 572 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
raphaelblg
Contributor III
  • 2 kudos

Hello @pshuk , Based on your description, you have an external pipeline that writes CSV files to a specific storage location and you wish to set up a DLT based on the output of this pipeline. DLT offers has access to a feature called Autoloader, whic...

  • 2 kudos
data-grassroots
by • New Contributor III
  • 1032 Views
  • 3 replies
  • 0 kudos

Resolved! Delta Sharing - Info about Share Recipient

What information do you know about a share recipient when they access a table shared to them via Delta Sharing?Wondering if we might be able to utilize something along the lines of is_member, is_account_group_member, session_user, etc for ROW and COL...

Community Discussions
Reply
  • 1032 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
data-grassroots
New Contributor III
  • 0 kudos

Now that I'm looking closer at the share credentials and the recipient entity you would really need a way to know the bearer token and relate that back to various recipient properties - databricks.name and any custom recipient property tags you may h...

  • 0 kudos
2 More Replies
Sudheer2
by • New Contributor II
  • 694 Views
  • 2 replies
  • 2 kudos

Updating Databricks SQL Warehouse using Terraform

 We can Update SQL Warehouse manually in Databricks.Click SQL Warehouses in the sidebarIn Advanced optionsWe can find Unity Catalog toggle button there! While Updating Existing SQL Warehouse in Azure to enable unity catalog using terraform, I couldn'...

Community Discussions
Reply
  • 694 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 2 kudos

Hi @Sudheer2, The Unity Catalog is a feature in Databricks SQL Warehouse that allows you to query data across multiple databases and tables seamlessly. It provides a unified view of your data.When you enable the Unity Catalog, you can access tables f...

  • 2 kudos
1 More Replies
MohsenJ
by • Contributor
  • 2157 Views
  • 7 replies
  • 0 kudos

log signature and input data for Spark LinearRegression

I am looking for a way to log my `pyspark.ml.regression.LinearRegression` model with input and signature ata. The usual example that I found around are using sklearn and they can simply do  # Log the model with signature and input example signature =...

Community Discussions
mlflow
model_registray
Reply
  • 2157 Views
  • 7 replies
  • 0 kudos
View Replies
Latest Reply
MohsenJ
Contributor
  • 0 kudos

me neither. But the mlflow documentation suggests the new version of mlflow should be able to handle Array and Objects (dict). maybe that could help? I haven't tired it myself.  Support for Array and Object types was introduced in MLflow version 2.10...

  • 0 kudos
6 More Replies
Rpswadhwa
by • New Contributor
  • 1176 Views
  • 1 replies
  • 0 kudos

Data analytics

Really excited that Data bricks is championing the cause of unifying the whole data analytics value chain. it’s a great service to the data community.

Community Discussions
Reply
  • 1176 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Rpswadhwa, We're thrilled to hear that you had a great experience at DAIS 2023! Your feedback is valuable to us, and we appreciate you taking the time to share it on the community platform.   We wanted to let you know that the Databricks Communit...

  • 0 kudos
BDietrich
by • New Contributor
  • 518 Views
  • 1 replies
  • 0 kudos

2023 Data AI summit

Have learned a lot through this experience, especially around productionalizing Gen AI models. Great time and great people!

Community Discussions
Reply
  • 518 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @BDietrich, We're thrilled to hear that you had a great experience at DAIS 2023! Your feedback is valuable to us, and we appreciate you taking the time to share it on the community platform.   We wanted to let you know that the Databricks Communit...

  • 0 kudos
yhyhy3
by • New Contributor III
  • 519 Views
  • 1 replies
  • 1 kudos

Databricks AI Summit 2023

Amazing talks from industry leaders and open source contributors! I'm astounded by the keynote presentations on Lakehouse IQ and Spark AI, really driving toward the new future of developers' and analysts' interaction with data. English as the new pro...

Community Discussions
Summit23
Reply
  • 519 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @yhyhy3, We're thrilled to hear that you had a great experience at DAIS 2023! Your feedback is valuable to us, and we appreciate you taking the time to share it on the community platform. We wanted to let you know that the Databricks Community Tea...

  • 1 kudos
maxnumbers
by • New Contributor III
  • 540 Views
  • 1 replies
  • 0 kudos

Summit has been a great combo of fun & informative!

Here from the Collins Aerospace group, and we've been enjoying the conference so far! We've learned a lot of new stuff to take back with us in our data transformation journey! The new features for being able to run a job within a job looks pretty exc...

Community Discussions
Summit23
Reply
  • 540 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @maxnumbers, We're thrilled to hear that you had a great experience at DAIS 2023! Your feedback is valuable to us, and we appreciate you taking the time to share it on the community platform. We wanted to let you know that the Databricks Community...

  • 0 kudos
Chaho
by • New Contributor
  • 364 Views
  • 1 replies
  • 0 kudos

See the future of data integrated with AI

Even if my company does not use Databricks yet, I could see the future of data with AI here at the summit.  

Community Discussions
Summit23
Reply
  • 364 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Chaho, We're thrilled to hear that you had a great experience at DAIS 2023! Your feedback is valuable to us, and we appreciate you taking the time to share it on the community platform. We wanted to let you know that the Databricks Community Team...

  • 0 kudos
rli
by • New Contributor
  • 413 Views
  • 1 replies
  • 0 kudos

delta live table merging

here at data AI summit day 3. very excited to hear about the delta live tables and it's functionality of applying changes, which actually provides a direct solution to the dilemma at our company when uploading data!

Community Discussions
Reply
  • 413 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @rli , We're thrilled to hear that you had a great experience at DAIS 2023! Your feedback is valuable to us, and we appreciate you taking the time to share it on the community platform. We wanted to let you know that the Databricks Community Team ...

  • 0 kudos
subham0611
by • New Contributor II
  • 456 Views
  • 1 replies
  • 0 kudos

How does coalesce works internally

Hi Databricks team,I am trying to understand internals of spark coalesce code(DefaultPartitionCoalescer) and going through spark code for this. While I understood coalesce function but I am not sure about complete flow of code like where its get call...

Community Discussions
Reply
  • 456 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
raphaelblg
Contributor III
  • 0 kudos

  Hello @subham0611 , The coalesce operation triggered from user code can be initiated from either an RDD or a Dataset, with each having distinct codepaths: RDD: https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/RDD...

  • 0 kudos
DavidKxx
by • New Contributor III
  • 662 Views
  • 5 replies
  • 1 kudos

Can't create branch of public git repo

Hi,I have cloned a public git repo into my Databricks account.  It's a repo associated with an online training course.  I'd like to work through the notebooks, maybe make some changes and updates, etc., but I'd also like to keep a clean copy of it. M...

Community Discussions
GIT
Reply
  • 662 Views
  • 5 replies
  • 1 kudos
View Replies
Latest Reply
NandiniN
Honored Contributor
  • 1 kudos

I get your issue, @DavidKxx. Until we do a git push on command line we do not see the Authentication failed git push origin test While in the Databricks UI, we fail early(screenshots below). We require the Databricks GitHub App as mentioned here to p...

  • 1 kudos
4 More Replies
Neeraj_Kumar
by • New Contributor
  • 409 Views
  • 1 replies
  • 0 kudos

Issues with Runtime 15.1/15.2Beta in shared access mode

We have been using runtime 14.2, share mode for our computing cluster in Databrick for quite some time.  We are now trying to upgrade to python 3.11 for some dependencies mangement, thereby requiring us to use runtime 15.1/15.2  as runtime 14.2 only ...

Community Discussions
Reply
  • 409 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Neeraj_Kumar,  Ensure that the necessary libraries are available in the repository used for installation.Verify that the library versions specified are correct and available.Consider installing the library with a different version or from a diffe...

  • 0 kudos
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels