cancel
Showing results for 
Search instead for 
Did you mean: 
Community Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

MohsenJ
by Contributor
  • 664 Views
  • 1 replies
  • 1 kudos

how to develop Notebooks on vscode for git repos?

I am able to use vscode extension + databricks connect to develop Notebooks on my local computer and run them on my databricks cluster. However I can not figure out how to develop the Notebooks that have the file `.py` extension but identified by Dat...

  • 664 Views
  • 1 replies
  • 1 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @MohsenJ, You can indeed develop and run Databricks notebooks with .py extension on your local computer using the Databricks extension for Visual Studio Code and Databricks Connect1. Here are the steps you need to follow: Enable Databricks Con...

  • 1 kudos
unity_Catalog
by New Contributor
  • 393 Views
  • 1 replies
  • 0 kudos

Assessment(Assessment job need to be deployed using Terraform)

Assessment(Assessment job need to be deployed using Terraform)1.Install latest version of UCX 2.UCX will add the assessment job and queries to the workspace3.Run the assessment using ClusterHow to write code for this by using Terraform. Can anyone he...

  • 393 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @unity_Catalog, To deploy an assessment job using Terraform, you’ll need to write a Terraform configuration that includes the following steps: Install the latest version of UCX: This will likely involve using a provider to manage software inst...

  • 0 kudos
Shawn_Eary
by Contributor
  • 7738 Views
  • 4 replies
  • 4 kudos

Resolved! Use SQL Server Management Studio to Connect to DataBricks?

The Notebook UI doesn't always provide the best experience for running exploratory SQL queries. Is there a way for me to use SQL Server Management Studio (SSMS) to connect to DataBricks? See Also:https://learn.microsoft.com/en-us/answers/questions/74...

  • 7738 Views
  • 4 replies
  • 4 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 4 kudos

What you can do is define a SQL endpoint as a linked server.  Like that you can use SSMS and T-SQL.However, it has some drawbacks (no/bad query pushdown, no caching).Here is an excellent blog of Kyle Hale of databricks:Tutorial: Create a Databricks S...

  • 4 kudos
3 More Replies
Cloud_Architect
by New Contributor III
  • 865 Views
  • 5 replies
  • 3 kudos

I am facing an issue while generating the DBU consumption report and need help.

I am trying to access the following system tables to generate a DBU consumption report, but I am not seeing this table in the system schema. Could you please help me how to access it?system.billing.inventory, system.billing.workspaces, system.billing...

  • 865 Views
  • 5 replies
  • 3 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 3 kudos

Hi @Cloud_Architect, In Azure Databricks, system tables are an analytical store of your account’s operational data found in the system catalog1. These tables can be used for historical observability across your account1. To access system tables, y...

  • 3 kudos
4 More Replies
pshuk
by New Contributor III
  • 623 Views
  • 1 replies
  • 2 kudos

ingest csv file on-prem to delta table on databricks

Hi,So I want to create a delta live table using a csv file that I create locally (on-prem). A little background: So I have a working ELT pipeline that finds newly generated files (since the last upload), and upload them to databricks volume and at th...

  • 623 Views
  • 1 replies
  • 2 kudos
Latest Reply
raphaelblg
Honored Contributor
  • 2 kudos

Hello @pshuk , Based on your description, you have an external pipeline that writes CSV files to a specific storage location and you wish to set up a DLT based on the output of this pipeline. DLT offers has access to a feature called Autoloader, whic...

  • 2 kudos
data-grassroots
by New Contributor III
  • 1107 Views
  • 3 replies
  • 0 kudos

Resolved! Delta Sharing - Info about Share Recipient

What information do you know about a share recipient when they access a table shared to them via Delta Sharing?Wondering if we might be able to utilize something along the lines of is_member, is_account_group_member, session_user, etc for ROW and COL...

  • 1107 Views
  • 3 replies
  • 0 kudos
Latest Reply
data-grassroots
New Contributor III
  • 0 kudos

Now that I'm looking closer at the share credentials and the recipient entity you would really need a way to know the bearer token and relate that back to various recipient properties - databricks.name and any custom recipient property tags you may h...

  • 0 kudos
2 More Replies
Sudheer2
by New Contributor II
  • 862 Views
  • 2 replies
  • 2 kudos

Updating Databricks SQL Warehouse using Terraform

 We can Update SQL Warehouse manually in Databricks.Click SQL Warehouses in the sidebarIn Advanced optionsWe can find Unity Catalog toggle button there! While Updating Existing SQL Warehouse in Azure to enable unity catalog using terraform, I couldn'...

  • 862 Views
  • 2 replies
  • 2 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 2 kudos

Hi @Sudheer2, The Unity Catalog is a feature in Databricks SQL Warehouse that allows you to query data across multiple databases and tables seamlessly. It provides a unified view of your data.When you enable the Unity Catalog, you can access tables f...

  • 2 kudos
1 More Replies
MohsenJ
by Contributor
  • 2452 Views
  • 7 replies
  • 1 kudos

log signature and input data for Spark LinearRegression

I am looking for a way to log my `pyspark.ml.regression.LinearRegression` model with input and signature ata. The usual example that I found around are using sklearn and they can simply do  # Log the model with signature and input example signature =...

Community Discussions
mlflow
model_registray
  • 2452 Views
  • 7 replies
  • 1 kudos
Latest Reply
MohsenJ
Contributor
  • 1 kudos

me neither. But the mlflow documentation suggests the new version of mlflow should be able to handle Array and Objects (dict). maybe that could help? I haven't tired it myself.  Support for Array and Object types was introduced in MLflow version 2.10...

  • 1 kudos
6 More Replies
Rpswadhwa
by New Contributor
  • 1225 Views
  • 1 replies
  • 0 kudos

Data analytics

Really excited that Data bricks is championing the cause of unifying the whole data analytics value chain. it’s a great service to the data community.

  • 1225 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Rpswadhwa, We're thrilled to hear that you had a great experience at DAIS 2023! Your feedback is valuable to us, and we appreciate you taking the time to share it on the community platform.   We wanted to let you know that the Databricks Communit...

  • 0 kudos
BDietrich
by New Contributor
  • 541 Views
  • 1 replies
  • 0 kudos

2023 Data AI summit

Have learned a lot through this experience, especially around productionalizing Gen AI models. Great time and great people!

  • 541 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @BDietrich, We're thrilled to hear that you had a great experience at DAIS 2023! Your feedback is valuable to us, and we appreciate you taking the time to share it on the community platform.   We wanted to let you know that the Databricks Communit...

  • 0 kudos
yhyhy3
by New Contributor III
  • 552 Views
  • 1 replies
  • 1 kudos

Databricks AI Summit 2023

Amazing talks from industry leaders and open source contributors! I'm astounded by the keynote presentations on Lakehouse IQ and Spark AI, really driving toward the new future of developers' and analysts' interaction with data. English as the new pro...

  • 552 Views
  • 1 replies
  • 1 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @yhyhy3, We're thrilled to hear that you had a great experience at DAIS 2023! Your feedback is valuable to us, and we appreciate you taking the time to share it on the community platform. We wanted to let you know that the Databricks Community Tea...

  • 1 kudos
maxnumbers
by New Contributor III
  • 569 Views
  • 1 replies
  • 0 kudos

Summit has been a great combo of fun & informative!

Here from the Collins Aerospace group, and we've been enjoying the conference so far! We've learned a lot of new stuff to take back with us in our data transformation journey! The new features for being able to run a job within a job looks pretty exc...

  • 569 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @maxnumbers, We're thrilled to hear that you had a great experience at DAIS 2023! Your feedback is valuable to us, and we appreciate you taking the time to share it on the community platform. We wanted to let you know that the Databricks Community...

  • 0 kudos
Chaho
by New Contributor
  • 395 Views
  • 1 replies
  • 0 kudos

See the future of data integrated with AI

Even if my company does not use Databricks yet, I could see the future of data with AI here at the summit.  

  • 395 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Chaho, We're thrilled to hear that you had a great experience at DAIS 2023! Your feedback is valuable to us, and we appreciate you taking the time to share it on the community platform. We wanted to let you know that the Databricks Community Team...

  • 0 kudos
rli
by New Contributor
  • 453 Views
  • 1 replies
  • 0 kudos

delta live table merging

here at data AI summit day 3. very excited to hear about the delta live tables and it's functionality of applying changes, which actually provides a direct solution to the dilemma at our company when uploading data!

  • 453 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @rli , We're thrilled to hear that you had a great experience at DAIS 2023! Your feedback is valuable to us, and we appreciate you taking the time to share it on the community platform. We wanted to let you know that the Databricks Community Team ...

  • 0 kudos
subham0611
by New Contributor II
  • 544 Views
  • 1 replies
  • 0 kudos

How does coalesce works internally

Hi Databricks team,I am trying to understand internals of spark coalesce code(DefaultPartitionCoalescer) and going through spark code for this. While I understood coalesce function but I am not sure about complete flow of code like where its get call...

  • 544 Views
  • 1 replies
  • 0 kudos
Latest Reply
raphaelblg
Honored Contributor
  • 0 kudos

  Hello @subham0611 , The coalesce operation triggered from user code can be initiated from either an RDD or a Dataset, with each having distinct codepaths: RDD: https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/RDD...

  • 0 kudos
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!