cancel
Showing results for 
Search instead for 
Did you mean: 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Shawn_Eary
by Contributor
  • 15725 Views
  • 3 replies
  • 4 kudos

Resolved! Use SQL Server Management Studio to Connect to DataBricks?

The Notebook UI doesn't always provide the best experience for running exploratory SQL queries. Is there a way for me to use SQL Server Management Studio (SSMS) to connect to DataBricks? See Also:https://learn.microsoft.com/en-us/answers/questions/74...

  • 15725 Views
  • 3 replies
  • 4 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 4 kudos

What you can do is define a SQL endpoint as a linked server.  Like that you can use SSMS and T-SQL.However, it has some drawbacks (no/bad query pushdown, no caching).Here is an excellent blog of Kyle Hale of databricks:Tutorial: Create a Databricks S...

  • 4 kudos
2 More Replies
pshuk
by New Contributor III
  • 1718 Views
  • 1 replies
  • 2 kudos

ingest csv file on-prem to delta table on databricks

Hi,So I want to create a delta live table using a csv file that I create locally (on-prem). A little background: So I have a working ELT pipeline that finds newly generated files (since the last upload), and upload them to databricks volume and at th...

  • 1718 Views
  • 1 replies
  • 2 kudos
Latest Reply
raphaelblg
Databricks Employee
  • 2 kudos

Hello @pshuk , Based on your description, you have an external pipeline that writes CSV files to a specific storage location and you wish to set up a DLT based on the output of this pipeline. DLT offers has access to a feature called Autoloader, whic...

  • 2 kudos
Cloud_Architect
by New Contributor III
  • 1758 Views
  • 3 replies
  • 3 kudos

I am facing an issue while generating the DBU consumption report and need help.

I am trying to access the following system tables to generate a DBU consumption report, but I am not seeing this table in the system schema. Could you please help me how to access it?system.billing.inventory, system.billing.workspaces, system.billing...

  • 1758 Views
  • 3 replies
  • 3 kudos
data-grassroots
by New Contributor III
  • 1899 Views
  • 2 replies
  • 0 kudos

Delta Sharing - Info about Share Recipient

What information do you know about a share recipient when they access a table shared to them via Delta Sharing?Wondering if we might be able to utilize something along the lines of is_member, is_account_group_member, session_user, etc for ROW and COL...

  • 1899 Views
  • 2 replies
  • 0 kudos
Latest Reply
data-grassroots
New Contributor III
  • 0 kudos

Now that I'm looking closer at the share credentials and the recipient entity you would really need a way to know the bearer token and relate that back to various recipient properties - databricks.name and any custom recipient property tags you may h...

  • 0 kudos
1 More Replies
MohsenJ
by Contributor
  • 1144 Views
  • 0 replies
  • 1 kudos

how to develop Notebooks on vscode for git repos?

I am able to use vscode extension + databricks connect to develop Notebooks on my local computer and run them on my databricks cluster. However I can not figure out how to develop the Notebooks that have the file `.py` extension but identified by Dat...

  • 1144 Views
  • 0 replies
  • 1 kudos
subham0611
by New Contributor II
  • 1921 Views
  • 1 replies
  • 0 kudos

Resolved! How does coalesce works internally

Hi Databricks team,I am trying to understand internals of spark coalesce code(DefaultPartitionCoalescer) and going through spark code for this. While I understood coalesce function but I am not sure about complete flow of code like where its get call...

  • 1921 Views
  • 1 replies
  • 0 kudos
Latest Reply
raphaelblg
Databricks Employee
  • 0 kudos

  Hello @subham0611 , The coalesce operation triggered from user code can be initiated from either an RDD or a Dataset, with each having distinct codepaths: RDD: https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/RDD...

  • 0 kudos
georgeyjy
by New Contributor II
  • 7107 Views
  • 2 replies
  • 0 kudos

Resolved! Why saving pyspark df always converting string field to number?

  import pandas as pd from pyspark.sql.types import StringType, IntegerType from pyspark.sql.functions import col save_path = os.path.join(base_path, stg_dir, "testCsvEncoding") d = [{"code": "00034321"}, {"code": "55964445226"}] df = pd.Data...

  • 7107 Views
  • 2 replies
  • 0 kudos
Latest Reply
daniel_sahal
Esteemed Contributor
  • 0 kudos

@georgeyjy Try opening the CSV as text editor. I bet that Excel is automatically trying to detect the schema of CSV thus it thinks that it's an integer.

  • 0 kudos
1 More Replies
Madhawa
by New Contributor II
  • 2652 Views
  • 1 replies
  • 0 kudos

Unable to access AWS S3 - Error : java.nio.file.AccessDeniedException

Reading file like this "Data = spark.sql("SELECT * FROM edge.inv.rm") Getting this error org.apache.spark.SparkException: Job aborted due to stage failure: Task 10 in stage 441.0 failed 4 times, most recent failure: Lost task 10.3 in stage 441.0 (TID...

  • 2652 Views
  • 1 replies
  • 0 kudos
NarenderKumar
by New Contributor III
  • 2308 Views
  • 2 replies
  • 0 kudos

Resolved! Unable to generate account level PAT for service principle

I am trying to generate PAT for a service principle.I am following the documentation as shown below:https://docs.databricks.com/en/dev-tools/auth/oauth-m2m.html#create-token-in-accountI have prepared the below curl command:I am getting below error:Pl...

NarenderKumar_0-1715695724302.png NarenderKumar_1-1715695859890.png NarenderKumar_2-1715695895738.png
  • 2308 Views
  • 2 replies
  • 0 kudos
Latest Reply
NarenderKumar
New Contributor III
  • 0 kudos

I was able to generate the workspace level token using the databricks cli.I set the following details in the databricks cli profile(.databrickscfg) file: host  = https://myworksapce.azuredatabricks.net/ account_id = (my db account id)client_id     = ...

  • 0 kudos
1 More Replies
NhanNguyen
by Contributor III
  • 4057 Views
  • 2 replies
  • 1 kudos

[Delta live table vs Workflow]

Hi Community Members,I have been using Databricks for a while, but I have only used Workflow. I have a question about the differences between Delta Live Table and Workflow. Which one should we use in which scenario?Thanks,

  • 4057 Views
  • 2 replies
  • 1 kudos
Latest Reply
Hkesharwani
Contributor II
  • 1 kudos

Hi, Delta Live Tables focuses on managing data ingestion, transformation, and management of Delta tables using a declarative framework. Job Workflows are designed to orchestrate and schedule various data processing and analysis tasks, including SQL q...

  • 1 kudos
1 More Replies
kazinahian
by New Contributor III
  • 3442 Views
  • 1 replies
  • 0 kudos

Enable or disable Databricks Assistant in the Community Edition.

Hello,Good afternoon great people. I was following the step-by-step instructions to enable or disable Databricks Assistant in my Databricks Community Edition to enable the AI assistance. However, I couldn't find the option and was unable to enable it...

  • 3442 Views
  • 1 replies
  • 0 kudos
Latest Reply
kazinahian
New Contributor III
  • 0 kudos

Thank you @Retired_mod 

  • 0 kudos
paritosh_sharma
by New Contributor
  • 4683 Views
  • 0 replies
  • 0 kudos

DAB template dbt-sql not working

Hi,We are trying to use the dbt-sql template provided for databricks asset bundles but getting error as follows: Looks like its regarding default catalog configuration. Has anyone faced this previously or can help with the same  

Screenshot 2024-05-17 at 10.25.38.png
  • 4683 Views
  • 0 replies
  • 0 kudos
NandiniN
by Databricks Employee
  • 3540 Views
  • 1 replies
  • 2 kudos

How to collect a thread dump from Databricks Spark UI.

If you observe a hung job, thread dumps are crucial to determine the root cause. Hence, it would be a good idea to collect the thread dumps before cancelling the hung job. Here are the Instructions to collect the Spark driver/executor thread dump:  ​...

  • 3540 Views
  • 1 replies
  • 2 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 2 kudos

Thank you for sharing @NandiniN

  • 2 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Top Kudoed Authors