cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

kashy
by New Contributor III
  • 4520 Views
  • 5 replies
  • 1 kudos

Access RDS Postgres DB via SSH Tunnel

Hello, How can I configure a foreign catalog connection to use SSH tunneling? I want to be able to use unity catalog

  • 4520 Views
  • 5 replies
  • 1 kudos
Latest Reply
Debayan
Databricks Employee
  • 1 kudos

Hi, in addition to our previous message, you can try https://docs.databricks.com/en/query-federation/foreign-catalogs.html and https://grant-6562.medium.com/connecting-to-sql-server-through-an-ssh-tunnel-with-python-17de859caca5. Also please tag @Deb...

  • 1 kudos
4 More Replies
dbx_687_3__1b3Q
by New Contributor III
  • 6981 Views
  • 1 replies
  • 2 kudos

Using Azure Event Grid for structured streaming

Can anyone point me to any Databricks documentation (or other resources) for configuring structured streaming to use Azure Event Grid for a source/sink? I found examples for Kafka and EventHubs but Azure Event Grid is different than Azure Event Hubs....

  • 6981 Views
  • 1 replies
  • 2 kudos
Latest Reply
dbx_687_3__1b3Q
New Contributor III
  • 2 kudos

I must be missing something. I don't see how the examples in the referenced document can be applied to Azure Event Grid.Is there another example that shows how to subscribe to an Event Grid topic?

  • 2 kudos
Kristjan
by New Contributor II
  • 2608 Views
  • 1 replies
  • 0 kudos

Unity Catalog - Invalid configuration value detected for fs.azure.account.key

Hi there,I am having issue with writing a df to a table or display it. I have three dataframes that I have unioned and after I have done the union I cannot display the dataframe.df_table1 = spark.sql(f'SELECT * FROM {sql_full_name}')df_table2 = ...df...

  • 2608 Views
  • 1 replies
  • 0 kudos
190857
by New Contributor II
  • 1448 Views
  • 1 replies
  • 0 kudos

sql end point and jdbc driver

When we try and connect to a sql warehouse endpoint with the databricks jdbc driver our query is failing if we use first_value().  We've rewritten the query to use limit 1, but we would like to understand if this is a gap in the simba/databricks driv...

  • 1448 Views
  • 1 replies
  • 0 kudos
Latest Reply
190857
New Contributor II
  • 0 kudos

A sample error message when using first_value() is:An error occurred while calling o132.csv. [Databricks][JDBC](10140) Error converting value to BigDecimal.

  • 0 kudos
sirishavemula20
by New Contributor III
  • 1599 Views
  • 1 replies
  • 0 kudos

Databricks certified Data engineer Associate exam got suspended_need immediate help(10/09/2023)

Hi team,I've scheduled my exam on 10th September,2023 at 15:15hrs Asia Calcutta timing but my exam got suspended stating no proper environment by the proctor.Please check this issue as I didn't attempt one single question too, this is not fair as I l...

  • 1599 Views
  • 1 replies
  • 0 kudos
Latest Reply
APadmanabhan
Databricks Employee
  • 0 kudos

Hi @sirishavemula20 We have addressed the case. If the issue is not resolved, please reply back to the case notes.

  • 0 kudos
AnirbanDas
by New Contributor III
  • 1313 Views
  • 0 replies
  • 0 kudos

Use a single cluster policy for multiple teams and capture team name via custom tags

I am planning to add team names in custom tags and was hoping can do it with allowList and then have the user choose from the list. I am trying to avoid having multiple policy files per team.Has anybody found a good way to do this? May be using globa...

Get Started Discussions
cluster plicy
cluster policy
custom tags
  • 1313 Views
  • 0 replies
  • 0 kudos
andresalvati
by New Contributor II
  • 2213 Views
  • 2 replies
  • 0 kudos

Cannot see 'Databricks Lakehouse Platform" on prod AWS Cost Explorer

Hi, Currently, we have two different AWS accounts: dev and prod. We also have two different workspaces: one for dev and another for prod. The strange thing is that prod costs are being added to the dev account on AWS Cost Explorer ("Databricks Lakeho...

  • 2213 Views
  • 2 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Databricks uses tags and AWS CloudTrail logs to connect and report costs to AWS.Tags can be used to monitor costs and attribute Databricks usage to different business units and teams.AWS CloudTrail logs can be used to calculate the exact cost of API ...

  • 0 kudos
1 More Replies
Skv
by New Contributor II
  • 2360 Views
  • 1 replies
  • 0 kudos

Snowflake filter Query giving empty results in Databricks, same query working in Snowflake.

I am trying to fetch filter data based on date format on a date column. Below is the query forming in Databricks. In Databricks giving empty results.SELECT * FROM Test.TestSchema.Address where TO_DATE(TO_VARCHAR(MODIFIEDDATE,'YYYY-MM-DD')) = '2023-09...

  • 2360 Views
  • 1 replies
  • 0 kudos
Paulv14
by New Contributor
  • 1094 Views
  • 1 replies
  • 0 kudos

Sql warehouse setting permission

How do you set permission to just execute a query but cannot modify settings on the sql warehouse

  • 1094 Views
  • 1 replies
  • 0 kudos
Latest Reply
Debayan
Databricks Employee
  • 0 kudos

Hi, Have you tried this: https://docs.databricks.com/en/security/auth-authz/access-control/sql-endpoint-acl.html Please let us know if this helps.  Thanks!

  • 0 kudos
botn
by New Contributor
  • 2153 Views
  • 1 replies
  • 0 kudos

Resolved! Structured streaming from Azure Event Hub, authenticating without SAS keys

Using SAS keys is a security issue that we would like to avoid, how do we utilize structured streaming from Event Hub while authenticating to Azure AD (client_id and secret).We know that we can use pythons Event Hub library, but that will make have t...

  • 2153 Views
  • 1 replies
  • 0 kudos
Latest Reply
Debayan
Databricks Employee
  • 0 kudos

Hi, Could you please try structured streaming event hubs integration? https://docs.databricks.com/en/_extras/notebooks/source/structured-streaming-event-hubs-integration.html

  • 0 kudos
venkatddatabrik
by New Contributor
  • 1468 Views
  • 1 replies
  • 1 kudos

How to connect to database in azure databricks from vs2022 console application to access the data

Hi Team,Am new to databricks and please find my below question and do the favor.I have created a cluster then database, tables and data also inserted, Now I want to access this table data from my dotnet console application which is in 6.0 framework(v...

  • 1468 Views
  • 1 replies
  • 1 kudos
Latest Reply
Debayan
Databricks Employee
  • 1 kudos

Hi, Databricks extension will help you to do so. https://learn.microsoft.com/en-us/azure/databricks/dev-tools/vscode-ext Please let us know if this helps. Thanks!

  • 1 kudos
AniP
by New Contributor II
  • 4397 Views
  • 3 replies
  • 3 kudos

I created the workspace successfully using QuickStart(recommended) method, now I want to create one more workspace and it is showing an error.

Hi,I created the workspace using QuickStart(recommended) method and at the time of creating workspace, it asked following parameters - AccountId   -AWSRegionBucketNameDataS3BucketIAMRole.PasswordUsernameWorkspaceNameThe workspace was created successf...

  • 4397 Views
  • 3 replies
  • 3 kudos
Latest Reply
Debayan
Databricks Employee
  • 3 kudos

Hi, could you please elaborate on the error code here? There is some misconfiguration which is causing the error. Thanks!

  • 3 kudos
2 More Replies
none
by New Contributor
  • 2766 Views
  • 2 replies
  • 0 kudos

What's going wrong in my attempt to start with DataBricks?

I'm trying to get going with DataBricks for the first time. It's told me to create a workspace, which takes me to AWS (I'm also new to AWS). Following the instructions through there gets it to start creating something, but then it just gets stuck on ...

  • 2766 Views
  • 2 replies
  • 0 kudos
Latest Reply
Debayan
Databricks Employee
  • 0 kudos

Hi, this looks like a workspace creation failure. Would like to know about the error details more. Thanks!

  • 0 kudos
1 More Replies
drii_cavalcanti
by New Contributor III
  • 984 Views
  • 0 replies
  • 0 kudos

Hive Metastore permission on DBX 10.4

I've been working on creating a schema in the Hive Metastore using the following command:spark.sql(f'CREATE DATABASE IF NOT EXISTS {database}')The schema or database is successfully created, but I encountered an issue where it's only accessible for m...

Get Started Discussions
clusters
hive_metastore
legacy
permission
  • 984 Views
  • 0 replies
  • 0 kudos
andresalvati
by New Contributor II
  • 1909 Views
  • 0 replies
  • 0 kudos

Tags to run S3 lifecycle rules

Hello,Is it possible to utilize S3 tags when writing a DataFrame with PySpark? Or is the only option to write the dataframe and then use boto3 to tag all the files?More information about S3 object tagging is here: Amazon S3 Object Tagging.Thank you.

  • 1909 Views
  • 0 replies
  • 0 kudos

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels