cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Frustrated_DE
by New Contributor III
  • 2305 Views
  • 1 replies
  • 0 kudos

DLT SQL demo pipeline issue

Hi,   First foray into DLT and following code exerts from the sample-DLT-notebook.I'm creating a notebook with the SQL below:CREATE STREAMING LIVE TABLE sales_orders_rawCOMMENT "The raw sales orders, ingested from /databricks-datasets."TBLPROPERTIES ...

  • 2305 Views
  • 1 replies
  • 0 kudos
Latest Reply
Frustrated_DE
New Contributor III
  • 0 kudos

If you change the notebook default language as opposed to using magic command. I normally have it set to Python, I've wrongly assumed DLT would transpose as can't use magic command but have to change default in order for it to work. 

  • 0 kudos
vieiradsousa
by New Contributor II
  • 2135 Views
  • 1 replies
  • 0 kudos

Validating Dlt Pipeline

Whenever I try validating a pipeline that already runs productively without any issue, it throws me the following error:BAD_REQUEST: Failed to load notebook '/Repos/(...).sql'. Only SQL and Python notebooks are supported currently.

  • 2135 Views
  • 1 replies
  • 0 kudos
HeijerM84
by New Contributor III
  • 7754 Views
  • 5 replies
  • 3 kudos

Resolved! Databricks runtime 14.3 gives error scala.math.BigInt cannot be cast to java.lang.Integer

We have a cluster running on 13.3 LTS (includes Apache Spark 3.4.1, Scala 2.12).We want to test with a different type of cluster (14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12))And all of a sudden we get errors that complain about a casting a Big...

  • 7754 Views
  • 5 replies
  • 3 kudos
Latest Reply
jeroenvs
New Contributor III
  • 3 kudos

I have logged the issue with Microsoft last week and they confirmed it is a Databricks bug. A fix is supposedly being rolled out at the moment across Databricks regions. As anticipated, we have engaged the Databricks core team to further investigate ...

  • 3 kudos
4 More Replies
tajinder123
by New Contributor II
  • 1670 Views
  • 0 replies
  • 0 kudos

broadcasted table reuse

in spark, table1 is small and broadcasted and joined with table 2. output is stored in df1. again, table1 is required to join with table3 and output need to be stored in df2. do it need to be broadcasted again?

  • 1670 Views
  • 0 replies
  • 0 kudos
chari
by Contributor
  • 1987 Views
  • 0 replies
  • 0 kudos

how to run a group of cells in databricks ?

Hello,I was experimenting with a ML model with different parameters and check the results. However, the important part of this code is contained in a couple of cells (say cell # 12, 13 & 14). I like to proceed to the next cell only when the results a...

  • 1987 Views
  • 0 replies
  • 0 kudos
AravindNani
by New Contributor
  • 3383 Views
  • 1 replies
  • 0 kudos

Unable to read data from API due to Private IP Restriction

I have data in my API Endpoint but am unable to read it using Databricks. My data is limited to my private IP address and can only be accessed over a VPN connection. I can't read data into Databricks as a result of this. I can obtain the data in VS C...

  • 3383 Views
  • 1 replies
  • 0 kudos
Latest Reply
Wojciech_BUK
Valued Contributor III
  • 0 kudos

Hi AravindNaniThis is more of infrastructure questions, you have to make sure that:1) Your databricks Workspace is provisioned in VNET Injection mode2) Your VNET is either peered to "HUB" network where you have S2S VPN Connection to API or you have t...

  • 0 kudos
hpicatto
by New Contributor III
  • 3457 Views
  • 1 replies
  • 0 kudos

using the api for getting cost in usd

I'm trying to use the API of billable usage and I do get a report but I have not been able to get the usd cost report, only the dbuHours. I guess I've to change the meter_name but I cannot find the key for that parameter anywhere

  • 3457 Views
  • 1 replies
  • 0 kudos
arkiboys
by Contributor
  • 4805 Views
  • 1 replies
  • 0 kudos

databricks email notification

In databricks, if a job fails, then an email is sent off as notification.The recipient, receives the email with the link to the databricks workspace.Question:How is it possible the email is sent without any link, just the plain text in the email is w...

  • 4805 Views
  • 1 replies
  • 0 kudos
RozaZaharieva
by New Contributor
  • 2055 Views
  • 0 replies
  • 0 kudos

set up Azure Databricks workspace and Unity catalog - how to automate not using Terraform

Hi everyone, I am looking for a way to automate initial setup of Azure Databricks workspace and Unity Catalog but can't find anything on this topic other than Terraform. Can you share if this is possible with powershell, for example. Thank you un adv...

  • 2055 Views
  • 0 replies
  • 0 kudos
mohaimen_syed
by New Contributor III
  • 9856 Views
  • 3 replies
  • 1 kudos

Fuzzy Match on PySpark using UDF/Pandas UDF

I'm trying to do fuzzy matching on two dataframes by cross joining them and then using a udf for my fuzzy matching. But using both python udf and pandas udf its either very slow or I get an error. @pandas_udf("int")def core_match_processor(s1: pd.Ser...

  • 9856 Views
  • 3 replies
  • 1 kudos
Latest Reply
mohaimen_syed
New Contributor III
  • 1 kudos

I'm now getting the error: (SQL_GROUPED_AGG_PANDAS_UDF) is not supported on clusters in Shared access mode.Even though this article clearly states that pandas udf is supported for shared cluster in databrickshttps://www.databricks.com/blog/shared-clu...

  • 1 kudos
2 More Replies
ntvdatabricks
by New Contributor II
  • 6455 Views
  • 2 replies
  • 1 kudos

Resolved! Okta and Unified login

Hey Folks anyone put Databricks behind Okta and enabled Unified Login with workspaces that have a Unity Catalog metastore applied and some that don't?There are some workspaces we can't move over yet and it isn't clear in documentation if Unity Catalo...

  • 6455 Views
  • 2 replies
  • 1 kudos
Latest Reply
Walter_C
Databricks Employee
  • 1 kudos

Yes, users should be able to use a single Okta application for all workspaces, regardless of whether the Unity Catalog metastore has been applied or not. The Unity Catalog is a feature that allows you to manage and secure access to your data across a...

  • 1 kudos
1 More Replies

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels