cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Yash_542965
by New Contributor II
  • 1237 Views
  • 1 replies
  • 0 kudos

DLT aggregation problem

I'm utilizing SQL to perform aggregation operations within a gold layer of a DLT pipeline. However, I'm encountering an error when running the pipeline while attempting to return a data frame using spark.sql.Could anyone please assist me with the SQL...

  • 1237 Views
  • 1 replies
  • 0 kudos
Latest Reply
lucasrocha
Contributor II
  • 0 kudos

Hello @Yash_542965 , I hope this message finds you well. Could you please share a sample of code you are using so that we can check it further? Best regards,Lucas Rocha

  • 0 kudos
vijaykumarbotla
by New Contributor III
  • 1185 Views
  • 1 replies
  • 0 kudos

Databricks Notebook error : Analysis Exception with multiple datasets

Hi All,I am getting below error when trying to execute the code.AnalysisException: Column Is There a PO#17748 are ambiguous. It's probably because you joined several Datasets together, and some of these Datasets are the same. This column points to ...

  • 1185 Views
  • 1 replies
  • 0 kudos
Latest Reply
lucasrocha
Contributor II
  • 0 kudos

Hello @vijaykumarbotla , I hope you're doing well. This is probably because both DataFrames contain a column with the same name, and Spark is unable to determine which one you are referring to in the select statement. To resolve this issue, you can u...

  • 0 kudos
User16752244127
by Contributor
  • 793 Views
  • 1 replies
  • 0 kudos
  • 793 Views
  • 1 replies
  • 0 kudos
Latest Reply
lucasrocha
Contributor II
  • 0 kudos

Hello @User16752244127 , I hope this message finds you well. Delta Live Tables supports loading data from any data source supported by Databricks. You can find the datasources supported here Connect to data sources, and JDBC is one of them. You can a...

  • 0 kudos
Sambit_S
by New Contributor III
  • 431 Views
  • 1 replies
  • 0 kudos

Exceptions are Not Getting Handled In Autoloader Write Stream

I have below logic implemented using Databricks Autoloader. ## Autoloader Write stream: Its calling forEachBatch function to write into respective datatype catalog table#  and using checkpoint to keeps track of processing files.try:    ##Observe raw ...

Sambit_S_0-1717689309381.png
  • 431 Views
  • 1 replies
  • 0 kudos
Latest Reply
raphaelblg
Honored Contributor II
  • 0 kudos

Hello @Sambit_S ,In your scenario, there is a merge failure. Your query won't be able to progress as the problematic batch can't be committed to sink. Even if you handle the exception in a try catch block, it's impossible for the autoloader to update...

  • 0 kudos
tgen
by New Contributor II
  • 984 Views
  • 2 replies
  • 0 kudos

Increase stack size Databricks

Hi everyoneI'm currently running a shell script in a notebook, and I'm encountering a segmentation fault. This is due to the stack size limitation. I'd like to increase the stack size using ulimit -s unlimited, but I'm facing issues with setting this...

  • 984 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @tgen, To increase the stack size for your shell script in Databricks Notebooks, follow these steps: Spark Configuration Property: With Databricks Runtime 12.2 LTS and above, you can increase the stack size by setting the Spark configuration p...

  • 0 kudos
1 More Replies
satishnavik
by New Contributor II
  • 7224 Views
  • 6 replies
  • 0 kudos

How to connect Databricks Database with Springboot application using JPA

facing issue with integrating our Spring boot JPA supported application with Databricks.Below are the steps and setting we did for the integration.When we are starting the spring boot application we are getting a warning as :HikariPool-1 - Driver doe...

  • 7224 Views
  • 6 replies
  • 0 kudos
Latest Reply
172036
New Contributor II
  • 0 kudos

Was there any resolution to this?  Is Spring datasource supported now?

  • 0 kudos
5 More Replies
AmitAharon
by New Contributor
  • 624 Views
  • 1 replies
  • 0 kudos

running git clone from databricks notebook

Hey,We have a use-case where we want to clone a git repository in Azure DevOps to a storage container (Blob storage).When I'm trying to run the "git clone" command to local storage I keep getting `Operation not supported` error.Git is installed and I...

  • 624 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @AmitAharon, Ensure that you have the necessary permissions to clone the repository. You mentioned having permissions to create files in your workspace, but also verify that you have access to the specific Azure DevOps repository you’re trying to ...

  • 0 kudos
JameDavi_51481
by New Contributor III
  • 4657 Views
  • 9 replies
  • 0 kudos

Can we add tags to Unity Catalog through Terraform?

We use Terraform to manage most of our infrastructure, and I would like to extend this to Unity Catalog. However, we are extensive users of tagging to categorize our datasets, and the only programmatic method I can find for adding tags is to use SQL ...

  • 4657 Views
  • 9 replies
  • 0 kudos
Latest Reply
dbruehlmeier
Contributor
  • 0 kudos

Having tags with terraform would help a lot. Add them on cluster and schema level is crucial. Looking forward for an update about the open PR.

  • 0 kudos
8 More Replies
djburnham
by New Contributor III
  • 1232 Views
  • 2 replies
  • 1 kudos

Resolved! How to get a list of workspace users who have the "unrestricted cluster create" entitlement ?

Hello - I'm hoping somebody can help me with this ... I have a lot of users configured with access to a workspace (100's) and I want to write a report to see if any of the users have  "unrestricted cluster create" entitlement in the workspace. This i...

  • 1232 Views
  • 2 replies
  • 1 kudos
Latest Reply
djburnham
New Contributor III
  • 1 kudos

Many thanks for you help @Yeshwanth  it put me on the right track. The API does have a filter option and that looks like it complies with rfc7644 but my attempts to use it were rather hit and miss - I suspect as the API is preview it is not fully imp...

  • 1 kudos
1 More Replies
abaet
by New Contributor
  • 385 Views
  • 1 replies
  • 0 kudos

Random NoClassDefFound Error when running job

We are running a job on a cluster with DBR 10.4 LTS Spark 3.2.1 and scala 2.12. Cluster is using 4 workers (spot instances) .The driver is not a spot instanceRandomly ( only on one environment and not all executions), we are getting the following err...

  • 385 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @abaet,  The error message suggests that the class failed to initialize. Check if there’s any uncaught exception during static initialization (e.g., in a static block or when creating static variables). If an exception occurs during class loading,...

  • 0 kudos
Anonymous
by Not applicable
  • 6015 Views
  • 11 replies
  • 2 kudos

Sql Serverless Option is missing when using Azure Databricks Workspace with No Public IP and VNET Injection

HelloAfter creating an Databricks Workspace in Azure with No Public IP and VNET Injection, I'm unable to use DBSQL Serverless because the option to enable it in SQL warehouse Settings is missing. ​Is it by design? Is it a limitation when using Privat...

  • 6015 Views
  • 11 replies
  • 2 kudos
Latest Reply
RomanLegion
New Contributor III
  • 2 kudos

Fixed, go to Profile -> Compute->  SQL Server Serverless -> On -> Save. For some reason this has been disabled for us.

  • 2 kudos
10 More Replies
jenshumrich
by Contributor
  • 1663 Views
  • 2 replies
  • 0 kudos

Resolved! R install - cannot open URL

Neither standard nor non standard repo seem available. Any idea how to debug/fix this? %r install.packages("gghighlight", lib="/databricks/spark/R/lib", repos = "http://cran.us.r-project.org") Warning: unable to access index for repository http://cra...

  • 1663 Views
  • 2 replies
  • 0 kudos
Latest Reply
jenshumrich
Contributor
  • 0 kudos

%sh nc -zv cran.us.r-project.org 80 It was a network issue. These lines above proved it and the network administrators had to open the IPs.

  • 0 kudos
1 More Replies
BobBubble2000
by New Contributor II
  • 3198 Views
  • 4 replies
  • 0 kudos

Delta Live Tables with Common Data Model as source

Hi,I'm investigating whether it's possible to use Common Data Model CDM (in particular the Dynamics 365 exported csv and cdm files) as a Delta Live Tables data source? Can someone point me in the right direction?Thanks!

  • 3198 Views
  • 4 replies
  • 0 kudos
Latest Reply
Suryanarayan
New Contributor II
  • 0 kudos

Using Delta Live Tables with Common Data Model (CDM) as a Source in DatabricksI'm investigating the use of Delta Live Tables (DLT) to process Common Data Model (CDM) files exported from Dynamics 365, and I found a solution that works well. Here’s a q...

  • 0 kudos
3 More Replies
Jackson1111
by New Contributor III
  • 522 Views
  • 3 replies
  • 1 kudos

get job detail API

Hello, is there an API interface for passing in batches of run_id to obtain job running details?

  • 522 Views
  • 3 replies
  • 1 kudos
Latest Reply
mhiltner
Contributor III
  • 1 kudos

Maybe this could help. Its not batch, but you can get the run_id details  https://docs.databricks.com/en/workflows/jobs/jobs-2.0-api.html#runs-get-output

  • 1 kudos
2 More Replies
eva_mcmf
by New Contributor II
  • 806 Views
  • 1 replies
  • 0 kudos

Autoloader with SQLite db files

Hi Everyone, Is it possible to ingest SQLite db files with Databricks Autoloader? Is it currently supported? If so, could you please share an example?

Data Engineering
autoloader
azure
ingestion
sqlite
  • 806 Views
  • 1 replies
  • 0 kudos
Latest Reply
lucasrocha
Contributor II
  • 0 kudos

Hello @eva_mcmf , I hope this message finds you well. As per the documentation, Auto Loader incrementally and efficiently processes new data files as they arrive in cloud storage. Auto Loader can load data files from AWS S3, Azure Data Lake Storage G...

  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels