cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
sriramkumar
by • New Contributor II
  • 3040 Views
  • 3 replies
  • 0 kudos

New Databricks Driver gives SQLNonTransientConnectionException when trying to connect to Databricks Instance

import com.databricks.client.jdbc.DataSource;   import java.sql.*;   public class testDatabricks { public static void main(String[] args) throws SQLException { String dbUrl = "jdbc:databricks://<hostname>:443;HttpPath=<HttpPath>;"; // Cop...

Data Engineering
Reply
  • 3040 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Atanu
Databricks Employee
  • 0 kudos

This looks like due to maintenance on US . Are you still facing the issue @Sriramkumar Thamizharasan​ Is your workspace on eastus and eastus2 ?

  • 0 kudos
2 More Replies
repcak
by • New Contributor III
  • 5185 Views
  • 4 replies
  • 3 kudos

Resolved! Delta Live Tables with EventHub

Hello,I would like to integrate Databricks Delta Live Tables with Eventhub, but i cannot install com.microsoft.azure:azure-eventhubs-spark_2.12:2.3.17 on delta live cluster.I tried installed in using Init script (by adding it in Json cluster settings...

image
Data Engineering
Reply
  • 5185 Views
  • 4 replies
  • 3 kudos
View Replies
Latest Reply
Atanu
Databricks Employee
  • 3 kudos

I think this has some details https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-kafka-spark-tutorial @Kacper Mucha​ is the issue resolved ?

  • 3 kudos
3 More Replies
Soma
by • Valued Contributor
  • 3227 Views
  • 5 replies
  • 6 kudos

Resolved! Start Up Notebook in Databricks

Hi Team,Is it possible to have a startup notebook in databricks similar to init sql script ( This need to run on start of every cluster )

Data Engineering
Reply
  • 3227 Views
  • 5 replies
  • 6 kudos
View Replies
Latest Reply
Soma
Valued Contributor
  • 6 kudos

@Kaniz Fatma​ sure will do

  • 6 kudos
4 More Replies
Soma
by • Valued Contributor
  • 3520 Views
  • 0 replies
  • 0 kudos

Spark Failure Error Unable to download spark docker Image

Cluster terminated. Reason: Spark Image Download Failure  "reason": { "code": "SPARK_IMAGE_DOWNLOAD_FAILURE", "type": "SERVICE_FAULT", "parameters": { "instance_id": "6565aa39b0ae4fe69c7fe6f313e3ca2a", "databricks_error_message": "Failed to set up th...

Data Engineering
Reply
  • 3520 Views
  • 0 replies
  • 0 kudos
venkad
by • Contributor
  • 1644 Views
  • 1 replies
  • 2 kudos

Resolved! Is Databricks Light Runtime Discontinued?

The last Databricks Light runtime release was 2.4 Extended Support. There was no Light version for Spark 3.x. Is Databricks Light runtime discontinued? If not, when we can expect the next DBR Light version?

Data Engineering
Reply
  • 1644 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Venkadeshwaran K​, I looked around, and it does look like there won't be future light runtimes. We can't hire enough engineers to maintain and develop everything, and light is one of the casualties of that.

  • 2 kudos
geertvanhove
by • New Contributor III
  • 8012 Views
  • 5 replies
  • 6 kudos

Resolved! connection to databricks data source in Visual Studio fails with SSL error

Hi,I'm new to databricks but am positively surprised by the product. We use databricks delta tables as source to build a tabular model, which will serve as data source for Power Bi. To develop our tabular model we use Visual studio to import tables ...

Data Engineering
Reply
  • 8012 Views
  • 5 replies
  • 6 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 6 kudos

In Visual Studio do you use databricks-connect? Maybe it is enough to rerun "databricks-connect configure"

  • 6 kudos
4 More Replies
ivanychev
by • Contributor II
  • 1658 Views
  • 3 replies
  • 3 kudos

AWS + Cluster with custom docker image = export metrics to CloudWatch?

Is there a working setup on setting up metrics export to CloudWatch while using custom docker images for cluster creation? I've tried to set up the CloudWatch agent manually, but launching `amzon-cloudwatch-agent-ctl` in the bootstrap script fails wi...

Data Engineering
Reply
  • 1658 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
Atanu
Databricks Employee
  • 3 kudos

We do not support gangila with custom docker too. but let me cross verify if we are supporting cloudwatch for the same. Sorry for the inconvenience @Sergey Ivanychev​ 

  • 3 kudos
2 More Replies
apw
by • New Contributor II
  • 2354 Views
  • 1 replies
  • 2 kudos

Arrow R package fails to install

# Databricks notebook source .libPaths()   # COMMAND ----------   dir("/databricks/spark/R/lib")   # COMMAND ----------   ## Add current working directory to library paths .libPaths(c(getwd(), .libPaths()))   # COMMAND ----------   ## The latest vers...

Arrow Fail Message&quot; data-fileid=&quot;0698Y00000JFZosQAH
Data Engineering
Reply
  • 2354 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
Atanu
Databricks Employee
  • 2 kudos

@Anthony McGrath​ can you please download and upload to DBFS and see if the issue still persists?You can check if any global initscript is reinstalling this to your cluster.

  • 2 kudos
Martin1
by • New Contributor II
  • 2439 Views
  • 2 replies
  • 2 kudos

Notebook metadata

HelloI would like to view metadata about the notebooks in the Workspace folder hierarchy, for example date created, modified, by user, etc.Is this possible?

Data Engineering
Reply
  • 2439 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Atanu
Databricks Employee
  • 2 kudos

@Martin Aronsson​, you need to follow the notebook revision history (https://docs.databricks.com/notebooks/notebooks-use.html#revision-history) I believe. Also you can try https://docs.microsoft.com/en-us/azure/databricks/administration-guide/account...

  • 2 kudos
1 More Replies
User16835756816
by • Valued Contributor
  • 4720 Views
  • 1 replies
  • 8 kudos

Announcing: Workflows!

Databricks is excited to announce the general availability of Databricks Workflows to you, our community. Databricks Workflows is the fully managed lakehouse orchestration service for all your teams to build reliable data, analytics, and AI workflow...

Data Engineering
Reply
  • 4720 Views
  • 1 replies
  • 8 kudos
View Replies
Latest Reply
PawanShukla
New Contributor III
  • 8 kudos

I am trying to run the Workflow Pipeline with smaple code shared in getting start.. and getting the below error :DataPlaneException: Failed to start the DLT service on cluster 0526-084319-7hucy1np. Please check the stack trace below or driver logs fo...

  • 8 kudos
as999
by • New Contributor III
  • 2039 Views
  • 2 replies
  • 3 kudos

Terraform import multiple notebook copy from repo's?

From below article, i am able to copy only single notebook to dbrick workspace and it's not supporting to copy the multiple notebook using asterisks i.e * and also under resource databrick_notebook, for_each statement is not recognizingdatabricks_n...

Data Engineering
Reply
  • 2039 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Atanu
Databricks Employee
  • 3 kudos

Hi @as999​ is there any error you are getting or it's just simply not copying multiple notebook, can you please share your code too so that I can take a look.Thanks.

  • 3 kudos
1 More Replies
Sophia_Ars
by • New Contributor II
  • 1298 Views
  • 1 replies
  • 1 kudos

Abrupt Subscription Cancellation Issues

Hello Community,I've got informed from Help desk to post this issue in community.We've contacted all supportive entities: billing team, help desk and sales team,but the issue hasn't solved yet.My team(Ars Praxia) has issue of sudden cancellation of s...

Data Engineering
Reply
  • 1298 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 1 kudos

This widget could not be displayed.
Hello Community,I've got informed from Help desk to post this issue in community.We've contacted all supportive entities: billing team, help desk and sales team,but the issue hasn't solved yet.My team(Ars Praxia) has issue of sudden cancellation of s...

This widget could not be displayed.
  • 1 kudos
This widget could not be displayed.
CHANDY
by • New Contributor
  • 889 Views
  • 0 replies
  • 0 kudos

real time data processing

Say I am getting a customer record from an website. I want to read the massage & then insert/update that one to snowflake table , depending on the records insert/update is successful I need to respond back the success / failure massage in say 1 sec. ...

Data Engineering
Reply
  • 889 Views
  • 0 replies
  • 0 kudos
Sunny
by • New Contributor III
  • 553 Views
  • 0 replies
  • 1 kudos

Integrate exe into workflow

We need to execute a long running exe running on a windows machine and thinking of ways to integrate with the workflow. The plan is to include the exe as a task in the Databricks workflow.​​We are thinking of couple of approachesCreate a DB table and...

Data Engineering
Reply
  • 553 Views
  • 0 replies
  • 1 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels