cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Oliver_Angelil
by Valued Contributor II
  • 1369 Views
  • 2 replies
  • 2 kudos

Resolved! Cell by cell execution of notebooks with VS code

I have the Databricks VS code extension setup to develop and run jobs remotely. (with Databricks Connect).I enjoy working on notebooks within the native Databricks workspace, especially for exploratory work because I can execute blocks of code step b...

  • 1369 Views
  • 2 replies
  • 2 kudos
Latest Reply
awadhesh14
New Contributor II
  • 2 kudos

Hi Folks,Is there a version upgrade for the resolution to this?

  • 2 kudos
1 More Replies
Ajay-Pandey
by Esteemed Contributor III
  • 2602 Views
  • 5 replies
  • 0 kudos

How we can send databricks log to Azure Application Insight ?

Hi All,I want to send databricks logs to azure application insight.Is there any way we can do it ??Any blog or doc will help me.

  • 2602 Views
  • 5 replies
  • 0 kudos
Latest Reply
floringrigoriu
New Contributor II
  • 0 kudos

hi @Debayan in the  https://learn.microsoft.com/en-us/azure/architecture/databricks-monitoring/application-logs. there is a github repository mentioned https://github.com/mspnp/spark-monitoring ? That repository is marked as  maintainance mode.  Just...

  • 0 kudos
4 More Replies
youssefmrini
by Honored Contributor III
  • 636 Views
  • 1 replies
  • 0 kudos
  • 636 Views
  • 1 replies
  • 0 kudos
Latest Reply
youssefmrini
Honored Contributor III
  • 0 kudos

Make sure to watch the following video https://www.youtube.com/watch?v=DkzwFTC7WWsThis section lists the requirements for Databricks Connect.Only Databricks Runtime 13.0 ML and Databricks Runtime 13.0 are supported.Only clusters that are compatible w...

  • 0 kudos
apayne
by New Contributor III
  • 427 Views
  • 0 replies
  • 0 kudos

Python Databricks SQL Connector vs Databricks Connect?

Connecting several Databricks tables to a Plotly Dash application. Can't seem to find much documentation on the differences between SQL Connector and Connect. Currently using the Connect approach to read tables into pyspark dataframes, is one better ...

  • 427 Views
  • 0 replies
  • 0 kudos
lsoewito
by New Contributor
  • 3287 Views
  • 3 replies
  • 3 kudos

Resolved! How to configure Databricks Connect to 'Assume Role' when accessing file from an AWS S3 bucket?

I have a Databricks cluster configured with an instance profile to assume role when accessing an AWS S3 bucket.Accessing the bucket from the notebook using the cluster works properly (the instance profile can assume role to access the bucket).However...

  • 3287 Views
  • 3 replies
  • 3 kudos
Latest Reply
Kaniz
Community Manager
  • 3 kudos

Hi @lsoewito​ , We haven't heard from you since my last response, and I was checking back to see if my suggestions helped you. Or else, If you have any solution, please share it with the community as it can be helpful to others. Also, please don't fo...

  • 3 kudos
2 More Replies
brendan-b
by New Contributor II
  • 6349 Views
  • 4 replies
  • 3 kudos

spark-xml not working with Databricks Connect and Pyspark

Hi all,I currently have a cluster configured in databricks with spark-xml (version com.databricks:spark-xml_2.12:0.13.0) which was installed using Maven. The spark-xml library itself works fine with Pyspark when I am using it in a notebook within th...

  • 6349 Views
  • 4 replies
  • 3 kudos
Latest Reply
Kaniz
Community Manager
  • 3 kudos

Hi @Brendan Banfield​ , This article describes how to read and write an XML file as an Apache Spark™ data source.

  • 3 kudos
3 More Replies
ayzm
by New Contributor
  • 855 Views
  • 0 replies
  • 0 kudos

[Databricks Connect] Cannot cross line reference when using lambda expression through db-connect

Hirun below code line at spark-shell through db-connectIt throw exception:java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f of type scala.Function3 in instance o...

image.png image
  • 855 Views
  • 0 replies
  • 0 kudos
YSF
by New Contributor III
  • 1213 Views
  • 3 replies
  • 1 kudos

Resolved! Issues with using Databricks-Connect and Petastorm

Has anyone successfully used Petastorm + Databricks-Connect + Delta Lake?The use case is being able to use DeltaLake as a data store regardless of whether I want to use the databricks workspace or not for my training tasks.I'm using a cloud-hosted ju...

  • 1213 Views
  • 3 replies
  • 1 kudos
Latest Reply
Kaniz
Community Manager
  • 1 kudos

Hi @Yusuf Khan​ , Please go through the link here.

  • 1 kudos
2 More Replies
User16790091296
by Contributor II
  • 810 Views
  • 1 replies
  • 1 kudos

Using Databricks Connect (DBConnect)

I'd like to edit Databricks notebooks locally using my favorite editor, and then use Databricks Connect to run the notebook remotely on a Databricks cluster that I usually access via the web interface.I run "databricks-connect configure" , as suggest...

  • 810 Views
  • 1 replies
  • 1 kudos
Latest Reply
sajith_appukutt
Honored Contributor II
  • 1 kudos

Here is the link to the configuration properties https://docs.databricks.com/dev-tools/databricks-connect.html#step-2-configure-connection-properties

  • 1 kudos
Labels