cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Oliver_Angelil
by • Valued Contributor II
  • 8121 Views
  • 2 replies
  • 2 kudos

Resolved! Cell by cell execution of notebooks with VS code

I have the Databricks VS code extension setup to develop and run jobs remotely. (with Databricks Connect).I enjoy working on notebooks within the native Databricks workspace, especially for exploratory work because I can execute blocks of code step b...

Data Engineering
Reply
  • 8121 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
awadhesh14
New Contributor II
  • 2 kudos

Hi Folks,Is there a version upgrade for the resolution to this?

  • 2 kudos
1 More Replies
Ajay-Pandey
by • Esteemed Contributor III
  • 5721 Views
  • 5 replies
  • 0 kudos

How we can send databricks log to Azure Application Insight ?

Hi All,I want to send databricks logs to azure application insight.Is there any way we can do it ??Any blog or doc will help me.

Data Engineering
Reply
  • 5721 Views
  • 5 replies
  • 0 kudos
View Replies
Latest Reply
floringrigoriu
New Contributor II
  • 0 kudos

hi @Debayan in the  https://learn.microsoft.com/en-us/azure/architecture/databricks-monitoring/application-logs. there is a github repository mentioned https://github.com/mspnp/spark-monitoring ? That repository is marked as  maintainance mode.  Just...

  • 0 kudos
4 More Replies
youssefmrini
by • Databricks Employee
  • 1519 Views
  • 1 replies
  • 0 kudos

Resolved! How Can I use Databricks Connect v2 ?

Data Engineering
Reply
  • 1519 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
youssefmrini
Databricks Employee
  • 0 kudos

Make sure to watch the following video https://www.youtube.com/watch?v=DkzwFTC7WWsThis section lists the requirements for Databricks Connect.Only Databricks Runtime 13.0 ML and Databricks Runtime 13.0 are supported.Only clusters that are compatible w...

  • 0 kudos
apayne
by • New Contributor III
  • 890 Views
  • 0 replies
  • 0 kudos

Python Databricks SQL Connector vs Databricks Connect?

Connecting several Databricks tables to a Plotly Dash application. Can't seem to find much documentation on the differences between SQL Connector and Connect. Currently using the Connect approach to read tables into pyspark dataframes, is one better ...

Data Engineering
Reply
  • 890 Views
  • 0 replies
  • 0 kudos
ayzm
by • New Contributor
  • 1629 Views
  • 0 replies
  • 0 kudos

[Databricks Connect] Cannot cross line reference when using lambda expression through db-connect

Hirun below code line at spark-shell through db-connectIt throw exception:java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f of type scala.Function3 in instance o...

image.png image
Data Engineering
Reply
  • 1629 Views
  • 0 replies
  • 0 kudos
lsoewito
by • New Contributor
  • 6400 Views
  • 1 replies
  • 1 kudos

How to configure Databricks Connect to 'Assume Role' when accessing file from an AWS S3 bucket?

I have a Databricks cluster configured with an instance profile to assume role when accessing an AWS S3 bucket.Accessing the bucket from the notebook using the cluster works properly (the instance profile can assume role to access the bucket).However...

Data Engineering
Reply
  • 6400 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hello, @lsoewito​ - My name is Piper, and I'm a moderator for the Databricks community. Welcome and thank you for coming to us with your question. I'm sorry to hear that you're having trouble. Let's give your peers a chance to answer your question. W...

  • 1 kudos
YSF
by • New Contributor III
  • 2388 Views
  • 2 replies
  • 1 kudos

Resolved! Issues with using Databricks-Connect and Petastorm

Has anyone successfully used Petastorm + Databricks-Connect + Delta Lake?The use case is being able to use DeltaLake as a data store regardless of whether I want to use the databricks workspace or not for my training tasks.I'm using a cloud-hosted ju...

Data Engineering
Reply
  • 2388 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
YSF
New Contributor III
  • 1 kudos

because its janky or why? I don't need it for customer facing production. More so for if I'm using my own HPC or local workstation, but I want to access data from delta lake. Figured it was easier/preferable to setting up my own spark environment loc...

  • 1 kudos
1 More Replies
brendan-b
by • New Contributor II
  • 10483 Views
  • 2 replies
  • 3 kudos

spark-xml not working with Databricks Connect and Pyspark

Hi all,I currently have a cluster configured in databricks with spark-xml (version com.databricks:spark-xml_2.12:0.13.0) which was installed using Maven. The spark-xml library itself works fine with Pyspark when I am using it in a notebook within th...

Data Engineering
Reply
  • 10483 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
brendan-b
New Contributor II
  • 3 kudos

@Sean Owen​ I do not believe I have. Do you have any documentation on how to install spark-xml locally? I have tried the following with no luck. IS this what you are referring to?PYSPARK_HOME/bin/pyspark --packages com.databricks:spark-xml_2.12:0.13....

  • 3 kudos
1 More Replies
User16790091296
by • Contributor II
  • 1676 Views
  • 1 replies
  • 1 kudos

Using Databricks Connect (DBConnect)

I'd like to edit Databricks notebooks locally using my favorite editor, and then use Databricks Connect to run the notebook remotely on a Databricks cluster that I usually access via the web interface.I run "databricks-connect configure" , as suggest...

Data Engineering
Reply
  • 1676 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
sajith_appukutt
Honored Contributor II
  • 1 kudos

Here is the link to the configuration properties https://docs.databricks.com/dev-tools/databricks-connect.html#step-2-configure-connection-properties

  • 1 kudos
Labels