cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

frank7
by New Contributor II
  • 1716 Views
  • 2 replies
  • 1 kudos

Resolved! Is it possible to write a pyspark dataframe to a custom log table in Log Analytics workspace?

I have a pyspark dataframe that contains information about the tables that I have on sql database (creation date, number of rows, etc)Sample data: { "Day":"2023-04-28", "Environment":"dev", "DatabaseName":"default", "TableName":"discount"...

  • 1716 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

@Bruno Simoes​ :Yes, it is possible to write a PySpark DataFrame to a custom log table in Log Analytics workspace using the Azure Log Analytics Workspace API.Here's a high-level overview of the steps you can follow:Create an Azure Log Analytics Works...

  • 1 kudos
1 More Replies
akanksha_gupta
by New Contributor II
  • 1227 Views
  • 2 replies
  • 0 kudos

ERROR : Failure starting repl. Try detaching and re-attaching the notebook. Getting this error when running any python command in 10.4LTS cluster configured with https://github.com/mspnp/spark-monitoring to send databricks spark logs to Log Analytics.

ERROR Description :java.lang.Exception: Cannot run program "/local_disk0/pythonVirtualEnvDirs/virtualEnv-5acc1ea9-d03f-4de3-b76b-203d42614000/bin/python" (in directory "."): error=2, No such file or directory at java.lang.ProcessB...

  • 1227 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

@Akanksha Gupta​ :The error message suggests that the Python executable file specified in the configuration of the Databricks cluster cannot be found or accessed. Specifically, it seems that the Python executable file at the path "/local_disk0/python...

  • 0 kudos
1 More Replies
SailajaB
by Valued Contributor III
  • 10503 Views
  • 5 replies
  • 7 kudos

Resolved! Best mechanism to logging the notebook run/metadata and error details

Hi,How we can integrate log analytics with databricks to log notebook run details and code validations.Thank you

  • 10503 Views
  • 5 replies
  • 7 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 7 kudos

I think you are looking for sending application logs.I'd use Log4j as this is already used by Databricks.The link does not use notebooks but it should work in notebooks too.

  • 7 kudos
4 More Replies
Labels