@Bruno Simoes​ :

Yes, it is possible to write a PySpark DataFrame to a custom log table in Log Analytics workspace using the Azure Log Analytics Workspace API.

Here's a high-level overview of the steps you can follow:

1. Create an Azure Log Analytics Workspace and obtain the Workspace ID and Primary Key.
2. Install the 'azure-loganalytics' library using pip.
3. Create a new instance of the 'LogAnalyticsWorkspace' class from the 'azure.loganalytics' module using the Workspace ID and Primary Key.
4. Convert your PySpark DataFrame to a Pandas DataFrame using the 'toPandas()' method.
5. Convert the Pandas DataFrame to a JSON string using the 'to_json()' method.
6. Use the 'LogAnalyticsWorkspace' instance to send the JSON string to the custom log table using the 'post_data()' method.

Here's some example code:

from azure.loganalytics import LogAnalyticsWorkspace
import pandas as pd
 
# Replace with your Workspace ID and Primary Key
workspace_id = 'YOUR_WORKSPACE_ID'
primary_key = 'YOUR_PRIMARY_KEY'
 
# Create a new instance of the LogAnalyticsWorkspace class
workspace = LogAnalyticsWorkspace(workspace_id, primary_key)
 
# Convert PySpark DataFrame to Pandas DataFrame
pandas_df = spark_df.toPandas()
 
# Convert Pandas DataFrame to JSON string
json_str = pandas_df.to_json(orient='records')
 
# Send JSON string to custom log table in Log Analytics workspace
workspace.post_data('CUSTOM_LOG_TABLE_NAME', json_str)

Replace 'YOUR_WORKSPACE_ID', 'YOUR_PRIMARY_KEY', and 'CUSTOM_LOG_TABLE_NAME' with your own values.