cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Anonymous
by • Not applicable
  • 1313 Views
  • 1 replies
  • 1 kudos

Resolved! Access to Cluster Logs for non-admins

Suppose I have a DevOps team that needs near real-time access to cluster logs to troubleshoot job failures. What is the best way for me to grant access to view logs without granting them admin access?

Data Engineering
Reply
  • 1313 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 1 kudos

Please use logging option and set destination for sending logs in cluster settings to other Azure Blob or S3 storage (need to be mounted first):

  • 1 kudos
User16765131552
by • Contributor III
  • 1346 Views
  • 1 replies
  • 0 kudos

Resolved! Cluster Log Partitioning

Customer wants to understand our strategy for breaking cluster logs into different partitions and files. They want to be able to ingest these logs into a tool that needs to understand this. They have indicated that the logs used to all be in one file...

Data Engineering
Reply
  • 1346 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
User16765131552
Contributor III
  • 0 kudos

Log files are rolled over by time/size criteria.

  • 0 kudos
User15813097110
by • New Contributor III
  • 3892 Views
  • 1 replies
  • 0 kudos

How to push Cluster Logs to Elastic Search?

Data Engineering
Reply
  • 3892 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
User15813097110
New Contributor III
  • 0 kudos

We can use the below steps to push Cluster Logs to Elastic Search:1. Download the log4j-elasticsearch-java-api repo and build the jar file:git clone https://github.com/Downfy/log4j-elasticsearch-java-api.git cd log4j-elasticsearch-java-api/ mvn clean...

  • 0 kudos
Labels