How to Hunt Down Hidden Server Error Logs

The Open Web Application Security Project (OWASP) offers three tips for maintaining trustworthy server-activity logs:

1. For compliance, audit, and liability purposes, logs should be created in a way that ensures they can’t be overwritten or deleted.
2. Logging frequency depends on the size and volume of the system, but all logs should be checked regularly to ensure the function is active (by running a simple cron job, for example).
3. Ensure that users aren’t shown stack traces, private information, or other sensitive details in error messages; stick with generic messages, such as the standard 404 and 500 HTTP status response codes.

The Apache documentation explains that the error log file in Unix is usually error_log, and in Windows is error.log; Unix systems may also direct the server to send errors to syslog or pipe them to an external program. To continuously monitor the error log, such as during testing, use this command: tail -f error_log. Other Apache log files are process ID (logs/httpd.pid), which is used to restart and terminate the daemon; the ScriptLog, which records the input to and output from CGI scripts; and the RewriteLog, which analyzes the transformation of requests by the rewriting engine.

What to do when a PHP error-log file goes missing?

A Stack Overflow post from October 2012 highlights how challenging it can be to track down a PHP error log that isn’t where you expect it to be. One solution offered was to add this line to the /etc/php.ini file: /var/log/php-scripts.log (other log-discovery options are shown in the image below).

The PHP error_log file can be customized to hide errors from users, log errors to syslog, or other purposes. Source: Stack Overflow

To find the log-file location of a Linux process, you can use lsof (list open files), as explained in a post on the Slash4 blog. Run the code shown below as a root user:

 Find all open log files on a Linux server by combining lsof and grep. Source: Slash4 blog

You can find the process ID (PID) of httpd, MySQL, or other services using the commands shown below:

 The lsof and grep commands can be used to find the PID of a process and to search for open log files. Source: Slash 4 blog Use .htaccess to create private, custom error logs

Apache’s .htaccess configuration file lets you customize your error reporting to ensure the only the people who need to be notified of specific errors can view the reports. Jeff Starr explains the process on his Perishable Press blog. Start by adding the .htaccess directives below to the httpd.conf file of the domain, or alternatively to the site’s root or other directory:

Add these commands to the .htaccess file to keep the error log private, enable PHP error logging, and restrict access to the error log. Source: Perishable Press 

These best practices and more were what we had in mind when we built the Morpheus Cloud Application Management Platform. Morpheus includes a robust logging and monitoring tool that allows you to easily visualize and monitor logs across all your distributed systems. It also makes it simple to consolidate logs across all of your apps, databases, and IT systems for faster trouble shooting. To learn more about Morpheus or to sign up for a free trial, click here.