Logs are usually the first place you should look for answers when you’re trying to fix an issue in linux.
In order to efficiently sift through logs you need to know where and what to look for. This can be a pretty broad topic depending on what type of problem you’re trying to solve. The scope of this article will cover the basics.
How to View Log Files:
Example commands that can be run to search words in the /var/log/messages file:
1. sudo less /var/log/messages 2. sudo grep -Ei "error|fail" /var/log/messages 3. sudo tail -f /var/log/messages | grep -E "error|fail"
After running command 1, type “/”, type “error”, “fail”, etc and then hit enter. The “n” key will jump to the next occurrence of the word. You can type the letter “q” to quit out of the “less” command. Number 2 will output only lines with either “error” or “fail” in the file. Lastly, 3 will output a continuous stream of log lines containing the words “error” or “fail” as it is written to the file. This is useful when you want to monitor a log file that frequently gets updated.
Log Locations and What to Look For
/var/log/messages
/var/log/syslog
If you’re experiencing general issues with your OS this can be a good starting place. You can reference these logs for information related to most core components of the OS as well as some service daemon messages.
Most of the time you will want to locate the section in a log that roughly matches the same time frame in which you experienced issues. Most log entries include a date at the beginning of the line. Some of the most common words to look for in logs are error or fail.
/var/log/auth.log
This is a great place to start looking if you’re trying to troubleshoot valid login failures or potential hacking attempts. These logs will store information related authentication and commands that have been run as sudo/root.
The most common things to search for are usernames or IPs. Software like fail2ban utilizes these types of logs to identify threat actors repeatedly attempting SSH connections and block their IP address using software firewalls.
RHEL,CentOS, etc: If you’re trying to track down some issue related to software installations/updates that may have caused some issues you’ll find timestamps and version numbers listed in this log file.
The sudo yum history command also displays this information as well as provides the ability to revert actions performed with yum. For example you could run sudo yum history undo 40 and the last yum command marked as ID 40 would be reverted.
/var/log/apt/history.log
Ubuntu,Debian, etc: These contain timestamped history of package installations, updates and removals. If you wanted to see all of the recent package installations or deletions you could run the following grep commands.
grep " install " /var/log/dpkg.log grep " remove " /var/log/dpkg.log
/var/log/httpd/
The locations of these logs can vary depending on your configuration so you may have to do a little searching. The following lsof command can find logs fairly easily. Replace the bold httpd with apache2 depending on what your Apache process is named.
sudo lsof -nc httpd | awk '{print $1 ": " $9}' |\
grep -Ei 'access.*log|error.*log' | sort --uniqueCommon things to look for in these logs are any errors that might help track down why certain pages might not be loading properly for users. If you have a page that isn’t loading I recommend using the tail -f mentioned earlier while you’re visiting your website to see if you can watch an error be generated on the fly.
/var/log/mariadb/
These are useful for finding errors that might keep your database server from starting or performing as expected. You can use lsof in order to locate log files open.
sudo lsof -nc mysqld | awk '{print $1 ": " $9}' |\
grep -Ei 'mariadb.*|mysqld.*log' | sort --uniqueA lot of times these logs will be helpful when trying to determine why your database service will not start. For example if your server has run out of disk space you should see some errors writing files. In this case you could use a command like the following to better understand disk usage.
sudo du -h --max-depth=1 --exclude /proc /
Conclusion
Log diving can be intimidating if you’re just getting started. However, the more practice you get with them the more familiar you will become when looking for the information that matters to you.
