Linux Logging Guide — MyAdminDesk

This is an original planning guide. It is not a vendor manual, an official standard, or legal advice. Example addresses use documentation ranges. Review every command before you paste it.

A log is evidence with a clock

journald is the default collector on systemd hosts. The journal is binary, on disk or in memory, and journalctl reads it. rsyslog and other syslog daemons read text and can write files or forward lines. A host can run both: journald collects, and a syslog daemon forwards. Know which one you are configuring before you edit a file from the other.

Time comes first. A log from a host whose clock is wrong cannot be lined up with a log from a host whose clock is right. The fix for a lying clock is the time source, not a longer retention. Write the timezone and the NTP or chrony state next to the logging decision.

Retention is a disk decision. A journal that is allowed to fill the root volume will take the host down, which is a performance incident caused by a logging setting. Set a maximum size or a maximum age you can afford, and alert before the volume is full. Forwarding to another host does not remove the need for a local limit. The network can be down on the day you need the lines.

Secrets do not belong in logs. A password, a session token, or a connection string in a debug line becomes a file you now have to protect like a password. If you find one, rotate the secret and stop the program logging it. Do not copy the line into the ticket. The security desk's log guide is the companion for investigations. This page is how the lines get kept.

A lab host that keeps seven days and refuses to fill the disk

The lab decision is seven days of journal on disk, with a size cap that leaves the root volume able to boot. SystemMaxUse and MaxRetentionSec are the kind of keys journald understands. The names on your version are the ones in man journald.conf, not a key you remember from another daemon. The journald generator on this site drafts a drop-in. Read every key. A draft that sets Storage=none will throw away the lines you think you are keeping.

If you also forward with rsyslog, the destination is a host you operate, on a prefix you named, and the forwarding is not the only copy until you have seen a line arrive there. The rsyslog generator drafts that forward. It does not open the collector. Test with one marked line, then search for that mark on the collector.

A service unit's logs are journalctl -u, which the systemd guide already uses. Do not also turn on a debug flag that prints tokens, to see if the journal works. A marked line that says logging-test and the hostname is enough.

After the change, write a line, reboot or restart the daemon as the ticket says, and confirm the line is still there if you asked for persistent storage. A volatile journal looks configured until the next boot.

# Review before you paste. Lab journald drop-in only.
# Confirm keys in man journald.conf on this host.
# Goal: persistent, size-capped, no secrets in the test line.
# logger -t logging-test edge-01
# Then find that line with journalctl, and again after a restart if you required persistence.

A change you can finish

Change retention on the lab host first. Confirm a test line survives the restart you care about, and confirm the cap is visible in the configuration you reloaded. Then copy the same drop-in to the next host, one host at a time.

Rollback is the previous drop-in. Keep it. A logging change that fills the disk is rolled back before you add a second collector.

Confirm the clock and write down the timezone.
Decide journald, syslog, or journald plus a forwarder.
Set a size or age cap that cannot fill the root volume.
Reload the daemon you edited and show the effective config.
Write one marked line with no secret in it.
Find the line locally, and on the collector if you forward.
Restart once if persistence is the requirement, and find the line again.

Mistakes that outlive the command

Linux logging fails in the same few ways. Each one looks finished on the day you type it. The list is here so the ticket can forbid them before the window, not so you can recognise them in an incident review.

Editing rsyslog when the host only has journald, or the reverse.
Storage=volatile when you needed the lines after a reboot.
No size cap, so the journal fills the disk and the host becomes the performance incident.
Forwarding as the only copy, then losing the network and the lines together.
A debug mode that logs passwords or tokens.
A test line that itself contains a real secret.
Clocks that disagree, so two hosts cannot tell one story.
A generator draft applied without deleting the key that disables storage.

Decisions to write down before the window

Decision 1 for Linux logging. The collector is named: journald, syslog, or both, and you know which file you are allowed to edit. Write that decision in the ticket before you change the host. The proof is the smallest command or screen that confirms it, recorded with the hostname and the time. If the host disagrees, the ticket changes. A second change does not go on top of a failed check. The rollback is the previous value you wrote down, not a reboot that hides which line mattered.

Decision 2 for Linux logging. The clock is correct enough to compare hosts. The timezone is written down. Write that decision in the ticket before you change the host. The proof is the smallest command or screen that confirms it, recorded with the hostname and the time. If the host disagrees, the ticket changes. A second change does not go on top of a failed check. The rollback is the previous value you wrote down, not a reboot that hides which line mattered.

Decision 3 for Linux logging. A size cap or an age cap exists and is smaller than the free space you need to boot. Write that decision in the ticket before you change the host. The proof is the smallest command or screen that confirms it, recorded with the hostname and the time. If the host disagrees, the ticket changes. A second change does not go on top of a failed check. The rollback is the previous value you wrote down, not a reboot that hides which line mattered.

Decision 4 for Linux logging. Persistence across reboot is either required and tested, or explicitly not required. Write that decision in the ticket before you change the host. The proof is the smallest command or screen that confirms it, recorded with the hostname and the time. If the host disagrees, the ticket changes. A second change does not go on top of a failed check. The rollback is the previous value you wrote down, not a reboot that hides which line mattered.

Decision 5 for Linux logging. Forwarding goes to a host you operate. The lab mark was found there. Write that decision in the ticket before you change the host. The proof is the smallest command or screen that confirms it, recorded with the hostname and the time. If the host disagrees, the ticket changes. A second change does not go on top of a failed check. The rollback is the previous value you wrote down, not a reboot that hides which line mattered.

Decision 6 for Linux logging. Secrets are out of scope for log lines. A token in a line is rotated, not filed. Write that decision in the ticket before you change the host. The proof is the smallest command or screen that confirms it, recorded with the hostname and the time. If the host disagrees, the ticket changes. A second change does not go on top of a failed check. The rollback is the previous value you wrote down, not a reboot that hides which line mattered.

Decision 7 for Linux logging. The generator draft was read. Storage is not none unless you meant to discard. Write that decision in the ticket before you change the host. The proof is the smallest command or screen that confirms it, recorded with the hostname and the time. If the host disagrees, the ticket changes. A second change does not go on top of a failed check. The rollback is the previous value you wrote down, not a reboot that hides which line mattered.

Decision 8 for Linux logging. The service unit logs are in the journal, and any extra file path is in the ticket. Write that decision in the ticket before you change the host. The proof is the smallest command or screen that confirms it, recorded with the hostname and the time. If the host disagrees, the ticket changes. A second change does not go on top of a failed check. The rollback is the previous value you wrote down, not a reboot that hides which line mattered.

Decision 9 for Linux logging. Rollback is the previous drop-in, kept beside the ticket. Write that decision in the ticket before you change the host. The proof is the smallest command or screen that confirms it, recorded with the hostname and the time. If the host disagrees, the ticket changes. A second change does not go on top of a failed check. The rollback is the previous value you wrote down, not a reboot that hides which line mattered.

Decision 10 for Linux logging. One host is done before the next host gets the same file. Write that decision in the ticket before you change the host. The proof is the smallest command or screen that confirms it, recorded with the hostname and the time. If the host disagrees, the ticket changes. A second change does not go on top of a failed check. The rollback is the previous value you wrote down, not a reboot that hides which line mattered.

Decision 11 for Linux logging. The security companion is for how you investigate. This ticket is only for how you keep the lines. Write that decision in the ticket before you change the host. The proof is the smallest command or screen that confirms it, recorded with the hostname and the time. If the host disagrees, the ticket changes. A second change does not go on top of a failed check. The rollback is the previous value you wrote down, not a reboot that hides which line mattered.

Logging checks before you rely on the lines

Check 1 in Logging checks before you rely on the lines. The clock and timezone are recorded. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 2 in Logging checks before you rely on the lines. The daemon you reloaded is the daemon you edited. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 3 in Logging checks before you rely on the lines. A size or age cap is present in the effective configuration. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 4 in Logging checks before you rely on the lines. The cap cannot fill the root volume. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 5 in Logging checks before you rely on the lines. A marked test line with no secret was found locally. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 6 in Logging checks before you rely on the lines. If persistence matters, the line survived a restart. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 7 in Logging checks before you rely on the lines. If you forward, the same mark was found on the collector. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 8 in Logging checks before you rely on the lines. The generator draft does not set storage to none. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 9 in Logging checks before you rely on the lines. No debug flag that prints secrets is enabled. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 10 in Logging checks before you rely on the lines. The previous configuration is saved for rollback. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 11 in Logging checks before you rely on the lines. journalctl -u still shows the service you care about. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 12 in Logging checks before you rely on the lines. A second person saw the test mark. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 13 in Logging checks before you rely on the lines. The next host is not changed until this host's check passed. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 14 in Logging checks before you rely on the lines. The hostname in the prompt matches the hostname in the ticket for Linux logging. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 15 in Logging checks before you rely on the lines. The result line includes the time and the person, not only the command. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 16 in Logging checks before you rely on the lines. You can point at the rollback value without searching a chat history. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 17 in Logging checks before you rely on the lines. Done is a sentence a second person can repeat: the proof for Linux logging happened, and the next host is not started early. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Tools for this guide

These tools run in the browser on this site. They do not log into your machines.

Guides in this cluster

These supporting pages belong to this pillar. A published page is linked. The others are named until they are published.

The content plan also names a companion in another desk: Security: Log Analysis for Security.