Linux Performance Troubleshooting Guide

This is an original planning guide. It is not a vendor manual, an official standard, or legal advice. Example addresses use documentation ranges. Review every command before you paste it.

A slow host is a measurement, then one change

Load average is the run queue, not a percentage. A load of 4 on a host with 4 CPUs is a busy host. A load of 4 on a host with 32 CPUs is a quiet one. Read the number next to the CPU count before you call it high. The load interpreter on this site does that arithmetic. It does not log into the host.

Memory pressure is about available memory and swap, not the word free. Linux uses free RAM for cache and will give it back. Available is the number that answers whether a new process can start. Swap in use with available memory still high is not automatically a crisis. Swap in use with available memory near zero is a host that is out of memory or about to be.

If the CPUs are idle and the load is high, look at disk wait. A volume at 100 percent, a RAID rebuild, or a backup running at the same time as the app will look like a slow application. The storage guide is the volume. This page is the order you look in so you do not tune the application while the disk is the stall.

One change per window. A sysctl, a bigger VM, and a new index, all in the same hour, leave you unable to say which one helped. This guide does not hand you a list of kernel tweaks. Most of them are how a host gets stranger. Measure, change the one bottleneck you can name, and keep the previous value.

A lab hour when the site feels slow

Start with the sentence a person can feel: pages take ten seconds, or the SSH session lags, or a job that took five minutes now takes thirty. Write the time of day. Then read uptime, the CPU count, and available memory. If load is below the CPU count and available memory is comfortable, you do not have a CPU or RAM emergency. Look at the disk and at the one process that is new.

The process list shows who is using CPU and who is waiting. A backup, a log rotate, or a find someone left walking the tree are common and boring, which is why you look for them before you resize the server. The hardware guide is the resize. Resizing a host that is waiting on a full disk gives you a larger host that is still waiting.

Keep one before and one after. The before is the uptime line, the available memory, and the disk percent. The after is the same three after the single change. If you cannot name the change in one sentence, you made more than one.

The health-check script generator drafts a read-only collection of those facts. Read it before you run it. A draft that changes a sysctl or restarts a service is not a health check. Delete those lines. The script should exit without writing configuration.

A change you can finish

The allowed changes in a performance window are small: stop a runaway you started, move a backup out of the busy hour, free a volume that is full because of a log you can rotate, or add the capacity the hardware ticket already approved. A kernel parameter is out of scope unless you already have a measured reason and the previous value.

Rollback is putting that previous value back, or starting the service you stopped, or letting the backup return to its old schedule. You write the rollback before the change, while the host is still slow and you can still think.

Write the symptom and the clock time.
Read load next to CPU count, and available memory, not only free.
If CPUs are idle, look at disk space and disk wait before you look at application code.
Name the one process or job that is new or huge.
Change one thing. Record the previous value.
Read the same three numbers again.
Stop if the numbers did not move. The next change is a new ticket.

Mistakes that outlive the command

Linux performance troubleshooting fails in the same few ways. Each one looks finished on the day you type it. The list is here so the ticket can forbid them before the window, not so you can recognise them in an incident review.

Treating load average as a percentage.
Rebooting because it was slow. The reboot erases the process list you needed.
Adding RAM to a host whose volume is full.
A handful of sysctls from a forum, with no previous values and no measurement.
Killing a process you cannot name. Stopping the database to see if the host feels faster is not a test.
Running a backup, a RAID rebuild, and a report at the same hour, then blaming the application.
A health-check script that restarts services.
Declaring victory from one minute of lower load while the daily job has not come around again.

Decisions to write down before the window

Decision 1 for Linux performance troubleshooting. Load is compared with CPU count. The interpreter's sentence is in the ticket. Write that decision in the ticket before you change the host. The proof is the smallest command or screen that confirms it, recorded with the hostname and the time. If the host disagrees, the ticket changes. A second change does not go on top of a failed check. The rollback is the previous value you wrote down, not a reboot that hides which line mattered.

Decision 2 for Linux performance troubleshooting. Available memory is the memory number you quote. Cache is not counted as pressure by itself. Write that decision in the ticket before you change the host. The proof is the smallest command or screen that confirms it, recorded with the hostname and the time. If the host disagrees, the ticket changes. A second change does not go on top of a failed check. The rollback is the previous value you wrote down, not a reboot that hides which line mattered.

Decision 3 for Linux performance troubleshooting. Disk space and disk wait are checked before any CPU tuning. Write that decision in the ticket before you change the host. The proof is the smallest command or screen that confirms it, recorded with the hostname and the time. If the host disagrees, the ticket changes. A second change does not go on top of a failed check. The rollback is the previous value you wrote down, not a reboot that hides which line mattered.

Decision 4 for Linux performance troubleshooting. One change. The previous value is written down. Write that decision in the ticket before you change the host. The proof is the smallest command or screen that confirms it, recorded with the hostname and the time. If the host disagrees, the ticket changes. A second change does not go on top of a failed check. The rollback is the previous value you wrote down, not a reboot that hides which line mattered.

Decision 5 for Linux performance troubleshooting. A reboot is not the diagnostic. Console access exists if you ever do reboot, and the hardware guide owns that decision. Write that decision in the ticket before you change the host. The proof is the smallest command or screen that confirms it, recorded with the hostname and the time. If the host disagrees, the ticket changes. A second change does not go on top of a failed check. The rollback is the previous value you wrote down, not a reboot that hides which line mattered.

Decision 6 for Linux performance troubleshooting. The health-check draft is read-only. You deleted anything that writes config or restarts a unit. Write that decision in the ticket before you change the host. The proof is the smallest command or screen that confirms it, recorded with the hostname and the time. If the host disagrees, the ticket changes. A second change does not go on top of a failed check. The rollback is the previous value you wrote down, not a reboot that hides which line mattered.

Decision 7 for Linux performance troubleshooting. Backups and rebuilds have a clock time that is not the busy hour, or the busy hour is accepted in writing. Write that decision in the ticket before you change the host. The proof is the smallest command or screen that confirms it, recorded with the hostname and the time. If the host disagrees, the ticket changes. A second change does not go on top of a failed check. The rollback is the previous value you wrote down, not a reboot that hides which line mattered.

Decision 8 for Linux performance troubleshooting. You can name the bottleneck in one sentence a second person understands. Write that decision in the ticket before you change the host. The proof is the smallest command or screen that confirms it, recorded with the hostname and the time. If the host disagrees, the ticket changes. A second change does not go on top of a failed check. The rollback is the previous value you wrote down, not a reboot that hides which line mattered.

Decision 9 for Linux performance troubleshooting. If the numbers did not move, you did not add a second change in the same window. Write that decision in the ticket before you change the host. The proof is the smallest command or screen that confirms it, recorded with the hostname and the time. If the host disagrees, the ticket changes. A second change does not go on top of a failed check. The rollback is the previous value you wrote down, not a reboot that hides which line mattered.

Decision 10 for Linux performance troubleshooting. Rollback was tested in the sense that you know the exact previous value, not in the sense that you rebooted. Write that decision in the ticket before you change the host. The proof is the smallest command or screen that confirms it, recorded with the hostname and the time. If the host disagrees, the ticket changes. A second change does not go on top of a failed check. The rollback is the previous value you wrote down, not a reboot that hides which line mattered.

Decision 11 for Linux performance troubleshooting. No sysctl was changed unless the ticket already contained the measurement and the old value. Write that decision in the ticket before you change the host. The proof is the smallest command or screen that confirms it, recorded with the hostname and the time. If the host disagrees, the ticket changes. A second change does not go on top of a failed check. The rollback is the previous value you wrote down, not a reboot that hides which line mattered.

Performance checks before you call the host fixed

Check 1 in Performance checks before you call the host fixed. The symptom sentence and the clock time are in the ticket. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 2 in Performance checks before you call the host fixed. Load was read next to the CPU count. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 3 in Performance checks before you call the host fixed. Available memory was read, not only the free column. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 4 in Performance checks before you call the host fixed. Disk use and wait were looked at before a resize. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 5 in Performance checks before you call the host fixed. The one changed thing is named. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 6 in Performance checks before you call the host fixed. The previous value is in the ticket. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 7 in Performance checks before you call the host fixed. Before and after numbers are both written down. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 8 in Performance checks before you call the host fixed. No reboot was used as the test. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 9 in Performance checks before you call the host fixed. The health-check script does not restart services. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 10 in Performance checks before you call the host fixed. A second person can say the bottleneck in one sentence. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 11 in Performance checks before you call the host fixed. The next busy hour is still ahead, or you watched a representative load. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 12 in Performance checks before you call the host fixed. Rollback text is the previous value, and it does not say reboot. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 13 in Performance checks before you call the host fixed. Logs from the same hour were kept, without secrets, for the logging guide's retention. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 14 in Performance checks before you call the host fixed. The hostname in the prompt matches the hostname in the ticket for Linux performance troubleshooting. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 15 in Performance checks before you call the host fixed. The result line includes the time and the person, not only the command. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 16 in Performance checks before you call the host fixed. You can point at the rollback value without searching a chat history. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 17 in Performance checks before you call the host fixed. Done is a sentence a second person can repeat: the proof for Linux performance troubleshooting happened, and the next host is not started early. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Tools for this guide

These tools run in the browser on this site. They do not log into your machines.

Guides in this cluster

These supporting pages belong to this pillar. A published page is linked. The others are named until they are published.