Linux Performance Troubleshooting Guide

This is an original planning guide. It is not a vendor manual, an official standard, or legal advice. Example addresses use documentation ranges. Review every command before you paste it.

A slow host is a measurement, then one change

Load average is the run queue, not a percentage. A load of 4 on a host with 4 CPUs is a busy host. A load of 4 on a host with 32 CPUs is a quiet one. Read the number next to the CPU count before you call it high. The load interpreter on this site does that arithmetic. It does not log into the host.

Memory pressure is about available memory and swap, not the word free. Linux uses free RAM for cache and will give it back. Available is the number that answers whether a new process can start. Swap in use with available memory still high is not automatically a crisis. Swap in use with available memory near zero is a host that is out of memory or about to be.

If the CPUs are idle and the load is high, look at disk wait. A volume at 100 percent, a RAID rebuild, or a backup running at the same time as the app will look like a slow application. The storage guide is the volume. This page is the order you look in so you do not tune the application while the disk is the stall.

One change per window. A sysctl, a bigger VM, and a new index, all in the same hour, leave you unable to say which one helped. This guide does not hand you a list of kernel tweaks. Most of them are how a host gets stranger. Measure, change the one bottleneck you can name, and keep the previous value.

A lab hour when the site feels slow

Start with the sentence a person can feel: pages take ten seconds, or the SSH session lags, or a job that took five minutes now takes thirty. Write the time of day. Then read uptime, the CPU count, and available memory. If load is below the CPU count and available memory is comfortable, you do not have a CPU or RAM emergency. Look at the disk and at the one process that is new.

The process list shows who is using CPU and who is waiting. A backup, a log rotate, or a find someone left walking the tree are common and boring, which is why you look for them before you resize the server. The hardware guide is the resize. Resizing a host that is waiting on a full disk gives you a larger host that is still waiting.

Keep one before and one after. The before is the uptime line, the available memory, and the disk percent. The after is the same three after the single change. If you cannot name the change in one sentence, you made more than one.

The health-check script generator drafts a read-only collection of those facts. Read it before you run it. A draft that changes a sysctl or restarts a service is not a health check. Delete those lines. The script should exit without writing configuration.

A change you can finish

The allowed changes in a performance window are small: stop a runaway you started, move a backup out of the busy hour, free a volume that is full because of a log you can rotate, or add the capacity the hardware ticket already approved. A kernel parameter is out of scope unless you already have a measured reason and the previous value.

Rollback is putting that previous value back, or starting the service you stopped, or letting the backup return to its old schedule. You write the rollback before the change, while the host is still slow and you can still think.

Write the symptom and the clock time.
Read load next to CPU count, and available memory, not only free.
If CPUs are idle, look at disk space and disk wait before you look at application code.
Name the one process or job that is new or huge.
Change one thing. Record the previous value.
Read the same three numbers again.
Stop if the numbers did not move. The next change is a new ticket.

Mistakes that outlive the command

Linux performance troubleshooting fails in the same few ways. Each one looks finished on the day you type it. The list is here so the ticket can forbid them before the window, not so you can recognise them in an incident review.

Treating load average as a percentage.
Rebooting because it was slow. The reboot erases the process list you needed.
Adding RAM to a host whose volume is full.
A handful of sysctls from a forum, with no previous values and no measurement.
Killing a process you cannot name. Stopping the database to see if the host feels faster is not a test.
Running a backup, a RAID rebuild, and a report at the same hour, then blaming the application.
A health-check script that restarts services.
Declaring victory from one minute of lower load while the daily job has not come around again.

Decisions to write down before the window

One decision for Linux performance troubleshooting: Load is compared with CPU count. The interpreter's sentence is in the ticket. Available memory is the memory number you quote. Cache is not counted as pressure by itself. Disk space and disk wait are checked before any CPU tuning. One change. The previous value is written down. A reboot is not the diagnostic. Console access exists if you ever do reboot, and the hardware guide owns that decision. The health-check draft is read-only. You deleted anything that writes config or restarts a unit. Backups and rebuilds have a clock time that is not the busy hour, or the busy hour is accepted in writing. You can name the bottleneck in one sentence a second person understands. If the numbers did not move, you did not add a second change in the same window. Rollback was tested in the sense that you know the exact previous value, not in the sense that you rebooted. And no sysctl was changed unless the ticket already contained the measurement and the old value.

Performance checks before you call the host fixed

Confirm this before you call the change done: The symptom sentence and the clock time are in the ticket. Load was read next to the CPU count. Available memory was read, not only the free column. Disk use and wait were looked at before a resize. The one changed thing is named. The previous value is in the ticket. Before and after numbers are both written down. No reboot was used as the test. The health-check script does not restart services. A second person can say the bottleneck in one sentence. The next busy hour is still ahead, or you watched a representative load. Rollback text is the previous value, and it does not say reboot. Logs from the same hour were kept, without secrets, for the logging guide's retention. The hostname in the prompt matches the hostname in the ticket for Linux performance troubleshooting. The result line includes the time and the person, not only the command. You can point at the rollback value without searching a chat history. And done is a sentence a second person can repeat: the proof for Linux performance troubleshooting happened, and the next host is not started early. If one of those fails, stop.

Tools for this guide

These tools run in the browser on this site. They do not log into your machines.

Guides in this cluster

These supporting pages belong to this pillar.