Debug a crash after it happened – without knowing about it upfront

Do you happen to find yourself in the situation that you have a program crash every now and then, but whenever this happens to you, you did not start the app in gdb?

Then I know that feeling; this guide will help you configure your test system to create a coredump whenever an app crashes, so if needed, you can use those at any later time to still create a backtrace. There are obviously some limitations, like stepping through and anything else funny you could do while debugging a running app. But it still does give you a good entry point.

NOTE: I do not advise to set this up on shared machines or critical machines. A coredump can contain sensitive data (memory extracts), which you would not want to be shared around.

With that cleared, it’s as simple as a few configuration steps:

Create any directory where you want to keep your coredumps. I use /cores
mkdir -m 777 /cores
This creates a world readable and writable directory

Configure your system to know WHERE to store the coredumps and how to name them. Create a file /etc/sysctl.d/99-coredump.conf, with the following content:
# I want to have core dumps in a world writable directory

kernel.core_pattern = /cores/%e-%t-%u.core
The pattern results in a filename /cores/NameOfBinary-TimeStamp-UserID.core; this helps yourself to identify the more recent crashes.

Modify the file /etc/security/limits.conf and add the line
* soft core unlimited
instructing the system to actually write coredumps for you.

Reboot your system and go ahead, make your apps crash. you will see files appear in /cores, if all is setup right.

You can at any time later start gdb with two parameters, first one pointing to the binary to debug, 2nd parameter to the corefile. Then normal gbd usage applies (if you miss debuginfo packages, you can even still install them at this time: the coredump remains valid). The usage of gdb goes beyond the scope of this article.