I’ve just been looking at a nice new tool called sysdig, which seems to be really useful for analysing and troubleshooting on production systems. There’s a great blog post by Gianluca Borello, detailing how he set up a number of honey-pot servers with poor passwords, and then captured system activity with sysdig, showing exactly how his server was compromised, and what the hacker did at each stage. The level of detail he was able to garner is astounding, and I can see how powerful this tool could be in the future, for any sort of troubleshooting where it’s not clear exactly what has happened/is happening on a system.
There’s also a nice post on using sysdig to find data from logfiles, without knowing the name of any of the log files before hand, and being able to correlate the output across multiple files, belonging to different applications. Really cool stuff!