Upguard Core Automation and Debugging
The Ups and Downs of UpguardOur compliance team wanted a way to be able to audit both Windows and *nix based servers and recently bought a product called Upguard. However, the reports available were not standardized, not downloadable in a meaningful form, and had different information than their legacy reporting tools.
AutomationUpguard offers the ability to run arbitrary scripts (with powershell and bash) and monitor their outputs and provides a mechanism to access node information through it's API. I created augmented scripts that gathered information, and bridged it all together in a unified presentation format over clusters of endpoints. The report is compiled automatically and given to the compliance team to review who then gives it to the audit team. Interaction is minimal, and simply involves selecting a save file destination.
Debugging a Kernel ModuleDuring enrollment of one of the servers, we discovered that installation of the Upguard client was causing a problem with a server and causing it to crash on reboot. I recreated the environment in a virtual machine and reproduced the crash.
After examining the Upguard client, it appeared that it installed essentially a rootkit kernel module. This module hooked several functions of the kernel not necessary for us, in this case, the 32bit network socket functions. We had another piece of software, control-m, that also had a module hooking this function, but that was necessary. Examining the memory pages from the VM at crash it appeared that control-m was asserting the original saved location of its hooked function, and when it did not match, was forcing the crash. Able to provide this information to the vendor, they were able to make a build of the software to allow selective hook binding and this solved our problem.