Patch tactics

Keeping security patches up to date at 120 National Guard sites throughout Alabama was becoming something of a headache for the Guard's IT administrators.'We support about 2,200 computers and servers,' said information security specialist Jim Crawford.That headache was only going to get bigger, because guardsmen issued Common Access Cards by the Defense Department must be given access to the Army's Internet domain.'We are going to have to support our 9,000 traditional National Guardsmen on a full-time basis,' Crawford said. That support will come by adding an additional six to eight computers at each National Guard site.The Alabama National Guard has turned to the Update patch management system from PatchLink Corp. of Scottsdale, Ariz., to handle the growing job. The Army requires the Guard to be fully updated on its patches within 30 days of release, a deadline that Alabama now is able to beat by pushing out patches received through the Update subscription.'In a week I can get it 90 percent patched,' Crawford said. 'In two weeks I'm 100 percent.'Software patches are a critical component in securing IT systems. This puts a premium on being able to trust a patch and automate its installation.PatchLink Update was not the first tool the Alabama Guard turned to for getting this job done. It started with Version 1 of Microsoft Corp.'s System Management Server, Crawford said.'It didn't work at all,' he said. 'Then we migrated to v. 2.0 and it worked pretty good,' except that it had to be heavily tailored for the Guard's environment. 'That's when I got disgusted with Microsoft.'He also tried the Hercules vulnerability management tool from Citadel Security Software Inc. of Dallas. But Hercules could make unexpected changes.There also were some problems with the 12-Kbps circuits that connected most of the Alabama sites. Both SMS and Hercules could eat up that bandwidth with batch distributions of patches. The state recently finished building out a Bell South network with T1 links to each National Guard site, and bandwidth no longer is a critical issue. But even before the upgrade, PatchLink was able to push out patches on an as-can basis without using up all the bandwidth.PatchLink Update is a server-agent system with an online service for delivering new vendor patches to the customer's server. Agents residing on client devices scan the system to determine what patches are needed and if they can be safely installed. They also can confirm successful installation and alert administrators when installation fails or creates a problem.Update confirms installation by searching for signatures created by the patch in each software platform, said Chris Andrew, PatchLink vice president of security technologies. If the signature is not present, something went wrong with the installation.New patches are delivered to Update servers daily. PatchLink also distributes third-party workarounds for vulnerabilities that do not have vendor patches. The patches and fixes are sent out after being tested by PatchLink against a library of up to 150 image configurations to help ensure against conflicts and unexpected results. The company's service level agreement calls for patches deemed critical to be delivered to customers within 24 hours, and longer for moderate and less important patches.Andrew said PatchLink has a 'high degree of confidence' in the testing, but that given the wide variety of end-user platforms, it is not foolproof.'We provide the next level of assurance on top of what the vendor provides,' he said. 'At this point, best practice for our customers is not to take that patch and roll it out, but to do their own tests' and install it in phases to limit any negative impact.The Alabama Guard keeps a pretty tight rein on its system profiles, with 'maybe three different hardware configurations,' Crawford said. This helps speed his local testing and installation. 'When I get it, it takes me an hour to push it out in my lab to see if it blows up anything,' he said. 'Then I'm patching.'The Update client agent has a small footprint, about 5MB, and runs in the background as a system service with negligible impact on performance, Andrew said. Because it runs as a system service, it can patch a locked-down system that the user is not supposed to modify.The agent also helps with ensuring the patching on the nearly 400 laptops that go home each night and connect to the network remotely. When remote users access the network over a virtual private network, the PatchLink server on the intranet is the first thing they hit. If the machine is not up to date, patches can be pushed out. In some rural areas of Alabama, VPN connections are not possible. Those connections over the Internet are pushed to the same PatchLink server for discovery and patching. Performance is about the same with each type of connection, Crawford said.'I don't see a lot of difference,' he said. 'I can't tell which connection was used when the patching was done.'