Testing 1-2-3: Open-source tools to ensure quality applications

When the HealthCare.gov rollout did not quite go according to plan, much was made about the absence of “testing.” There have been myriad newspaper columns, cable talk show segments, even exchanges at congressional hearings dedicated to the topic. Though the attention is gratifying to a software guy like me, implicit in the discussion is the premise that testing is a monolithic activity to be performed once development is complete.

I object to that premise. Testing is much more nuanced than has been suggested by recent commentary. Developers, testers and domain experts can work together to leverage a combination of open-source tools, automation and discipline in order to build quality into their applications from the start.

These four kinds of tests set up a foundation for quality:

• Unit and integration tests. Written by developers as they build features to verify their code works as expected and to help refine design. Note the distinction is the subject of much debate. 
• Acceptance tests. Written by testers with the project sponsor and developers to verify an application works as expected for its stakeholders.
• Non-functional tests. Acceptance tests pertaining to characteristics of the application  — that means performance, scalability and security.
• Code quality tests. Configured by developers to scan for poorly written code that lends itself to more bugs and less maintainability.

Let’s examine each in some detail.

Unit and integration tests

Unit tests verify individual components in isolation through mock objects standing in for real dependencies. Consequently, they should run extremely fast and produce idempotent results. Integration tests verify the components against real, or real-ish, parts of the system. 

Again, the distinction is fuzzy, but just know there are many open-source options. In Java development, for example, devs can run a suite of tests with JUnit or TestNG. EasyMock can mock components, and popular frameworks like Spring, RESTEasy, and Apache Camel offer mock implementations of their components as well. Developers can also test against a lightweight Web server like Jetty and/or an in-memory database like H2 or HSQLDB. Analogs exist for other ecosystems as well — both on the front end and back end. 

Acceptance tests

Once developers get started on a new feature, testers should get to work immediately on a suite of tests to verify that feature is production-quality — if not necessarily 100 percent bug-free. 

Cucumber is a crown jewel of Behavior-Driven Development (which can be applied to unit and integration testing as well). Though written in Ruby, Cucumber works with all major ecosystems. Here is an example of a specification from a Cucumber tutorial:

Feature: Contact me
In order to get in touch
A visitor
Should send me a message by contact form

Scenario: Sends a contact message
Given I am on the contact page
And I fill in "contact visitor name" with "John"
And I fill in "contact visitor email" with "john@mail.me"
And I fill in "contact subject" with "Hello"
And I fill in "contact message" with "Great post!"
When I press "Send message"

It resembles plain English  — albeit from a robot. A tester should work with the project sponsor to create specifications and then with developers to map each line in a scenario to a line of code performing the function described. Cucumber also works with other popular tools in the space like Selenium. It provides a nexus for domain experts and developers to share knowledge.

Non-functional tests

There are few viable open-source options for non-functional tests. 

Performance tests help uncover bottlenecks, memory leaks and so on. Rails has performance testing built in. Commercial products appear to be the best bet with other ecosystems.

There is one good open-source option for scalability tests: Apache JMeter. It is a Java desktop application that can be a bit complicated to set up, but its load-testing functionality is not confined to Java applications.

With security, members of the  information assurance group will have something they like to use. Work with them to set up regular scans of the application so vulnerabilities become apparent immediately.

Code quality tests

Bad code is hard to fix and harder to change, which obstructs progress. Breaches of convention, repetitive blocks of code, dependencies on implementation details over abstractions and other common gotchas can cripple a project. Findbugs, PMD and Checkstyle for Java, CodeNarc for Groovy and FxCop for .NET are examples of open-source tools that can uncover these problems. 

It takes a village

Quality demands commitment from the entire team. The project sponsor must resist the urge to sacrifice quality for the illusion of productivity and must commit to provide input to testers on acceptance tests. Developers and testers must maximize the value of their tools to write tests fast and to run tests fast. The book "xUnit Test Patterns" is an expansive source for them to learn how while building great biceps. 

As a library of tests is built, it becomes the one form of documentation the team can count on to be current. Something this critical demands an infrastructure dedicated to automation and metrics. Automation confirms the application is always in a working state by ensuring new changes don’t break existing functionality and disseminating immediate feedback otherwise. Metrics convey project health and opportunities to get even better.

Continuous integration (CI) interfaces with version control form the fulcrum of project automation. Configure Jenkins or Hudson to run unit tests (because they are fast) every time someone commits to version control so it’s immediately apparent when a test that once passed now fails. To balance the latency of longer running tests with the need for prompt feedback, devs can, for example, configure integration tests to run every four hours, acceptance and quality tests to run every night and JMeter tests to run every week. Integration of code coverage tools like Cobertura for Java or SimpleCov for Ruby will verify the tests are thorough. Hopefully performance and security tests can be hooked into CI as well. Note that CI can automate just about anything. Beyond testing, CI promotes quality also by automating builds, documentation and configuration management.

SonarQube provides a dashboard of quality metrics for all ecosystems. While it comes with its own functionality, it also integrates with many of the other tools mentioned. SonarQube displays a diverse, configurable array of metrics and trends in an intuitive interface that depicts the health of a project instantly.

HealthCare.gov was a wake-up call to the entire federal IT community. Let’s make sure we take the right lesson from that experience by leveraging open-source tools and team discipline to build quality into our products from the start.