Smoke Testing of Cloud Systems

Setting up a reliable and automated testing environment for cloud systems can be challenging due to the many issues that might compromise the stability of the deployments, such as poorly configured services and faulty infrastructure scripts. Test failures due to deployment procedures that have not terminated correctly may cause a significant waste of effort, resources, and time to organizations. Smoke testing can be used to design test suites that can be quickly executed to check if a system under test is operational before running any other test suite, which is executed only once the smoke test cases pass. Indeed, smoke testing can help prevent the generation of spurious test failures that would not be worth inspecting. So far, the design of smoke test suites has been entirely left to the intuition of testers, who may apply arbitrary and potentially weak criteria for their design. This paper addresses this issue by approaching smoke testing of cloud applications systematically. In particular, it introduces a reference model and nine adequacy criteria that can help practitioners design effective smoke test suites for their cloud systems. The assessment of smoke test suites satisfying the nine adequacy criteria on 60 versions of two industrial systems provides evidence of the trade-offs between the cost and the failure detection capability of the criteria.