Setting up a reliable and automated testing environment for cloud systems can be challenging due to the many issues that might compromise the stability of the deployments, such as poorly configured services and faulty infrastructure scripts. Test failures due to deployment procedures that have not terminated correctly may cause a significant waste of effort, resources, and time to organizations. Smoke testing can be used to design test suites that can be quickly executed to check if a system under test is operational before running any other test suite, which is executed only once the smoke test cases pass. Indeed, smoke testing can help prevent the generation of spurious test failures that would not be worth inspecting. So far, the design of smoke test suites has been entirely left to the intuition of testers, who may apply arbitrary and potentially weak criteria for their design. This paper addresses this issue by approaching smoke testing of cloud applications systematically. In particular, it introduces a reference model and nine adequacy criteria that can help practitioners design effective smoke test suites for their cloud systems. The assessment of smoke test suites satisfying the nine adequacy criteria on 60 versions of two industrial systems provides evidence of the trade-offs between the cost and the failure detection capability of the criteria.

Cannavacciuolo, C., Mariani, L. (2022). Smoke Testing of Cloud Systems. In Proceedings - 2022 IEEE 15th International Conference on Software Testing, Verification and Validation, ICST 2022 (pp.47-57). Institute of Electrical and Electronics Engineers Inc. [10.1109/ICST53961.2022.00016].

Smoke Testing of Cloud Systems

Mariani L.
2022

Abstract

Setting up a reliable and automated testing environment for cloud systems can be challenging due to the many issues that might compromise the stability of the deployments, such as poorly configured services and faulty infrastructure scripts. Test failures due to deployment procedures that have not terminated correctly may cause a significant waste of effort, resources, and time to organizations. Smoke testing can be used to design test suites that can be quickly executed to check if a system under test is operational before running any other test suite, which is executed only once the smoke test cases pass. Indeed, smoke testing can help prevent the generation of spurious test failures that would not be worth inspecting. So far, the design of smoke test suites has been entirely left to the intuition of testers, who may apply arbitrary and potentially weak criteria for their design. This paper addresses this issue by approaching smoke testing of cloud applications systematically. In particular, it introduces a reference model and nine adequacy criteria that can help practitioners design effective smoke test suites for their cloud systems. The assessment of smoke test suites satisfying the nine adequacy criteria on 60 versions of two industrial systems provides evidence of the trade-offs between the cost and the failure detection capability of the criteria.
paper
build verification testing; cloud testing; flaky tests; kubernetes; microservices; smoke testing; testing distributed systems;
English
15th IEEE International Conference on Software Testing, Verification and Validation, ICST 2022 - 4 April 2022 through 13 April 2022
2022
Proceedings - 2022 IEEE 15th International Conference on Software Testing, Verification and Validation, ICST 2022
9781665466790
2022
47
57
none
Cannavacciuolo, C., Mariani, L. (2022). Smoke Testing of Cloud Systems. In Proceedings - 2022 IEEE 15th International Conference on Software Testing, Verification and Validation, ICST 2022 (pp.47-57). Institute of Electrical and Electronics Engineers Inc. [10.1109/ICST53961.2022.00016].
File in questo prodotto:
Non ci sono file associati a questo prodotto.

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/10281/395071
Citazioni
  • Scopus 5
  • ???jsp.display-item.citation.isi??? 3
Social impact