Software Testing Automation: A Comparative Study on Productivity Rate of Open Source Automated Software Testing Tools For Smart Manufacturing

Ashok Sivaji, Rosnisa Abdul Razak, Nur Faezah Mohamad, Nurshakirin Sazali, Afiqah Musa, Norzam Mohd Bajuri, Aslinda Md Hashim, Mohd Solehuddin Abdullah, Nur Diyana Joha, Nadia Ellyani Azis, Anjana Devi N. Kuppusamy, Azlan Deniel, Ngip Khean Chuan, Torkil Clemmensen

Research output: Chapter in Book/Report/Conference proceedingArticle in proceedingsResearchpeer-review


With the growth in emerging technologies, manual software testing possesses a challenge to project team in terms of test planning, test case creation, execution and reporting. Software testing automation with open source tools promises productivity savings for software test engineers. The objective of this study is to compare the level of productivity of open source Smart Manufacturing toolkits for web application testing. Together with literature analysis, subject matter experts in software testing were interviewed and were requested to experiment and compare the productivity rate of two tools in use for the various steps in the software development lifecycle. Katalon Studio 7.0 was found to be a more smart and productive tool than its predecessor, Robot Framework 3.0 in terms of installation, scripting, verification and maintenance for web application testing. By employing a Wilcoxon Matched Pairs Signed-Rank Test, results revealed that the null hypothesis of “ no statistically significant differences between SME productivity rate in using RF and KS to performing software test automation in terms of time taken” can be rejected , Z= −6.399, p < 0.05. Future work could replicate similar studies for selecting mobile manufacturing toolkits that will be suitable and improve productivity rate.
Original languageEnglish
Title of host publication2020 IEEE Conference on Open Systems (ICOS)
EditorsMohd Hazli Bin Mohamed Zabil
Number of pages6
Place of PublicationLos Alamitos, CA
Publication date2020
Article number9293650
ISBN (Print)9781728190204
Publication statusPublished - 2020
Event2020 IEEE Conference on Open Systems (ICOS): Online Conference - Online
Duration: 17 Nov 202019 Nov 2020


Conference2020 IEEE Conference on Open Systems (ICOS)
Internet address


  • Software testing automation
  • Smart tools
  • Productivity rate
  • Software test engineers
  • Open source tool

Cite this