Visual Testing of GUIs by Abstraction
2020, ArXiv
Abstract
Ensuring the correct visual appearance of graphical user interfaces (GUIs) is important because visual bugs can cause substantial losses for businesses. An application might behave functionally correct in an automated test, but visual bugs can make the GUI effectively unusable for the user. Most of today's approaches for visual testing are pixel-based and tend to have flaws that are characteristic for image differencing. For instance, minor and unimportant visual changes often cause false positives, which confuse the user with unnecessary error reports. Our idea is to introduce an abstract GUI state (AGS), where we define structural relations to identify relevant GUI changes and ignore those that are unimportant from the user's point of view. In addition, we explore several strategies to address the GUI element identification problem in terms of AGS. This allows us to provide rich diagnostic information that help the user to better interpret changes. Based on the principles ...
References (47)
- A. Alameer, S. Mahajan, and W. G. J. Halfond, "Detecting and localizing in- ternationalization presentation failures in web applications," in Proceedings of the 9th IEEE International Conference on Software Testing, Verification and Validation, ser. ICST '16, Chicago, IL, USA: IEEE, 2016, pp. 202-212.
- F. N. Egger, ""trust me, i'm an online vendor": Towards a model of trust for e-commerce system design, " in CHI 2000 Extended Abstracts on Human Factors in Computing Systems, ser. CHI EA '00, The Hague, Netherlands: ACM, 2000, pp. 101-102.
- A. Everard and D. F. Galletta, "How presentation flaws affect perceived site quality, trust, and intention to purchase from an online store, " Journal of Man- agement Information Systems, vol. 22, no. 3, pp. 56-95, Jan. 2005.
- B. J. Fogg, J. Marshall, O. Laraki, A. Osipovich, C. Varma, N. Fang, J. Paul, A. Rangnekar, J. Shon, P. Swani, and M. Treinen, "What makes web sites credible? a report on a large quantitative study," in Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, ser. CHI '01, Seattle, WA, USA: ACM, 2001, pp. 61-68.
- W. Li, M. J. Harrold, and C. Görg, "Detecting user-visible failures in ajax web applications by analyzing users' interaction behaviors," in Proceedings of the 25th ACM/IEEE International Conference on Automated Software Engineering, ser. ASE '10, Antwerp, Belgium: ACM, 2010, pp. 155-158.
- M. Battat. (Apr. 2019). What is applitools? [Online]. Available: https://applitools. com/blog/what-is-applitools/.
- I. Althomali, G. M. Kapfhammer, and P. McMinn, "Automatic visual verification of layout failures in responsively designed web pages, " in Proceedings of the 12th IEEE International Conference on Software Testing, Verification and Validation, ser. ICST '19, Xi'an, China: IEEE, 2019, pp. 183-193.
- C. Luo. (May 2018). Resizemybrowser, [Online]. Available: http://resizemybrowser. com/.
- QuirkTools. (Apr. 2019). Screenfly, [Online]. Available: http://quirktools.com/ screenfly/.
- NETBURN web solutions. (Apr. 2019). Testsize.com, [Online]. Available: http: //testsize.com/.
- Galen Framework. (Mar. 2019). Galen framework, [Online]. Available: http: //galenframework.com/.
- S. Hallé, N. Bergeron, F. Guerin, G. Le Breton, and O. Beroual, "Declarative layout constraints for testing web applications, " Journal of Logical and Algebraic Methods in Programming, vol. 85, no. 5, pp. 737-758, Aug. 2016.
- P. Panchekha, A. T. Geller, M. D. Ernst, Z. Tatlock, and S. Kamil, "Verifying that web pages have accessible layout," in Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation, ser. PLDI '18, Philadelphia, PA, USA: ACM, 2018, pp. 1-14.
- D. Zaiats. (Apr. 2019). Automotion, [Online]. Available: https://automotion. itarray.net/.
- B. Slatkin. (Jun. 2013). The secret to safe continuous deployment, [Online]. Available: https://youtu.be/UMnZiTL0tUc/.
- M. C. Feathers, Working effectively with legacy code, 1st ed. Upper Saddle River, NJ, USA: Prentice Hall, 2004.
- S. Mahajan, B. Li, P. Behnamghader, and W. G. J. Halfond, "Using visual symp- toms for debugging presentation failures in web applications, " in Proceedings of the 9th IEEE International Conference on Software Testing, Verification and Validation, ser. ICST '16, Chicago, IL, USA: IEEE, 2016, pp. 191-201.
- S. Mahajan and W. G. J. Halfond, "Detection and localization of html presenta- tion failures using computer vision-based techniques, " in Proceedings of the 8th IEEE International Conference on Software Testing, Verification and Validation, ser. ICST '15, Graz, Austria: IEEE, 2015, pp. 1-10.
- S. Roy Choudhary, M. R. Prasad, and A. Orso, "X-pert: Accurate identification of cross-browser issues in web applications," in Proceedings of the 35th Inter- national Conference on Software Engineering, ser. ICSE '13, San Francisco, CA, USA: IEEE Press, 2013, pp. 702-711.
- T. o. Saar, M. Dumas, M. Kaljuve, and N. Semenenko, "Browserbite: Cross- browser testing via image processing," Software: Practice and Experience, vol. 46, no. 11, pp. 1459-1477, Nov. 2016.
- Applitools. (Apr. 2019). Ai powered visual ui testing & monitoring, [Online]. Available: https://applitools.com/.
- B. Slatkin. (Aug. 2016). Bslatkin/dpxdt, [Online]. Available: https://github.com/ bslatkin/dpxdt/.
- Percy. (Apr. 2019). Percy, [Online]. Available: https://percy.io/.
- Applitools. (Mar. 2019). Selenium java tutorial, [Online]. Available: https:// applitools.com/tutorials/selenium-java.html.
- E. Alégroth, A. Karlsson, and A. Radway, "Continuous integration and visual gui testing: Benefits and drawbacks in industrial practice, " in Proceedings of the 11th International Conference on Software Testing, Verification and Validation, ser. ICST '18, Vasteras, Sweden: IEEE, 2018, pp. 172-181.
- K. Moran, C. Watson, J. Hoskins, G. Purnell, and D. Poshyvanyk, "Detecting and summarizing gui changes in evolving mobile apps," in Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ser. ASE '18, Montpellier, France: ACM, 2018, pp. 543-553.
- M. Grechanik, C. W. Mao, A. Baisal, D. Rosenblum, and B. M. Hossain, "Differ- encing graphical user interfaces, " in Proceedings of the 18th IEEE International Conference on Software Quality, Reliability and Security, ser. QRS '18, Lisbon, Portugal: IEEE, 2018, pp. 203-214.
- T. A. Walsh, G. M. Kapfhammer, and P. McMinn, "Automated layout failure detection for responsive web pages without an explicit oracle, " in Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis, ser. ISSTA '17, Santa Barbara, CA, USA: ACM, 2017, pp. 192-202.
- Q. Xie, M. Grechanik, C. Fu, and C. Cumby, "Guide: A gui differentiator," in Proceedings of the 25th IEEE International Conference on Software Maintenance, ser. ICSM '09, New York, NY, USA: IEEE, 2009, pp. 395-396.
- ReTest. (May 2019). Retest/recheck, [Online]. Available: https://github.com/ retest/recheck/.
- --, (May 2019). Retest/recheck-web, [Online]. Available: https://github.com/ retest/recheck-web/.
- D. Kraus. (Apr. 2019). Beatngu13/visual-testing, [Online]. Available: https : //github.com/beatngu13/visual-testing/.
- S. McMaster and A. M. Memon, "An extensible heuristic-based framework for gui test case maintenance, " in Proceedings of the 2009 International Conference on Software Testing, Verification, and Validation Workshops, ser. ICSTW '09, Denver, CO, USA: IEEE, 2009, pp. 251-254.
- Selenium. (May 2016). Selenium webdriver, [Online]. Available: https://www. seleniumhq.org/projects/webdriver/.
- Wikipedia. (Aug. 2019). List of most popular websites, [Online]. Available: https://en.wikipedia.org/wiki/List_of_most_popular_websites.
- ReTest. (May 2019). Review, [Online]. Available: https://retest.de/review/.
- Z. Gao, Z. Chen, Y. Zou, and A. M. Memon, "Sitar: Gui test script repair, " IEEE Transactions on Software Engineering, vol. 42, no. 2, pp. 170-186, Feb. 2016.
- M. Ermuth and M. Pradel, "Monkey see, monkey do: Effective generation of gui tests with inferred macro events," in Proceedings of the 25th ACM SIG- SOFT International Symposium on Software Testing and Analysis, ser. ISSTA '16, Saarbrücken, Germany: ACM, 2016, pp. 82-93.
- Z. Gao, C. Fang, and A. M. Memon, "Pushing the limits on automation in gui regression testing, " in Proceedings of the 26th IEEE International Symposium on Software Reliability Engineering, ser. ISSRE '15, Gaithersbury, MD, USA: IEEE, 2015, pp. 565-575.
- K. Mao, M. Harman, and Y. Jia, "Sapienz: Multi-objective automated testing for android applications," in Proceedings of the 25th ACM SIGSOFT Interna- tional Symposium on Software Testing and Analysis, ser. ISSTA '16, Saarbrücken, Germany: IEEE, 2016, pp. 94-105.
- R. M. L. M. Moreira, A. C. R. Paiva, M. Nabuco, and A. M. Memon, "Pattern- based gui testing: Bridging the gap between design and quality assurance," Software: Testing, Verification and Reliability, vol. 27, no. 3, e1629, May 2017.
- J. A. Saddler and M. B. Cohen, "Eventflowslicer: A tool for generating realistic goal-driven gui tests, " in Proceedings of the 32nd IEEE/ACM International Con- ference on Automated Software Engineering, ser. ASE '17, Urbana-Champaign, IL, USA: IEEE Press, 2017, pp. 955-960.
- W. Song, X. Qian, and J. Huang, "Ehbdroid: Beyond gui testing for android applications, " in Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering, ser. ASE '17, Urbana-Champaign, IL, USA: IEEE Press, 2017, pp. 27-37.
- E. Alegroth, M. Nass, and H. H. Olsson, "Jautomate: A tool for system-and acceptance-test automation, " in Proceedings of the 6th IEEE International Confer- ence on Software Testing, Verification and Validation, ser. ICST '13, Washington, DC, USA: IEEE Computer Society, 2013, pp. 439-446.
- E. Borjesson and R. Feldt, "Automated system testing using visual gui testing tools: A comparative study in industry, " in Proceedings of the 5th IEEE Interna- tional Conference on Software Testing, Verification and Validation, ser. ICST '12, Montreal, QC, Canada: IEEE Computer Society, 2012, pp. 350-359.
- T. Yeh, T.-H. Chang, and R. C. Miller, "Sikuli: Using gui screenshots for search and automation," in Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology, ser. UIST '09, Victoria, BC, Canada: ACM, 2009, pp. 183-192.
- E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, "The oracle problem in software testing: A survey, " IEEE Transactions on Software Engineering, vol. 41, no. 5, pp. 507-525, May 2015.