OwlEye

A tool to automatically detect and localize UI display issues in the screenshots of the application under test.

We give dataset (./Dataset/), UI display issues detection and localization methods (./OwlEye/), data augmentation methods (./DataAug/).

Dataset (./Dataset/)

The experimental dataset comes from two sources. The first is the screenshots from crowdtesting, which contains 4,470 non-duplicate screenshots with UI display issues and equal number of bug-free non-duplicate screenshots.The second is the screenshots generated with the data augmentation method.

TrainData

./Dataset/traindata/ is the training dataset of the experiment, with 21,980 images.

• bug-crowd.rar : 3,170 screenshots with UI display issues from the crowdtesting dataset.
• bug-aug.rar : 7,820 screenshots with UI display issues generated by data augmentation method.
• normal-crowd.rar : 3,170 screenshots without UI display issues from the crowdtesting dataset.
• normal-aug.rar : 7,820 screenshots without UI display issues generated by data augmentation method.

TestData

./Dataset/test.rar is the testing dataset of the experiment, with 1,600 images.

• 1.jpg ~ 800.jpg : 800 screenshots with UI display issues from the crowdtesting dataset.
• 1001.jpg ~ 1800.jpg : 800 screenshots without UI display issues from the crowdtesting dataset.

ValData

./Dataset/val.rar is the Val dataset of the experiment, with 1,000 images.

• 1.jpg ~ 500.jpg : 500 screenshots with UI display issues from the crowdtesting dataset.
• 1001.jpg ~ 1500.jpg : 500 screenshots without UI display issues from the crowdtesting dataset.

Issues Detection and Localization Approach (./OwlEye/)

OwlEye is named considering it is like the owl's eye to effectively spot UI display issues, also considering this nocturnal animal serves as the supplement of traditional automated GUI testing. Automatically detect and localize UI display issues based on the screenshots of the mobile apps under test.

• getdata.py : Preprocessing dataset
• network.py : Neural network structure of the model
• train.py : Training model
• test.py : Testing model
• val.py : Validating the model
• localization.py : UI display issues localization methods
• ./data/ : Decompress the dataset and put it into their respective folders (./train/ & ./test/ & ./val/)
• ./model/ : Model file
• ./record/ : Training record
• ./input_pic/ : Screenshot to locate
• ./output_pic/ : Localization result

Heuristic-based Data Augmentation (./DataAug/)

It is a heuristics-based training data augmentation method which can automatically generate screenshots of UI display issues with normal screenshots.

• DataAug.py : The heuristics-based training data augmentation method
• ./picture/ : Normal screenshots
• ./json/ : Run-time view hierarchy of screenshots
• ./IMG/ : Pre-prepared image icon
• ./output/ : Generated screenshots with UI display issues

Confirmed or fixed issues (RQ3)

We evaluate the usefulness of our model by applying our OwlEye in detecting the UI display issues in the real-world apps from Google Play and F-Droid. Among 2,200 apps, we find that 56 of them are with UI display issues. We issued bug reports to the development team and 25 are confirmed and fixed by developers.

APP Name	Version	Package Name	Category	APKSource	Download	Issueid	Status
Open Food Facts	3.2.8	openfoodfacts.github.scrachx.openfood_328	Health	F-droid	500K+	#3051	confirmed
Transdroid	2.5.17	org.transdroid.full_237	Tool	F-droid	100K+	#542	confirmed
Linphone	4.2.3	org.linphone_4230	Communication	F-droid	500K+	#965	confirmed
NewPipe Legacy	0.18.6	org.schabi.newpipe_870	Media	F-droid	8K+	#24	fixed
CEToolbox	1.3	com.github.cetoolbox_5	Medical	F-droid	500+	#4	confirmed
Yucata Envoy	0.5.3	kaba.yucata.envoy_503	Tool	F-droid	N/A	#3	confirmed
MTG Familiar	3.6.4	com.gelakinetic.mtgfam_70	Utilities	F-droid	500K+	#512	fixed
OpenTracks-OSM	1.5.0	de.storchp.opentracks.osmplugin_7	Health	F-droid	10+	#26	fixed
LessPass	9.1.2	com.lesspass.android_90102	Productivity	F-droid	5K+	#519	fixed
ClassyShark3xodus	1.0-12	com.oF2pks.classyshark3xodus_12	Tool	F-droid	N/A	#3	confirmed
VlcFreemote	1.10	com.nicolasbrailo.vlcfreemote_10	Media	F-droid	N/A	#24	confirmed
Transistor	3.2.4	org.y20k.transistor_71	Music	F-droid	10K+	#254	fixed
Enhanced Controller for Onkyo and Pioneer	1.10	com.mkulesh.onpc_23	Music	F-droid	10K+	#138	fixed
DemocracyDroid	3.7.1	com.workingagenda.democracydroid_43	News	F-droid	10K+	#51	confirmed
ApowerMirror	1.5.9.34	ApowerMirror Mirror Control_v1.5.9.34_apkpure.com	Tool	Google	5M+	email	confirmed
Degoo Cloud Storage	1.57.7	Degoo Cloud Storage_v1.57.7.200316_apkpure.com	Tool	Google	10M+	email	fixed
Deezer Music Player	6.1.22.49	Deezer Music Player Songs Playlists Podcasts_v6.1.22.49_apkpure.com	Music	Google	500K+	email	fixed
MediaFire	4.2.2	MediaFire_v4.2.2_apkpure.com	Productivity	Google	5M+	email	confirmed
Secure VPN	2.2.5	Secure VPN – A high speed ultra secure VPN_v2.2.5_apkpure.com	Tool	Google	10M+	email	confirmed
Nox security	1.7.6	Nox Security Antivirus Master Clean Virus Free_v1.7.3_apkpure.com	Tool	Google	10M+	email	fixed
Music Player	3.8.8	Music Player Audio Player_v3.8.8_apkpure.com	Music	Google	50M+	email	confirmed
Proxynel	2.2.9	Proxynel Unblock Websites Free VPN Proxy Browser_v2.29_apkpure.com	Tool	Google	10M+	email	confirmed
Postegro	1.23	Postegro Any Profile Viewer_v1.23_apkpure.com	Communication	Google	500K+	email	fixed
Perfect Piano	7.5.1	Perfect Piano_v7.5.1_apkpure.com	Music	Google	50M+	email	confirmed
Paytm	2.14.1	Paytm Canada_v2.14.1_apkpure.com	Finance	Google	100K+	email	confirmed
Thunder VPN	3.1.12	Thunder VPN A Fast Unlimited Free VPN Proxy_v3.1.12	Tool	Google	10M+	email	confirmed