Skip to content

GUI testing

Jump to bottom
Antti Kervinen edited this page Oct 10, 2016 · 22 revisions

fMBT provides utilities for GUI testing:

  • Python libraries that implement similar GUI test interface for multiple platforms
    • fmbtandroid for Android phones, tablets and emulator
    • fmbtchromiumos for Chromium OS devices
    • fmbttizen for Tizen mobile and emulator (X server) and Tizen IVI (Wayland + Weston)
    • fmbtvnc for anything that runs VNC servers
    • fmbtwindows for Windows desktops, laptops and tablets
    • fmbtx11 for X servers
  • fmbt-scripter - a tool for
    • capturing reference bitmaps from screenshots from Android, Chromium OS, Tizen, VNC, Windows and X
    • editing and debugging Python GUI test scripts.

GUI test interface

GUI test interfaces for multiple platforms have a lot in common, but also some platform-specific extensions.

While this document gives only an overview, you will get full parameter and return value documentation for each GUI test interfaces with Python:

  • python -c 'import fmbtandroid; help(fmbtandroid.Device)'
  • python -c 'import fmbtchromiumos; help(fmbtchromiumos.Device)'
  • python -c 'import fmbttizen; help(fmbttizen.Device)'
  • python -c 'import fmbtvnc; help(fmbtvnc.Screen)'
  • python -c 'import fmbtwindows; help(fmbtwindows.Device)'
  • python -c 'import fmbtx11; help(fmbtx11.Screen)'

If you want help from a single method, use python -c 'import fmbtandroid; help(fmbtandroid.Device.refreshScreenshot)', for instance.

Receiving screen contents

  • refreshScreenshot fetches the latest screenshot from the device under test. <act>Bitmap and <act>OcrText methods (like tapBitmap, swipeOcrText) use most recently fetched screenshot.
  • refreshView fetches UI elements currently on the screen. <act>Text methods (like tapText) use most recently fetched UI element information.
  • Screenshots are automatically stored in the "screenshots" directory under the working directory, unless something else is specified with setScreenshotDir and setScreenshotSubdir.
  • Disk space needed by screenshots can be controlled with setScreenshotLimit and setScreenshotArchiveMethod.
  • screenshot().save(filename) saves the latest screenshot to given file.
  android chromiumos tizen vnc windows x11
refreshScreenshot Y Y Y Y Y Y
refreshView Y       Y Y

Verifying screen contents

  • verifyBitmap and verifyOcrText return True if a bitmap or a text is recognized from latest screenshot.
  • screenshot().findItemsByBitmap and screenshot().findItemsByOcr return full list of items matching a bitmap or a text. Full documentation on OIR and OCR parameters can be found with help(obj.oirEngine()) and help(obj.ocrEngine()).
  • verifyText returns True if a UI element with given text is found from latest UI element information.
  • view().findItemsByClass, view().findItemsById, view().findItemsByText return full list of UI elements matching given criteria.
  • Items found with findItemsBy... can be acted upon with tapItem and swipeItem.
  android chromiumos tizen vnc windows x11
verifyBitmap verifyOcrText Y Y Y Y Y Y
verifyText Y       Y Y
screenshot Y Y Y Y Y Y
view Y       Y Y
ocrEngine Y Y Y Y Y Y
oirEngine Y Y Y Y Y Y

Sending touch or mouse input

  • tap<what> sends touch down and touch up (or mouse button down, mouse button up) events on given location. The location can be given as absolute coordinates or unity coordinates, or it can be based on a bitmap, text recognized by OCR, or a text from UI elements.
  • platforms that support both touch and mouse events will get a touch event by default. When called with optional argument button=X, a mouse event with button X is sent. If button=0, then only mouse move is sent, resulting in hovering over given location.
  android chromiumos tizen vnc windows x11
drag touch mouse touch mouse mouse touch mouse mouse
swipe, swipeBitmap, swipeOcrText touch mouse touch mouse mouse touch mouse mouse
swipeText touch       touch mouse  
tap, tapBitmap, tapOcrText touch mouse touch mouse mouse touch mouse mouse
tapText touch       touch mouse  

Sending keyboard input

  • pressKey sends a key press and release.
  • type types given text.
  • keyNames returns list of key names supported by pressKey.
  android chromiumos tizen vnc windows x11
keyNames Y     Y Y  
pressKey Y Y Y Y Y Y
type Y Y Y Y Y Y

Executing commands remotely

  • shellSOE executes shell command and returns triplet: (Status, Output, Error), that is
    1. child process exit status
    2. what was printed to standard output
    3. what was printed to standard error
  android chromiumos tizen vnc windows x11
shellSOE Y Y Y   Y  

Logging test runs

  • enableVisualLog starts tracing all GUI test interface method calls, return values, exceptions, captured screenshots and reference bitmaps. The log is written to given HTML file during the test run.
  • fmbtlogger.text and fmbtlogger.csv log method calls, return values and exceptions of any Python object, including instances of GUI test interfaces. The log format is plain text or CSV. See python -c 'import fmbtlogger; help(fmbtlogger.text)' for more information.
  android chromiumos tizen vnc windows x11
enableVisualLog Y Y Y Y Y Y

fmbtandroid

Prerequisites:

  • adb from Android SDK (platform tools)
  • Enable USB debugging from device settings.

Example:

import fmbtandroid
d = fmbtandroid.Device()
d.swipe((0.5, 0.85), "east") # open lock screen

Connect using fmbtandroid.Device(uiautomatorDump=True) in order to use refreshView() with an unrooted device.

fmbtchromiumos

Prerequisites:

  • Enable ssh login to the Chromium OS device without password.
  • Python on the device.

Steps for Qemu:

  • Launch Qemu with port forwarding, for instance ... -net nic,model=virtio -net user,hostfwd=tcp::9222-:22.
  • Press Ctrl-Alt-T in the browser. When in crosh> prompt, enter shell.
  • Copy your ssh public key: scp [email protected] 'cat .ssh/id_rsa.pub' > ~/.ssh/authorized_keys.

Steps for Chromebooks:

Chromium OS in Qemu example:

import fmbtchromiumos
d = fmbtchromiumos.Device("ssh -p 9222 chronos@localhost")
print d.shellSOE("echo Welcome, $(whoami); pwd")[1]

fmbttizen

Prerequisites for Tizen device:

  • python. (See issue #18 for installing Python on Tizen mobile. Python is already installed on Tizen IVI systems.)

Prerequisites:

  • sdb from Tizen SDK for testing a Tizen mobile device. (You may need to execute sdb root on on the host, see issue #27.)
  • ssh for testing Tizen IVI

Tizen mobile example:

import fmbttizen
d = fmbttizen.Device()
d.pressPower()

Tizen IVI example:

import fmbttizen
d = fmbttizen.Device(loginCommand="ssh root@ivi")
d.refreshScreenshot().save("current-screen.png")

fmbtvnc

Prerequisites:

Example:

import fmbtvnc
d = fmbtvnc.Screen("IP-ADDRESS-OF-VNC-SERVER")
d.tap((0.5, 0.5), button=0) # hover mouse on the middle of the screen

fmbtwindows

Prerequisites for Windows device:

  • Install Python 2.X.
  • Copy fMBT's pythonshare directory to Windows.
  • In the pythonshare directory, run c:\python27\python.exe setup.py install
  • Launch pythonshare-server to which fmbtwindows can connect to:
cd \python27\scripts
python pythonshare-server --interface=all --password=xxxxxxxx

Example:

import fmbtwindows
d = fmbtwindows.Device("IP-ADDRESS-OF-THE-DEVICE", password="xxxxxxxx")
d.type("Hello Windows")

fmbtx11

Prerequisites:

  • Install pyatspi and enable accessibility (gsettings set org.gnome.desktop.interface toolkit-accessibility true) if you need toolkit data in your test (methods like refreshView(), verifyText(), and tapText()).

Example:

import fmbtx11
d = fmbtx11.Screen(":2")
d.enableVisualLog("calls.html")
d.refreshScreenshot()
Clone this wiki locally