Introduction
Testing is an important part of the application development process. For Android, it is particularly important because the devices are very different from each other in the following ways:
- Screen size and resolution
- Android version
- Form factor
- Instruction set of the processor
- The presence of the front camera, NFC, external keyboard, etc.
Your Android application should be tested on many devices.
Application testing processes include many types of testing. Let's take a look at manual functional testing. The tester needs to carefully check all the functionality and reset the device to an initial state. The tester repeats these steps for each application and each device. Done manually, this process is very time-consuming.
Automated functional testing can be performed regularly without additional costs. For example, test a build every night on all devices, analyze the results in the morning, and fix the bugs.
In this article I will review several tools for automated functional testing. I'll review only tools included in Android SDK or distributed under Open Source license.
The Concept of Automated Testing
Our goal is to automate actions that are performed manually for maximum precision. Let's review these actions. We'll use several applications and several Android devices.
For each application and each device we need to perform the following steps:
- Install the application on the device
- Launch the application
- Test the application using the selected method
- Remove the application
- Reset the device to the initial stage
At each step you need to collect and analyze data such as logs and screenshots. Below, we’ll discuss the tools that automate these steps.
Control Android devices
First, you need to select the computer that will run the automated test and install the Android SDK on it. I'll use a desktop computer with Linux* as an example. You need to disable lock screen and increase "time before sleep mode" to the maximum level on each device.For some testing methods, you need to disable screen orientation change.
There are two utilities in the Android SDK to control Android devices: ADB and monkeyrunner*. I'll describe in detail how to automate actions that are used in manual testing.
Control the Android device using ADB
ADB (Android Debug Bridge) is a command line tool to control Android devices. The ADB home page is: http://developer.android.com/tools/help/adb.html
The ADB tool is located in the directory <android_sdk>/platform-tools/.You should put this directory in the PATH environment variable.
Checking the ADB installation
Install and configure the Android SDK, then connect Android devices to your computer and run the command:
adb devices
This command will list all plugged-in Android devices. If the list of devices is not empty, then ADB is working properly.
Working with multiple devices
You should use "-s" parameter to specify which device ADB should use.
adb -s [serial_number] [command]
For example:
adb -s [serial_number] logcat
The serial number of the device can be obtained from the output of «adb devices» command. Parameter -s allows you to work simultaneously with multiple connected devices.
Basic ADB commands
Open a console on the device:
adb shell
Run a command on the device:
adb shell [command]
Many standard Linux utilities are included in Android: ls, cat, dmesg, ...
Install the application from the apk file:
adb install example.apk
Uninstall the application:
adb uninstall [package]
Get a package name from the apk file:
aapt dump badging example.apk | grep "Package"
Download a file from the device to the computer:
adb pull [path-on-device] [file]
Upload a file from the computer to the device:
adb push [file] [path-on-device]
Note:
Only read access is allowed to most directories on the Android device. Write access is allowed to /sdcard (but you can’t run programs from this directory) and /data/local/tmp.
Start an application:
adb shell am start -n [package]/[activity]
Run the specified activity.
You can extract the activity name from the apk file:
aapt dump badging example.apk | grep "launchable-activity"
Reading logs
Logcat is a command that reads logs from Android devices.
Logcat homepage:http://developer.android.com/tools/help/logcat.html
Read logs from the device (blocks until you press Ctrl-C):
adb logcat
Clear log buffer on the device:
adb logcat -c
Dump log buffer on the device (shows current buffer contents, non-blocking):
adb logcat -d
Example:
adb logcat -c # clear the buffer log
# Action
adb logcat -d > file.log # save the current contents of the log buffer to file.log
Capture screenshot using screencap
The screencap utility saves the current contents of the screen to a graphic file:
adb shell screencap /sdcard/screen.png
adb pull /sdcard/screen.png screen.png
adb shell rm /sdcard /screen.png
The screencap utility is available on phones with Android 4.x and above. On previous Android versions you can capture screenshots using monkeyrunner.
BASH script to test the application using ADB
Script: app_test.sh
Control the Android device with MonkeyRunner
The monkeyrunner tool provides APIs for scripts that control Android devices. You can write a Python* script with monkeyrunner that installs an Android application, launches it, emulates user actions, takes screenshots, and saves them to your computer. Monkeyrunner uses Jython* to run scripts.
The monkeyrunner home page and API reference: http://developer.android.com/tools/help/monkeyrunner_concepts.html
Reading logs with monkeyrunner
File log.py:
# coding: utf-8
from com.android.monkeyrunner import MonkeyRunner, MonkeyDevice
def log(fn, device):
msg = device.shell('logcat -d')
f_log = open(fn, 'at')
if msg is None:
msg = 'None'
f_log.write(msg.encode('utf-8'))
f_log.close()
device.shell('logcat -c')
if __name__ == '__main__':
device = MonkeyRunner.waitForConnection()
device.shell('logcat -c') # Clear logs buffer
# ...
log('example.log', device) # Write logs
Starting:
monkeyrunner log.py
The script will write logs to a file example.log in the current directory.
Capture screenshot using MonkeyRunner
File screen.py:
# coding: utf-8
from com.android.monkeyrunner import MonkeyRunner, MonkeyDevice
if __name__ == '__main__':
device = MonkeyRunner.waitForConnection()
image = device.takeSnapshot()
image.writeToFile('screenshot.png','png')
Starting:
monkeyrunner screen.py
The script takes a screenshot and saves it to a filescreenshot.png in the current directory.
Example of device control using monkeyrunner
Script: monkeyrunner_test.py
Starting:
monkeyrunner monkeyrunner_test.py
Automated Testing Methods
Testing with Monkey*
Imagine that the device you are testing is given to a very active and creative monkey—the Monkey tool is designed to simulate this situation.The Monkey tool, a part of the Android SDK, sends a stream of random user events.Command line parameters specify the number of user actions, the ratio of each event type, and a name of a package (so Monkey will not go beyond the limits of the tested application and will not start sending SMS to all the contacts in the Address Book).
Examples and a list of parameters are on the Monkey home page: http://developer.android.com/tools/help/monkey.html
The main advantage of the Monkey tool is zero maintenance costs. Furthermore, stress testing can detect non-trivial bugs.
Disadvantages of testing with the Monkey tool:
- Monkey can’t simulate complex workloads such as authentication. In such cases, application functionality remains untested.
- Games with complex control that require quick reactions and complex gestures will be completed at the beginning, or do not begin.
- It’s very difficult to reproduce errors found by Monkey.
- Monkey doesn’t check application status during the test.
Automated testing with Monkey is a good starting point for any application. It is possible that this method will show adequate results for a particular application.If the testing quality is low, you should use other testing methods.
Testing with MonkeyRunner
You can not only develop Android device control scripts using MonkeyRunner, you can also write scripts to test the application on a specific device.
Advantages
Disadvantages:
- The complexity of writing scripts, even in simple cases.
Developing monkeyrunner scripts takes a lot of time, so this method is usually not justified.However, in special cases, this method might work.
Testing with getevent and sendevent
Getevent and sendevent utilities allow the user to record a sequence of events and replay the sequence.Root permissions are not required to run these tools.
Advantages
- Event sequences can be recorded without additional costs in manual testing, if it is carried out.
- Programming skills are not required to record an event sequence.
Disadvantages:
- Sequence should be recorded separately for each application and each device.If you change the interface of an application, all recorded actions should be redone.
- This method doesn’t check application status during the test. If application response is delayed (for example, web page loading), the testing results will be incorrect.
- Rapid and complex sequences will be played longer than recorded. So it is not always suitable for testing dynamic games where response time is critical.
Record the sequence of events:
# Record event sequence
# Do actions on the device, press Ctrl-C to finish
adb shell getevent -t > events.txt
# Convert event sequence to script
./decode_events.py events.txt > events.sh
# Load the script to the device
adb push events.sh /data/local/tmp/
# Set the permissions
adb shell chmod 755 /data/local/tmp/events.sh
# Run script
adb shell sh /data/local/tmp/events.sh
Script: decode_events.py
Replay the recorded sequence of events on the device.
Testing with Robotium*
Robotium is not a part of Android SDK, and it’s distributed under the Open Source license. The Robotium home page is: http://code.google.com/p/robotium/.
Robotium scripts define actions on the application UI level rather than the input device level.
For example, a script needs to tap on an «OK» button. The monkeyrunner script will be implemented as "Tap at the screen point (x0, y0)". A Robotium script will be implemented as "Press the button with the text "OK" ."
When actions are described at the interface level, the testing script can be made independent of interface layout, screen resolution, and orientation.
Additionally, Robotium allows you to check the application response to the action.For example, after clicking the «OK» button, the list with the "Item 1" item should appear.You can check list element names with Robotium. If you check the application state after each step, it is easy to find at which step the error occurred.
Disadvantages
- You need to develop a test script in Java* for each application. This requires programming skills and time.
- When the application interface changes, the event sequence must be redone.
- It’s harder to write Robotium scripts compared to recording events using getevent / sendevent.
In general, Robotium allows you to develop the highest quality test cases with adequate cost.
Testing methods comparison
| Testing method |
Advantages |
Disadvantages |
|
Monkey - a stream of random user actions
|
No maintenance costs.
Independent of device.
Stress testing can detect non-trivial errors.
|
Quality of testing varies from application to application.
Bug reports are hard to reproduce.
Monkey doesn’t check the state of the application during testing.
|
|
monkeyrunner – device control script
|
Flexibility
|
The complexity of scripting, even for simple applications.
|
|
getevent/sendevent - record/replay user actions
|
Programming skills are not required to record event sequence.
|
Recorded sequence of actions will only fit one device at a fixed screen orientation.
When the application interface changes, the event sequence must be redone.
This method doesn’t check state of application during testing.
|
|
Robotium - test script API to verify the status
|
Actions are described at the application UI level.
Script may be independent of the screen resolution and screen orientation.
Script can check application state after an action.
|
The complexity of writing scripts in Java.
If you change the application interface, you will need to modify the script.
|
Result analysis
Now we need to analyze the logs and screenshots collected during the automated testing process for errors.
Log analysis
You can do a search for strings:
- I/DEBUG
- FATAL EXCEPTION
- WIN DEATH
This list can be extended with error messages that were found during manual testing.
Screenshot analysis
You can prepare a series of screenshots at key moments of testing and compare them to the screen content during automated testing. This will determine if the automated testing process is running correctly.
It's useful to compare the initial screenshot with the screenshot after the application launches. This helps to detect cases when the application silently fails.
Monkeyrunner allows you to compare two screenshots with the specified tolerance in percents:
image1 = device.takeSnapshot()
# ...
image2 = device.takeSnapshot()
if image2.sameAs(image1, 0.1):
print 'image1 and image2 are the same (10%)'
Unfortunately, there is no MonkeyImage API to load an image from a file. You can write a custom function to compare images using, for example, the Python* Imaging Library.
Reset the device to the initial state
Devices should be returned to the initial state after testing.This can be achieved in several ways:
- Press the «Back» button multiple times.
- Reboot the device.
- Restart the zygote process.
The first option is usually the most appropriate.
Press the Back button multiple times
Press the "Back" button using monkeyrunner:
for i in xrange(0, 10):
device.press('KEYCODE_BACK', MonkeyDevice.DOWN_AND_UP)
time.sleep(0.5)
In practice, this is a good option because it emulates the behavior of a real user.
Conclusion
In this article, we covered some methods of automated testing for Android applications. We reviewed advantages and disadvantages of automated testing methods.
We also discussed the Monkey and monkeyrunner tools included in the Android SDK and the Robotium tool.
Automated testing does not replace other types of testing.A properly constructed testing process that combines different testing methods including automated testing is an essential part of a high quality application development process.
Other Related Articles and Resources
