If you are testing a mobile app you are probably not sitting behind a desk with all your standard reporting tools and you don’t have all the test cases writing down. So how can you collect your mobile app test results?
Goal
My goal when testing is to give insight into where the defects are and make that as clear to all the stakeholders. By giving this information the possible risks can be evaluated. When testing I can’t explain all the steps I take on a mobile device? I could of course prepare all my test case during a test analysis period and during execution refer to these steps. I try to add print screens and movies to the test results to help a developer finding solutions for the found defects.
Less then ideal
The first problem I have is that the design and the software are developed at the same time (Agile/scrum team), so there is no separate preparation period. The second problem is that I usually can’t prepare every test case in detail because the number of situations and the relation between all the variables are far to complex to grasp in simple steps. How can I verify a defect on one device to all the other devices. Beside the platform and manufacturer differences there are many more factors that determine if there is a defect and/or the defect to manifest. Think about type of network and speed, location, movement, number of apps running, amount of free memory, who is executing the test, his or her emotional state. And there are many more.
One versus a group
If you are testing an app or part of the app on a group then the results need to be clustered and generalized. This is more a process a and quality item then a tooling or statistical issue. I would organize sessions with the group, slice the app and prepare tours with different perspectives. Look for more info about tours at Cem Kaner article. A tours focus on the tasks a user can perform with the app. Use a technique like equivalence class to spread the setup over the tours.
Setup
After a product risk analysis the chosen priority and area’s become clear. These results is translated by me into a setup. This setup is used during the (preparation) of the test cases/tours. An example of a setup is shown below. Every column in the table contains the chosen variation.
Devices | Users | Network type | Motion | Tours | Perspectives | Platforms | Sprints |
iPhone 5s – 8.0.2 | Product owner | offline | Binnen – zitten | App in its life time | User Scenario’s | iOS | 1 |
iPhone 5s – 8.1 | Interne tester | wifi | Binnen – lopen | Change your mind | Interruptions and interactions | Android | 2 |
iPhone 6 – 8.1 | Persona – Daan | 3G | Buiten – lopen | Comparison | Function | Cross platform | 3 |
iPhone 6p – 8.1 | Persona – Lieke | 4G | Buiten – rijden | Connectivity | Store submission | 4 | |
HTC One M7 – 4.4.3 | Externe gebruiker | Multiple | Feature | Data | 5 | ||
Nexus 5 – 5.0.1 | FedEx or CRUD Tour | Network conditions | 6 | ||||
Sony Xperia Arc HD – 4.1.2 | Guidebook or Competitor tour | Communication |
Tooling
A developer needs all the context you can report to create a solution. Here is a list of the tools that I use to record the context of the defect:
– Product risk analysis
– A test setup
– (logical) Test cases on high level
– Multiple devices with SIM cards and a data bundle
– Multiple persons that execute tests
– Test data in all sort and flavors that can be created or searched for on the spot
– software to report, record and analyse like Xmind for test ideas and coverage, Mobizen and Reflector for mirroring, Shou screen recording, Postman plugin for API analysis, ADB and iTools for log files, Jira for defect reporting, Hip chat for informal communication.
Before I report a defect I need a reference situation to compare my situation to. Usually this is the functional perspective. From this point you start varying using the test setup, experience, hints for the team and your feeling.
Conclusion
Reporting the test results is more than using an application making movies. I like the combination of high level preparation and exploratory testing. I’m in direct contact with the developers and can create my own test data. I have a fysical test lab, test in side and out side. I use multiple devices and my laptop so I can repeat the scenario, analyse and compare the results.
Links
For note taking I’m still looking into iTester app but this is only available on iOS. More info about note taking and exploratory testing:
- http://thesocialtester.co.uk/explaining-exploratory-testing-relies-on-good-notes/
- http://university.utest.com/note-taking-tools-exploratory-testing/
- http://altom.ro/itester#SBTM
- http://ios.wonderhowto.com/how-to/record-your-ipads-iphones-screen-no-jailbreak-computer-required-0156864/ and https://emu4ios.net/
- http://kaner.com/?p=96