Narrative-Based Assessment of Quality in Context

 

The goal of software testing is to qualitatively research, evaluate and describe quality and communicate it as information. This information is in the form of stories, or the who, what, where, when, why and how of value. Stories can also serve hypotheses of delivered or threatened value for testing. Telling stories to stakeholders allows them to understand quality of software in the best, most truthful way possible and make the best business decisions.

The Testing and Quality Story is the result of testing. You can never know the “actual” quality of a software product— you can’t “verify” quality, as such— but by performing tests, you can make an assessment, and that takes the form of a story you tell (including bugs, curios, etc.) (Bach, 2020)

 

Events

In the same way calendar events contain a time, a location and people for a particular purpose (value), software quality is delivered (or lost) in events of time, value and people too. So each story is an assessment of quality known as an event. There are an infinite number of ways to evaluate and describe quality so there are an infinite number of stories that can be told. The goal is to identify and tell the important stories that matter. What makes testing extra challenging is these stories exist in time making the stories temporary. A prosthetic leg doesn’t deliver any value whilst it’s not being used. It may not deliver any value when it’s being used for a task other than walking. A messaging application loses all its value when everyone is gathered together in person, but becomes a lot more useful when everyone is remotely communicating. Context of can change any time which means stories come and go making the quality relationship dynamic.

 

Like any story, it’s made up of various aspects that represent important contextual factors, both objective and subjective. Below is what that story structure may look like based on previous definitions, and items from the Heuristic Test Strategy Model (HTSM) (Bach, 2020):

Quality in a software product, system or service can be evaluated, described and communicated as instances of: value in some way to someone who matters by having their expectations met or exceeded at some point in time via some method of interaction (or lack of interaction) with a strong product element (an exploited asset) and identified by a positive emotion (or lack of negative emotion). Lack of quality in a software product, system or service can be evaluated, described and communicated as instances of:: little value or harm in some way to who someone matters by not having their expectations met or exceeded at some point in time via some method of interaction (or lack of interaction) with a weak product element (an exploited vulnerability) and identified by a negative emotion (or lack of positive emotion)   Value (or harm) in some way is primarily the different dimensions of quality and the associated benefit or risk, such as its ability (or inability) as a social prosthesis or tool. Value and harm can also be evaluated as time or cost (monetary worth) or perceived importance (or insignificance). Someone who matters are stakeholders, the subjects of the subjective quality, or those affected by quality in a way that is cared about by someone else. Expectations met or exceed describes consistencies (or inconsistencies) between related things, actual experiences of the software compared to expected experiences from mental models At some point in time means all this can change from moment to moment and is not fixed. Some method of interaction are the different things a user or other stakeholder does (or doesn’t do) with the software to get that value, loss or harm. Strong/weak product element the part(s) of the software or project that the method of interaction is with Identified by an emotion or lack of emotion, the feeling that the stakeholder experiences, especially if it’s born from a cognitive dissonance.    As project quality is linked to product quality, the story can further be expanded to identify the root cause of any delivered or threatened value. This consequence is due to a decision (or lack of a decision) made by a project team member driven by a strong or weak project process. In a story of risk, the latter is referred to as a “cause”, the likelihood of which increase with certain problem drivers (van Daele, 2017)*(Kaner, Bach & Pettichord, 2002a). This not only allows stakeholders to make decisions on the quality of the software, but also allows project team members and managers to make decisions on the quality of the project processes that are linked to it. For testers, this means also reporting on the quality of testing itself and the software project (Bolton, 2012). *Note van Daele’s story of risk refers to “events”, but rather than an instance of the entire quality story described here, these are analogous to “methods of interaction” only. “Result” in their same story refers to the “lost value or harm” described here (the consequence).

Examples

• A tax calculation program could have an oversight that allows division by zero (vulnerability) that if encountered (exploited) by an accountant (someone who matters) inputting numbers (method of interaction) for a tax calculation (value benefit) could cause an error (empirical observation). This is a functional bug (dimension) because it risks someone not being frustrated (feelings) at not being able to complete their tax calculations for a client on time (lost value or harm).
• A shopping website could be designed in such a way so that it causes a bottleneck during peak usage (vulnerability) that if encountered (exploited) by a customer (someone who matters) trying to click on link (method of interaction) to buy some clothes (value benefit) is prevented from doing so due to a persistent busy circle (empirical observation). This is a performance bug (dimension) because it risks someone becoming impatient (feeling) and leaving to buy their clothes from a competitor (lost value or harm).
• A government’s management information system (MIS) has doesn’t sanitise one of its data input fields  (vulnerability), that if encountered (exploited) by a malicious user (a disfavoured stakeholder), could allow a database injection attack (method of interaction) resulting in unauthorised access (empirical observation). This is a security bug (dimension) because it risks personal or financial data being stolen (loss or harm).

 

User Stories

Initial claims contain some contextual information that can feed into stories of quality. User stories were coined as part of extreme programming by allowing discussions around what users would actually do and described in such a way that allows simple understanding and communication (Beck, 1999). The user story was refined into three simple aspects that represent who, what and why of the quality story respectively (Davis, 2001) (Agile Alliance). A popular example of user stories contains a type of user (some person who matters) who wants to achieve a goal (method of interaction) so that they can achieve a benefit (value in some way, particularly capability):

As a <type of user>, I want <some goal> so that <some reason>

(North, 2007), (Cohn, 2008)

While everyone else in a project is focused on stories of value and benefit with user stories or other claims of value, no one is really considering the opposite. Testers real value in a project comes from researching and evaluating lost value, harm and risk. To assist in this process, testers can construct “anti-user stories” or counter claims of value. The “user” is the same but story describes the goal not being met and the consequence of lost value or harm:

As a <type of user>, I can’t achieve <some goal> which leads to <lost value/harm>

As with the user stories of value and benefit, testers can further expand these to add more contextual factors from the quality story to give weight to bugs or serve as hypotheses to test.

Summary Quality is evaluated and communicated as series of stories known as events. The events contain context on the who, what, where, when, why and how of delivered or threatened value. Testers identify, evaluate, describe and communicate the most important stories to stakeholders in a way they can easily understand and make business decisions on.

 

References

• Agile Alliance, User Story Template. [online] Glossary. Available at: Link
• Bach, J., 2020. Heuristic Test Strategy Model. Version 5.7.5 [online] Satisfice. Available at: Link
• Beck, K., 1999.  Extreme Programming Explained: Embrace Change. 1st ed. Boston: Addison-Wesley
• Bolton, M., 2012. Braiding The Stories (Test Reporting Part 2) [online] DevelopSense. Available at: Link
• Cohn, M., 2008. Advantages of the “As a user, I want” user story template. [online] Mountain Goat Software. Available at: Link, see also: Link
• Davis, R., 2001. XP2001 Conference (see Agile Alliance reference)
• Kaner, C., Bach, J. and Pettichord, B., 2002. Lessons Learned In Software Testing: A Context Driven Approach. 1st ed. New York: Wiley, p.61
• North, D., 2007. What’s in a Story? [online] Dan North & Associates Ltd. Available at: Link
• van Daele, B., 2017. RiskStorming …A cheesy name for a cool Risk focused format! [online] Slides: RiskStorming shared. p.8. Available at: Link

Revised and updated: 16/10/2021