Index

Complete list of all articles organised by category.

Fundamentals

Fundamental concepts of software testing and quality as applied research and a social science.

  • Science & Testing - Testing is applied research into software quality using scientific approaches and evidence to help stakeholders make informed decisions
  • Computer Science & Software Engineering - Testing isn't algorithms, problem solving, solution building or process management but a knowledge of both is important for testing
  • Quality - Defined as value to some person(s), which can include expectations met and positive feelings
  • Value - Primarily an ability to solve a problem, complete a task or achieve a goal making software a social or cognitive prosthesis or tool
  • Feelings & Emotions - Focus on people's feelings as the indicator to quality but be careful to avoid being fooled by illusions
  • Expectations & Oracles - Expectations are mental models of how software works or delivers value; built from references which can be used as oracles to evaluate and identify problems
  • Objective vs Subjective vs Relative - Quality is subjective, relative truth with an objective, relative analysis and is formed from relationships between people and software
  • Quantitative vs Qualitative - Testing is journaling, note-taking and immersion resulting in a written evaluation of quality carefully supported by metrics where appropriate
  • Reliability vs Validity - Avoid misleading information by ensuring consistent and accurate testing through diversity of testing methods and approaches
  • Confirmation vs Falsification - Testing is scepticism, falsification and invalidation not confirmation, verification or demonstration
  • Deduction vs Induction - Testing is constantly building and falsifying hypotheses of quality using logical reasoning to uncover information
  • Method - Testing scientific model and process of research question, background investigation, analysis, hypothesis, experiment, evaluation, action (reporting)
  • Story - Quality is evaluated and communicated as stories of context known as events that contain the who, what, where, when, why and how of value to someone

Approaches

High-level approaches to software quality management and testing that apply to every project

  • Black vs Clear Box - Testing with and without access or knowledge of software internals to find different problems
  • Static vs Dynamic - Testing with and without executing the software's code to find different types of problems
  • Loss vs Harm - Risk/threat modelling 1: Problems occur when value isn't deliver or when additional harm occurs
  • Something vs Nothing - Risk/threat modelling 2: Problems occur when nothing happens when expected but also something happened when unexpected
  • Cost/Value vs Building/Researching - Agile quadrants of high customer value vs low business cost and building software vs studying software
  • Left vs Right - Continuous, parallel testing in throughout the software DevOps lifecycle
  • Idea & Artefact - Software starts with an idea which is communicated and refined through artefacts tested with heuristics and imaginations
  • Strategy - Coverage mindmaps, requirements/risk, charters/questions, designs, environmental/personnel, journaling/noting results

Methods & Tools

Specific micro-techniques and tools used to generate ideas and design tests for certain contextual needs, grouped by type.

Capability

  • Galumph Complete tasks in clumsy, ponderous, roundabout, redundant ways
  • CRUD Creating, reading, updating and deleting things
  • Files Saving, opening, overwriting, renaming, exporting, closing files
  • Drive Full, almost full, write-protected, network, save interrupt
  • 0/1/n/all/n+ Doing zero, one, many, all, too many of things
  • Reverse Executing workflows in reverse, especially wizards
  • Login Authenticating and authorising with the system
  • Flexibility Find the different ways a task can, or should be able to, be achieved
  • Do/Undo Can users undo their last action when accidental click or change of mind?
  • Guide Does it always tell the user the next action or step to take?
  • Prompt Prompt critical actions in case of accidental click or changing mind.
  • Confirm Does the product allow users to check their info before submitting?
  • Feedback Is the product doing something or just hanging? How long to go?
  • Docs Testing user help, install/config, quick start, links, context etc
  • Keyboard Complete tasks only using the keyboard inc. cursor highlighting
  • Touchscreen Complete tasks only using a touchscreen (no mouse/keyboard)
  • Sort Are lists of items sorted correctly, by common understanding and reversable?
  • Bookmark Navigate to a URL including arguments that you don’t have access to
  • Unsaved Populate a lot of data without yet saving. How can it be lost? Time out and log back in?

 

Performance

  • 0.1/1/10s Time between user input and feedback for responsiveness, depending on context
  • Loading… Clear indication of running background task(s) and not just unresponsive
  • Progress Long-running processes give indication and estimation of progress so users can do something else
  • Baseline Performance decreasing over time with continued usage or with every new release
  • CPU/RAM/SD/NET Virtual resource usage and cost
  • Leak Memory leak where memory is consumed over long periods of time

 

Usability

  • Repeat How often are actions completed more than once that are annoying or unnecessary?
  • 3-Clicks Requiring 3 clicks or less to navigate to any function within an application
  • Intuitive Is it obvious how to use the product as a new user without help?
  • Clarity All operations that can be performed are clear and simple.
  • Tooltips Pop-ups on hover that give more info on a form field or input
  • Workflow Test for fluency, elegance and effortlessness
  • Reassure Reassuringly stating what’s gone wrong and what to do next
  • Layout Logical layout of UI components and menus, incl. plenty of spacing.
  • KISS (Keep It Simple, Stupid) aka Simplicity: Is the solution the simplest way?
  • Minimalist What could you remove from the product without affecting value?
  • Readability How easy is it to read all information on screen?
  • Strain Does it cause eyestrain after hours, days, months, years of use?
  • Least surprise principle

 

Accessibility

  • Alt-Text Images with alt-text and audio/video with transcripts/subtitles
  • Zoom Layout and text scale up and down with zoom, up to 200%
  • Magnifier Using the product with a magnifier tool, e.g. operating system default
  • Readers Using the product with screen readers and narrator tools
  • Colourblind Important context or info communicated in colour only
  • Flashing Checking for on-screen flashing (by design or not)
  • Timing Testing for things happening too quickly
  • Wording Text assessment for reading needs, e.g. dyslexia
  • Contrast Using the product under high colour contrast mode
  • Systematic HTML
  • Hemmingway
  • WAVE
  • ARIA (Accessible Rich Internet Applications)

 

Concurrency

  • Race One operation failing due to a mistimed dependent operation
  • Deadlock Trying to lock access to a resource with concurrent connections
  • Conflict An operation can’t logically be performed at the same time as another

 

Reliability

  • Validate Checking for useful warnings for incorrect user input
  • Interrupt Stop, cancel or close something whilst it’s processing
  • Recover Test system recoverability from errors and crashes
  • Network See what happens with network connection losses
  • Endure Running a routine for days or weeks to find errors
    Fuzz Purposefully scramble data before processing to see what happens
  • SLA Potential risks to service-level agreement
  • 5-9s A common Service Level Agreement (SLA) of 99.999% availability

 

Security

  • Auth - Authentication vs Authorisation: Confirming users are who they say they are and assigning what users can do and access
  • MAC/DAC/RBAC - For authentication and authorisation, logical access control can be mandatory, discretionary or role-based
  • MFA - Authentication via multiple method types of knowledge, possession (token) and inherence (biometric) where needed

  • Brute Force authentication via repeated password guessing  due to lack /timing of attempts, captchas or logging
  • Format Exploit SQL injection, JS injection, cross-site scripting, clickjacking
  • Policy Policy enforcement including password complexity
  • DoS (Denial of service) Remove the availability of a product, service or aspect
  • Social Engineering via phishing or physical access via tailgating
  • Glean Restricted information viewable from other places and/or by other means
  • Audit Logs of who has done what and when to check for access and action
  • Repudiate Perform actions without knowledge e.g. without logging or by cleaning audit logs without logging
  • OWASP ZAP Vulnerability scanner tool
  • Burpsuite Vulnerability scanner tool
  • Fiddler Network proxy tool
  • Wireshark Network sniffer tool
  • Bug Magnet Browser extension – format exploits

 

Portability

  • OS Windows, Mac, Linux, iOS, Android
  • Device Desktop, laptop, tablet, phone, smart TV, projector, VR headset
  • Frag Screen sizes and resolutions, particularly most common

 

Localisation

  • TZ Different time zones where the software will or may be used, including clients/servers in different time zones
  • DST Daylight saving time zone changes, including across time change
  • Language Different countries, languages and translation options
  • Decimal Different decimal formats e.g. period, comma, space, none including currency
  • Date/Time Different date and time formats e.g. endians, 12/24HR

 

Scalability

  • Peak When and what are the peak times of usage that put load on the system?
  • Load How is the product affected when put under heavy load (e.g. users?)
  • Heavy What are the heavy operations that affect the product the most?
  • Stress Increasing load until breaking point to see what happens to the product.
  • StresStimulus™ Tool to measuring performance and load
  • Big/Little Large files, amount of data, numbers etc

 

Operability

  • Install How easy to install, uninstall and reinstall the application
  • Repair How easy to repair any corrupted or failed installation
  • Setup How easy to setup the application for first use
  • Config How easy to configure and reconfigure the application to suit needs
  • Customise How easy to customise the application (including installing/removing modifications) to suit needs
  • Update/Downgrade How easy to perform updates or even downgrades when required
  • Patch/Hotfix How quickly and easy to deploy patches or hotfixes into production
  • Deploy/Rollback How easy to deploy updates or hotfixes in production
  • CI/CD Easy/frequency/logging of automated pipeline for deployments, rollbacks, hotfixes
  • Diagnose How easy is it to find the cause of problems or failures in production
  • Monitor How easy is it to monitor production for potential problems
  • Version Is the version number readily available and copyable for tech support?

 

Testability

  • Config Options that make testing easier, e.g. hidden testing interface
  • Observe Aspects that make it easier to observe what’s happening
  • Log Suitable logging and logging levels to fish for and pinpoint problems
  • Hook Anchor points for automation scripting e.g. CSS IDs for selectors

 

Maintainability

  • Spaghetti Easy to read, understand and maintain source code
  • Unit Adequate level of unit and integration tests for an area of functionality
  • Comment Adequate code comments and commit messages

 

Aesthetics

  • Style Contemporary and overall look and feel e.g. colours
  • Align Alignment and spacing of GUI elements, including text, images, CSS etc
  • Proof Incorrect spelling and grammar
  • Brand Missing, incorrect or out-of-date corporate logos, colours, taglines etc

 

Data

  • Formats Text, integers, decimals, Booleans, single chars
  • Violate Special chars, negatives, nulls, empty strings, Unicode

 

Documentation

  • Help Is user help easily available, accessible, context-aware and up-to-date?
  • Guides Are installation, configuration and quick-start guides available?
  • FAQs For frequently encountered problems

Dimensions

Different ways in which value of a software product or project can be delivered or threatened

  • Introduction to Quality Dimensions - Different ways in which value or harm is delivered and recognised by "-ility" nouns
  • Capability - Ability to complete tasks in reasonable and expected ways, possibly in multiple different ways (flexibility).
  • Performance - Ability to complete tasks in reasonable time and with reasonable responsiveness
  • Usability - Ability to complete those tasks reasonably easily, simply and intuitively.
  • Accessibility - Ability to complete those tasks reasonably well as accessibility-needs users
  • Security - Ability to reasonably guard against completing or altering tasks as unauthorised users or prevent authorised users from doing so
  • Reliability - Ability to complete tasks reasonable accurately and when needed
  • Concurrency Ability to complete tasks or user functions reasonably simultaneously with similar tasks and with other tasks
  • Efficiency Ability to complete tasks or user functions with reasonable system resources and running cost
  • Portability Ability to complete tasks or user functions on different devices and environments
  • Localisation Ability to complete tasks or user functions reasonably well in different countries or cultures
  • Scalability Ability of all Quality Criteria to reasonably scale with increased usage, size, complexity, market, time etc
  • Operability Ability to reasonably deploy, install, configure, update and resolve problems in production with reasonable disruption and time scale
  • Testability Ability to reasonably observe and configure the product for testing purposes
  • Maintainability Ability to reasonably maintain the product for future development including bug fixing, expansion and repurposing
  • Aesthetics Ability to otherwise give a good impression of quality and be reasonably marketable

References

Different points of comparison for oracles when identifying and describing aspects of software quality such as bugs

  • Intention Consistency with the intended purpose of software and the value it delivers to stakeholders (whether implicit or explicit)
  • Documentation Consistency with explicit claims made in project documentation, such as specifications, guides and notes
  • Conversation Consistency with explicit claims mentioned by stakeholders in meetings, discussions, stakeholder interviews or in-passing
  • Application Consistency with other software products, systems or services that share similarities in purpose or function
  • Version Consistency with past versions of the same software program
  • Reflection Consistency with the software program itself, whether the same feature or function, or similar features and functions
  • Jurisdiction Consistency with national or regional governmental, legislative or judicial regulations, statutes or precedents
  • Standardisation Consistency with recommended best practices by external professional bodies and recognised industry experts
  • Organisation Consistency with policies, procedures and precedents set internally by the organisation
  • Reputation Consistency with how stakeholders wish to be seen or the image they wish to project
  • Restriction Consistency with real-world constraints, such as language, mathematics and data
  • Problem Inconsistency with previous problems that stakeholders decided needed fixing (which relate to another reference above)

Diagrams

General techniques used to generate ideas and design tests for most common software projects.

Management

Software quality management and management of testing teams.

  • Titles - Individual contributors: apprentice, associate, mid, senior, principal, architect and administrative officers: lead, manager, director
  • Metrics - Assigning numbers to test processes including information or motivation; value vs cost; execution, reporting and setup; dysfunction and construct validity
  • Process - Process are repeatable tasks to achieve something modellable as flow diagrams
  • Motivation - Measurement management via theory X external vs delegation management via theory Y internal, plus values and relationships
  • Assurance vs Control - Proactive vs reactive approaches to putting good things in and taking bad things out of testing processes to assure and control testing quality
  • Mentor - Process where seniors help/assist/teach/advise/guide juniors/interns/apprentices to onboard, upskill, achieve goal, get promoted informally/ad-hoc or part of a programme
  • 1:1 - (One-to-one) Regular, private, non-status update, tester-driven agenda meetings to build human connections, trust and rapport with manager