Complete list of all articles organised by category.
Fundamental concepts of software testing and quality as applied research and a social science.
- Science & Testing - Testing is applied research into software quality using scientific approaches and evidence to help stakeholders make informed decisions
- Computer Science & Software Engineering - Testing isn't algorithms, problem solving, solution building or process management but a knowledge of both is important for testing
- Quality - Defined as value to some person(s), which can include expectations met and positive feelings
- Value - Primarily an ability to solve a problem, complete a task or achieve a goal making software a social or cognitive prosthesis or tool
- Feelings & Emotions - Focus on people's feelings as the indicator to quality but be careful to avoid being fooled by illusions
- Expectations & Oracles - Expectations are mental models of how software works or delivers value; built from references which can be used as oracles to evaluate and identify problems
- Objective vs Subjective vs Relative - Quality is subjective, relative truth with an objective, relative analysis and is formed from relationships between people and software
- Quantitative vs Qualitative - Testing is journaling, note-taking and immersion resulting in a written evaluation of quality carefully supported by metrics where appropriate
- Reliability vs Validity - Avoid misleading information by ensuring consistent and accurate testing through diversity of testing methods and approaches
- Confirmation vs Falsification - Testing is scepticism, falsification and invalidation not confirmation, verification or demonstration
- Deduction vs Induction - Testing is constantly building and falsifying hypotheses of quality using logical reasoning to uncover information
- Method - Testing scientific model and process of research question, background investigation, analysis, hypothesis, experiment, evaluation, action (reporting)
- Story - Quality is evaluated and communicated as stories of context known as events that contain the who, what, where, when, why and how of value to someone
High-level approaches to software quality management and testing that apply to every project
- Black vs Clear Box - Testing with and without access or knowledge of software internals to find different problems
- Static vs Dynamic - Testing with and without executing the software's code to find different types of problems
- Loss vs Harm - Risk/threat modelling 1: Problems occur when value isn't deliver or when additional harm occurs
- Something vs Nothing - Risk/threat modelling 2: Problems occur when nothing happens when expected but also something happened when unexpected
- Cost/Value vs Building/Researching - Agile quadrants of high customer value vs low business cost and building software vs studying software
- Left vs Right - Continuous, parallel testing in throughout the software DevOps lifecycle
- Idea & Artefact - Software starts with an idea which is communicated and refined through artefacts tested with heuristics and imaginations
- Strategy - Coverage mindmaps, requirements/risk, charters/questions, designs, environmental/personnel, journaling/noting results
Methods & Tools
Specific micro-techniques and tools used to generate ideas and design tests for certain contextual needs, grouped by type.
- Galumph Complete tasks in clumsy, ponderous, roundabout, redundant ways
- CRUD Creating, reading, updating and deleting things
- Files Saving, opening, overwriting, renaming, exporting, closing files
- Drive Full, almost full, write-protected, network, save interrupt
- 0/1/n/all/n+ Doing zero, one, many, all, too many of things
- Reverse Executing workflows in reverse, especially wizards
- Login Authenticating and authorising with the system
- Flexibility Find the different ways a task can, or should be able to, be achieved
- Do/Undo Can users undo their last action when accidental click or change of mind?
- Guide Does it always tell the user the next action or step to take?
- Prompt Prompt critical actions in case of accidental click or changing mind.
- Confirm Does the product allow users to check their info before submitting?
- Feedback Is the product doing something or just hanging? How long to go?
- Docs Testing user help, install/config, quick start, links, context etc
- Keyboard Complete tasks only using the keyboard inc. cursor highlighting
- Touchscreen Complete tasks only using a touchscreen (no mouse/keyboard)
- Sort Are lists of items sorted correctly, by common understanding and reversable?
- Bookmark Navigate to a URL including arguments that you don’t have access to
- Unsaved Populate a lot of data without yet saving. How can it be lost? Time out and log back in?
- 0.1/1/10s Time between user input and feedback for responsiveness, depending on context
- Loading… Clear indication of running background task(s) and not just unresponsive
- Progress Long-running processes give indication and estimation of progress so users can do something else
- Baseline Performance decreasing over time with continued usage or with every new release
- CPU/RAM/SD/NET Virtual resource usage and cost
- Leak Memory leak where memory is consumed over long periods of time
- Repeat How often are actions completed more than once that are annoying or unnecessary?
- 3-Clicks Requiring 3 clicks or less to navigate to any function within an application
- Intuitive Is it obvious how to use the product as a new user without help?
- Clarity All operations that can be performed are clear and simple.
- Tooltips Pop-ups on hover that give more info on a form field or input
- Workflow Test for fluency, elegance and effortlessness
- Reassure Reassuringly stating what’s gone wrong and what to do next
- Layout Logical layout of UI components and menus, incl. plenty of spacing.
- KISS (Keep It Simple, Stupid) aka Simplicity: Is the solution the simplest way?
- Minimalist What could you remove from the product without affecting value?
- Readability How easy is it to read all information on screen?
- Strain Does it cause eyestrain after hours, days, months, years of use?
- Least surprise principle
- Alt-Text Images with alt-text and audio/video with transcripts/subtitles
- Zoom Layout and text scale up and down with zoom, up to 200%
- Magnifier Using the product with a magnifier tool, e.g. operating system default
- Readers Using the product with screen readers and narrator tools
- Colourblind Important context or info communicated in colour only
- Flashing Checking for on-screen flashing (by design or not)
- Timing Testing for things happening too quickly
- Wording Text assessment for reading needs, e.g. dyslexia
- Contrast Using the product under high colour contrast mode
- Systematic HTML
- ARIA (Accessible Rich Internet Applications)
- Race One operation failing due to a mistimed dependent operation
- Deadlock Trying to lock access to a resource with concurrent connections
- Conflict An operation can’t logically be performed at the same time as another
- Validate Checking for useful warnings for incorrect user input
- Interrupt Stop, cancel or close something whilst it’s processing
- Recover Test system recoverability from errors and crashes
- Network See what happens with network connection losses
- Endure Running a routine for days or weeks to find errors
Fuzz Purposefully scramble data before processing to see what happens
- SLA Potential risks to service-level agreement
- 5-9s A common Service Level Agreement (SLA) of 99.999% availability
- Auth - Authentication vs Authorisation: Confirming users are who they say they are and assigning what users can do and access
- MAC/DAC/RBAC - For authentication and authorisation, logical access control can be mandatory, discretionary or role-based
- MFA - Authentication via multiple method types of knowledge, possession (token) and inherence (biometric) where needed
- Brute Force authentication via repeated password guessing due to lack /timing of attempts, captchas or logging
- Format Exploit SQL injection, JS injection, cross-site scripting, clickjacking
- Policy Policy enforcement including password complexity
- DoS (Denial of service) Remove the availability of a product, service or aspect
- Social Engineering via phishing or physical access via tailgating
- Glean Restricted information viewable from other places and/or by other means
- Audit Logs of who has done what and when to check for access and action
- Repudiate Perform actions without knowledge e.g. without logging or by cleaning audit logs without logging
- OWASP ZAP Vulnerability scanner tool
- Burpsuite Vulnerability scanner tool
- Fiddler Network proxy tool
- Wireshark Network sniffer tool
- Bug Magnet Browser extension – format exploits
- OS Windows, Mac, Linux, iOS, Android
- Device Desktop, laptop, tablet, phone, smart TV, projector, VR headset
- Frag Screen sizes and resolutions, particularly most common
- TZ Different time zones where the software will or may be used, including clients/servers in different time zones
- DST Daylight saving time zone changes, including across time change
- Language Different countries, languages and translation options
- Decimal Different decimal formats e.g. period, comma, space, none including currency
- Date/Time Different date and time formats e.g. endians, 12/24HR
- Peak When and what are the peak times of usage that put load on the system?
- Load How is the product affected when put under heavy load (e.g. users?)
- Heavy What are the heavy operations that affect the product the most?
- Stress Increasing load until breaking point to see what happens to the product.
- StresStimulus™ Tool to measuring performance and load
- Big/Little Large files, amount of data, numbers etc
- Install How easy to install, uninstall and reinstall the application
- Repair How easy to repair any corrupted or failed installation
- Setup How easy to setup the application for first use
- Config How easy to configure and reconfigure the application to suit needs
- Customise How easy to customise the application (including installing/removing modifications) to suit needs
- Update/Downgrade How easy to perform updates or even downgrades when required
- Patch/Hotfix How quickly and easy to deploy patches or hotfixes into production
- Deploy/Rollback How easy to deploy updates or hotfixes in production
- CI/CD Easy/frequency/logging of automated pipeline for deployments, rollbacks, hotfixes
- Diagnose How easy is it to find the cause of problems or failures in production
- Monitor How easy is it to monitor production for potential problems
- Version Is the version number readily available and copyable for tech support?
- Config Options that make testing easier, e.g. hidden testing interface
- Observe Aspects that make it easier to observe what’s happening
- Log Suitable logging and logging levels to fish for and pinpoint problems
- Hook Anchor points for automation scripting e.g. CSS IDs for selectors
- Spaghetti Easy to read, understand and maintain source code
- Unit Adequate level of unit and integration tests for an area of functionality
- Comment Adequate code comments and commit messages
- Style Contemporary and overall look and feel e.g. colours
- Align Alignment and spacing of GUI elements, including text, images, CSS etc
- Proof Incorrect spelling and grammar
- Brand Missing, incorrect or out-of-date corporate logos, colours, taglines etc
- Formats Text, integers, decimals, Booleans, single chars
- Violate Special chars, negatives, nulls, empty strings, Unicode
- Help Is user help easily available, accessible, context-aware and up-to-date?
- Guides Are installation, configuration and quick-start guides available?
- FAQs For frequently encountered problems
Different ways in which value of a software product or project can be delivered or threatened
- Introduction to Quality Dimensions - Different ways in which value or harm is delivered and recognised by "-ility" nouns
- Capability - Ability to complete tasks in reasonable and expected ways, possibly in multiple different ways (flexibility).
- Performance - Ability to complete tasks in reasonable time and with reasonable responsiveness
- Usability - Ability to complete those tasks reasonably easily, simply and intuitively.
- Accessibility - Ability to complete those tasks reasonably well as accessibility-needs users
- Security - Ability to reasonably guard against completing or altering tasks as unauthorised users or prevent authorised users from doing so
- Reliability - Ability to complete tasks reasonable accurately and when needed
- Concurrency Ability to complete tasks or user functions reasonably simultaneously with similar tasks and with other tasks
- Efficiency Ability to complete tasks or user functions with reasonable system resources and running cost
- Portability Ability to complete tasks or user functions on different devices and environments
- Localisation Ability to complete tasks or user functions reasonably well in different countries or cultures
- Scalability Ability of all Quality Criteria to reasonably scale with increased usage, size, complexity, market, time etc
- Operability Ability to reasonably deploy, install, configure, update and resolve problems in production with reasonable disruption and time scale
- Testability Ability to reasonably observe and configure the product for testing purposes
- Maintainability Ability to reasonably maintain the product for future development including bug fixing, expansion and repurposing
- Aesthetics Ability to otherwise give a good impression of quality and be reasonably marketable
Different points of comparison for oracles when identifying and describing aspects of software quality such as bugs
- Intention Consistency with the intended purpose of software and the value it delivers to stakeholders (whether implicit or explicit)
- Documentation Consistency with explicit claims made in project documentation, such as specifications, guides and notes
- Conversation Consistency with explicit claims mentioned by stakeholders in meetings, discussions, stakeholder interviews or in-passing
- Application Consistency with other software products, systems or services that share similarities in purpose or function
- Version Consistency with past versions of the same software program
- Reflection Consistency with the software program itself, whether the same feature or function, or similar features and functions
- Jurisdiction Consistency with national or regional governmental, legislative or judicial regulations, statutes or precedents
- Standardisation Consistency with recommended best practices by external professional bodies and recognised industry experts
- Organisation Consistency with policies, procedures and precedents set internally by the organisation
- Reputation Consistency with how stakeholders wish to be seen or the image they wish to project
- Restriction Consistency with real-world constraints, such as language, mathematics and data
- Problem Inconsistency with previous problems that stakeholders decided needed fixing (which relate to another reference above)
General techniques used to generate ideas and design tests for most common software projects.
- Boundary Value Analysis - Testing one value either side of valid and invalid data ranges and partitions
Software quality management and management of testing teams.
- Titles - Individual contributors: apprentice, associate, mid, senior, principal, architect and administrative officers: lead, manager, director
- Metrics - Assigning numbers to test processes including information or motivation; value vs cost; execution, reporting and setup; dysfunction and construct validity
- Process - Process are repeatable tasks to achieve something modellable as flow diagrams
- Motivation - Measurement management via theory X external vs delegation management via theory Y internal, plus values and relationships
- Assurance vs Control - Proactive vs reactive approaches to putting good things in and taking bad things out of testing processes to assure and control testing quality
- Mentor - Process where seniors help/assist/teach/advise/guide juniors/interns/apprentices to onboard, upskill, achieve goal, get promoted informally/ad-hoc or part of a programme
- 1:1 - (One-to-one) Regular, private, non-status update, tester-driven agenda meetings to build human connections, trust and rapport with manager