Index

Complete list of all articles organised by category and glossary of terms.

Fundamentals

Fundamental concepts of software testing and quality as applied research and a social science.

  • Science & Testing - Testing is applied research into software quality using scientific approaches and evidence to help stakeholders make informed decisions
  • Computer Science & Software Engineering - Testing isn't algorithms, problem solving, solution building or process management but a knowledge of both is important for testing
  • Quality - Defined as value to some person(s), which can include expectations met and positive feelings
  • Value - Primarily an ability to solve a problem, complete a task or achieve a goal making software a social or cognitive prosthesis or tool
  • Stakeholder - Forming part of the value/risk story: Anyone who has a vested interest in the success (or failure) of the project
  • Feelings & Emotions - Focus on people's feelings as the indicator to quality but be careful to avoid being fooled by illusions
  • Expectations & Oracles - Expectations are mental models of how software works or delivers value; built from references which can be used as oracles to evaluate and identify problems
  • Objective vs Subjective vs Relative - Quality is subjective, relative truth with an objective, relative analysis and is formed from relationships between people and software
  • Quantitative vs Qualitative - Testing is journaling, note-taking and immersion resulting in a written evaluation of quality carefully supported by metrics where appropriate
  • Reliability vs Validity - Avoid misleading information by ensuring consistent and accurate testing through diversity of testing methods and approaches
  • Confirmation vs Falsification - Testing is scepticism, falsification and invalidation not confirmation, verification or demonstration
  • Deduction vs Induction - Testing is constantly building and falsifying hypotheses of quality using logical reasoning to uncover information
  • Method - Testing scientific model and process of research question, background investigation, analysis, hypothesis, experiment, evaluation, action (reporting)
  • Story - Quality is evaluated and communicated as stories of context known as events that contain the who, what, where, when, why and how of value to someone

Approaches

High-level approaches to software quality management and testing that apply to every project

  • Black vs Clear Box - Testing with and without access or knowledge of software internals to find different problems
  • Static vs Dynamic - Testing with and without executing the software's code to find different types of problems
  • Loss vs Harm - Risk/threat modelling 1: Problems occur when value isn't deliver or when additional harm occurs
  • Something vs Nothing - Risk/threat modelling 2: Problems occur when nothing happens when expected but also something happened when unexpected
  • Cost/Value vs Building/Researching - Agile quadrants of high customer value vs low business cost and building software vs studying software
  • Left vs Right - Continuous, parallel testing in throughout the software DevOps lifecycle
  • Idea & Artefact - Software starts with an idea which is communicated and refined through artefacts tested with heuristics and imaginations
  • Strategy - Coverage mindmaps, requirements/risk, charters/questions, designs, environmental/personnel, journaling/noting results
  • Risk Mapping & Threat Modelling - Risk assessment framework via value, scope, assets, categories, risks, threats, vulnerabilities then prioritise document, manage, monitor, retrospect

Methods & Tools

Specific techniques, tools and frameworks used to generate ideas and design tests for certain contextual needs, grouped by type.

Capability

  • Galumph Complete tasks in clumsy, ponderous, roundabout, redundant ways
  • CRUD Creating, reading, updating and deleting things
  • Files Saving, opening, overwriting, renaming, exporting, closing files
  • Drive Full, almost full, write-protected, network, save interrupt
  • 0/1/n/all/n+ Doing zero, one, many, all, too many of things
  • Reverse Executing workflows in reverse, especially wizards
  • Login Authenticating and authorising with the system
  • Flexibility Find the different ways a task can, or should be able to, be achieved
  • Do/Undo Can users undo their last action when accidental click or change of mind?
  • Guide Does it always tell the user the next action or step to take?
  • Prompt Prompt critical actions in case of accidental click or changing mind.
  • Confirm Does the product allow users to check their info before submitting?
  • Feedback Is the product doing something or just hanging? How long to go?
  • Docs Testing user help, install/config, quick start, links, context etc
  • Keyboard Complete tasks only using the keyboard inc. cursor highlighting
  • Touchscreen Complete tasks only using a touchscreen (no mouse/keyboard)
  • Sort Are lists of items sorted correctly, by common understanding and reversable?
  • Bookmark Navigate to a URL including arguments that you don’t have access to
  • Unsaved Populate a lot of data without yet saving. How can it be lost? Time out and log back in?

 

Performance

  • 0.1/1/10s Time between user input and feedback for responsiveness, depending on context
  • Loading… Clear indication of running background task(s) and not just unresponsive
  • Progress Long-running processes give indication and estimation of progress so users can do something else
  • Baseline Performance decreasing over time with continued usage or with every new release
  • CPU/RAM/SD/NET Virtual resource usage and cost
  • Leak Memory leak where memory is consumed over long periods of time

 

Usability

  • Repeat How often are actions completed more than once that are annoying or unnecessary?
  • 3-Clicks Requiring 3 clicks or less to navigate to any function within an application
  • Intuitive Is it obvious how to use the product as a new user without help?
  • Clarity All operations that can be performed are clear and simple.
  • Tooltips Pop-ups on hover that give more info on a form field or input
  • Workflow Test for fluency, elegance and effortlessness
  • Reassure Reassuringly stating what’s gone wrong and what to do next
  • Layout Logical layout of UI components and menus, incl. plenty of spacing.
  • KISS (Keep It Simple, Stupid) aka Simplicity: Is the solution the simplest way?
  • Minimalist What could you remove from the product without affecting value?
  • Readability How easy is it to read all information on screen?
  • Strain Does it cause eyestrain after hours, days, months, years of use?
  • Least surprise principle

 

Accessibility

  • Alt-Text Images with missing alt-text and audio/video with missing transcripts/subtitles
  • Zoom Layout and text scale up and down with zoom, up to 200%
  • Magnifier Using the product with a magnifier tool, e.g. operating system default
  • Readers Using the product with screen readers and narrator tools
  • Colourblind Important context or info communicated in colour only
  • Flashing Checking for on-screen flashing (by design or not)
  • Timing Testing for things happening too quickly
  • Wording Text assessment for reading needs, e.g. dyslexia
  • Contrast Using the product under high colour contrast mode
  • Systematic HTML
  • Hemmingway
  • WAVE
  • ARIA Accessible Rich Internet Applications

 

Concurrency

  • Race One operation failing due to a mistimed dependent operation
  • Deadlock Trying to lock access to a resource with concurrent connections
  • Conflict An operation can’t logically be performed at the same time as another

 

Reliability

  • Validate Checking for useful warnings for incorrect user input
  • Interrupt Stop, cancel or close something whilst it’s processing
  • Recover Test system recoverability from errors and crashes
  • Network See what happens with network connection losses
  • Endure Running a routine for days or weeks to find errors
  • Fuzz Purposefully scramble data before processing to see what happens
  • SLA Potential risks to service-level agreement
  • 5-9s A common Service Level Agreement (SLA) of 99.999% availability

 

Security

  • STRIDE Spoof, tamper, repudiate. info disclose, DoS, expand authority (elevate privilege)
  • PASTA Process for Attack Simulation and Threat Analysis (Red Teaming)
  • OWASP Top 10 Top 10 most common and highest risk security threats per year
  • CVSS Common Vulnerability Scoring System wit many tools that can perform this automatically
  • DREAD Threat Damage (impact), Reproducibility, Exploitability, Affected users, Discoverability
  • Trike Structured auditing and documenting using risk scoring and system interactions
  • VAST Visual, Agile and Simple Threat application and operational flow diagramming
  • ATT&CK MITRE Red team DB of adversary Tactics, Techniques, Procedures (TTP) to map attack patterns and identify coverage gaps
  • D3FEND MITRE DB of countermeasures to attacks
  • IEC 27005 Agile and DevSecOps asset-driven infosec risk management framework based on ISO/IEC 31000
  • NIST SP 800-30 Structured and procedural threat and vulnerability 9-step process for regulated environments
  • FAIR Factor Analysis of Information Risk: Quantitative, business cost focussed
  • GDPR General Data Protection Regulation for strict information controlling and processing for EU
  • Mandiant Threat vendor report
  • CrowdStrike Threat vendor report
  • Recorded Future Threat vendor report
  • NVD NIST National Vulnerability Database
  • Exploit-DB OFFSEC vulnerability exploit DB for testers
  • VulnDB Vulnerability management and threat intelligence platform
  • CVE Common Vulnerabilities and Exposures: Public infosec DB
  • CWE Common Weakness Enumeration: Categories for hardware, software weaknesses and vulnerabilities
  • CAPEC Common Attack Pattern Enumeration and Classification: Catalog of known cyber security attack patterns
  • TIP Threat Intelligence Platforms: Threat analysis and management tools
  • Obfuscate Ability to remove sensitive data from a larger data set
  • Brute Force authentication via repeated password guessing  due to lack /timing of attempts, captchas or logging
  • SQL Format exploits for SQL injection
  • JS Format exploits for JavaScript injection
  • XSS Format exploit for cross-site scripting
  • Clickjack Format exploit for clickjacking
  • Policy Policy enforcement including password complexity
  • DoS Denial of Service: Remove the availability of a product, service or aspect
  • Social Engineering via phishing or physical access via tailgating
  • Audit Logs of who has done what and when to check for access and action
  • Repudiate Perform actions without knowledge e.g. without logging or by cleaning audit logs without logging
  • Tamper Unauthorised modification to files, network connections etc
  • Info Disclose Confidentiality breach: Authorised access to asset data (including meta-data)
  • Glean Info Disclosure viewable from other places and/or by other means than the typical authorised route
  • Expand Authority (Elevate Privilege) Perform actions above or outside privilege level
  • Q-Day Quantum-ready cryptography for the day existing approaches become redundant
  • MS TMT: Threat Modelling Tool by Mircosoft
  • IriusRisk DevSecOps CI/ID pipelines, diagram import using automation from sources
  • RSA Archer GRC risk management tool
  • OneTrust GRC risk management tool
  • OWASP Threat Dragon Web-based modelling tool for threat modelling
  • OWASP ZAP Vulnerability scanner tool
  • Burpsuite Vulnerability scanner tool
  • Fiddler Network proxy tool
  • Wireshark Network sniffer tool
  • Bug Magnet Browser extension – format exploits

 

Portability

  • OS Issues on Windows, Mac, Linux, iOS, Android etc
  • Device Issues on desktop, laptop, tablet, phone, smart TV, projector, VR headset etc
  • Frag Issues on screen sizes and resolutions, particularly most common

 

Localisation

  • TZ Different time zones where the software will or may be used, including clients/servers in different time zones
  • DST Daylight saving time zone changes, including across time change
  • Language Different countries, languages and translation options
  • Decimal Different decimal formats e.g. period, comma, space, none including currency
  • Date/Time Different date and time formats e.g. endians, 12/24HR

 

Scalability

  • Peak When and what are the peak times of usage that put load on the system?
  • Load How is the product affected when put under heavy load (e.g. users?)
  • Heavy What are the heavy operations that affect the product the most?
  • Stress Increasing load until breaking point to see what happens to the product.
  • StresStimulus™ Tool to measuring performance and load
  • Big/Little Large files, amount of data, numbers etc

 

Operability

  • Install How easy to install, uninstall and reinstall the application
  • Repair How easy to repair any corrupted or failed installation
  • Setup How easy to setup the application for first use
  • Config How easy to configure and reconfigure the application to suit needs
  • Customise How easy to customise the application (including installing/removing modifications) to suit needs
  • Update/Downgrade How easy to perform updates or even downgrades when required
  • Patch/Hotfix How quickly and easy to deploy patches or hotfixes into production
  • Deploy/Rollback How easy to deploy updates or hotfixes in production
  • CI/CD Easy/frequency/logging of automated pipeline for deployments, rollbacks, hotfixes
  • Diagnose How easy is it to find the cause of problems or failures in production
  • Monitor How easy is it to monitor production for potential problems
  • Version Is the version number readily available and copyable for tech support?

 

Testability

  • Config Options that make testing easier, e.g. hidden testing interface
  • Observe Aspects that make it easier to observe what’s happening
  • Log Suitable logging and logging levels to fish for and pinpoint problems
  • Hook Anchor points for automation scripting e.g. CSS IDs for selectors

 

Maintainability

  • Spaghetti Easy to read, understand and maintain source code
  • Unit Adequate level of unit and integration tests for an area of functionality
  • Comment Adequate code comments and commit messages

 

Aesthetics

  • Style Contemporary and overall look and feel e.g. colours
  • Align Alignment and spacing of GUI elements, including text, images, CSS etc
  • Proof Incorrect spelling and grammar
  • Brand Missing, incorrect or out-of-date corporate logos, colours, taglines etc

 

Data

  • Formats Text, integers, decimals, Booleans, single chars
  • Violate Special chars, negatives, nulls, empty strings, Unicode

 

Documentation

  • Help Easily available, accessible, context-aware and up-to-date? E.g. F1 key, right-click menu, versioned
  • Guides Are installation, configuration and quick-start guides available?
  • FAQs For frequently encountered problems

Dimensions

Different ways in which value of a software product or project can be delivered or threatened

 

Product

  • Capability - Ability to complete tasks in reasonable and expected ways, possibly in multiple different ways (flexibility).
  • Performance - Ability to complete tasks in reasonable time and with reasonable responsiveness
  • Usability - Ability to complete those tasks reasonably easily, simply and intuitively.
  • Accessibility - Ability to complete those tasks reasonably well as accessibility-needs users
  • Security - Ability to reasonably guard against completing or altering tasks as unauthorised users or prevent authorised users from doing so
  • Reliability - Ability to complete tasks reasonable accurately and when needed

  • Efficiency Ability to complete tasks or user functions with reasonable system resources and running cost
  • Portability Ability to complete tasks or user functions on different devices and environments
  • Localisation Ability to complete tasks or user functions reasonably well in different countries or cultures
  • Scalability Ability of all Quality Criteria to reasonably scale with increased usage, size, complexity, market, time etc
  • Operability Ability to reasonably deploy, install, configure, update and resolve problems in production with reasonable disruption and time scale
  • Testability Ability to reasonably observe and configure the product for testing purposes
  • Maintainability Ability to reasonably maintain the product for future development including bug fixing, expansion and repurposing
  • Aesthetics Ability to otherwise give a good impression of quality and be reasonably marketable

 

Project

  • Security (Project) - Ability for teams to guard against information confidentiality, integrity, availability issues

  • Productivity Ability to complete work and deliver value to stakeholders to time/cost and quality
  • Agility Ability to complete work and solve problems using an agile mindset and principles
  • Diversity Ability to bring different ideas, viewpoints, approaches and risk to problem-solving and solution-building
  • Growth Ability to learn, understand, explore and try out new ideas to see how they fail or succeed
  • Morale Ability to be happy, motivated, engaged and fulfilled by the project and testing work
  • Safety Ability to feel psychologically safe to speak out, try new things and to fail

 

References

Different points of comparison for oracles when identifying and describing aspects of software quality such as bugs

  • Intention Consistency with the intended purpose of software and the value it delivers to stakeholders (whether implicit or explicit)
  • Documentation Consistency with explicit claims made in project documentation, such as specifications, guides and notes
  • Conversation Consistency with explicit claims mentioned by stakeholders in meetings, discussions, stakeholder interviews or in-passing
  • Application Consistency with other software products, systems or services that share similarities in purpose or function
  • Version Consistency with past versions of the same software program
  • Reflection Consistency with the software program itself, whether the same feature or function, or similar features and functions
  • Jurisdiction Consistency with national or regional governmental, legislative or judicial regulations, statutes or precedents
  • Standardisation Consistency with recommended best practices by external professional bodies and recognised industry experts
  • Organisation Consistency with policies, procedures and precedents set internally by the organisation
  • Reputation Consistency with how stakeholders wish to be seen or the image they wish to project
  • Restriction Consistency with real-world constraints, such as language, mathematics and data
  • Problem Inconsistency with previous problems that stakeholders decided needed fixing (which relate to another reference above)

Diagrams

General techniques used to generate ideas and design tests for most common software projects.

Automation

Software projects and tools to support testing activities, including automated encoded checks and test cases at various stack layers.

Elements

Components, processes and data that can make up the assets and vulnerabilities of a software product, system or project

Components

  • Help Help guides, quick start guides, installation and configuration guides
  • FAQs For frequently encountered problems and questions

 

Processes

  • Auth - Authentication vs Authorisation: Confirming users are who they say they are and assigning what users can do and access
  • MAC/DAC/RBAC - For authentication and authorisation, logical access control can be mandatory, discretionary or role-based
  • MFA - Authentication via multiple method types of knowledge, possession (token) and inherence (biometric) where needed

  • Privilege Configured actions that a user account can perform (above normal/anon level)
  • Permissions The actions or tasks a user can perform or complete with their account privileges
  • Authority What a specific program is allow to do, can replace privilege/permission thinking

 

Data

  • PKI Public Key Infrastructure: Public/private key pairs for digital certificate authentication in trusted, secure communication
  • CBOM Certificates, keys & secrets, tokens, crypto libs, protocols, configs

 

Project

Management & Leadership

General leadership, coaching and mentoring, software quality management and management of testing teams.

Leadership

  • Servant - Leadership through selfless empowerment, growth, improvement, moral/values led, action, responsibility
  • Motivation - Measurement management via theory X external vs delegation management via theory Y internal, plus values and relationships
  • Environment - Foster trust, support, happy, fulfilled working conditions for teams
  • Values - A strong set based on strong moral and integral principles to lead by, and to facilitate in teams
  • Moral - Guides core values, direction and decisions towards a greater good with sacrifice, subordination, worthy cause, justification and relationships
  • Institution - Build a model team of shared purpose, support, fulfilment, growth, nobility through shared goals/concepts, discipline, service, structure, trustees
  • First/Action - Show the way, invite followers and lead by example, also consult with team, show patience, speak last but make decision, take first step and accept responsibility for failure
  • Open vs Closed Door: Decision-making by assessing reversibility; if not, take a lot more time or reconsider
  • 37% Rule: Decision-making at statistically the best time “optimal stopping problem” by waiting and observing 37% of the available time
  • Red Team Adversarial thinking from an opposition viewpoint, e.g. competitor, attacker

 

People

  • 1:1 - (One-to-one) Regular, private, non-status update, tester-driven agenda meetings to build human connections, trust and rapport with manager
  • Titles - Individual contributors: apprentice, associate, mid, senior, principal, architect and administrative officers: lead, manager, director

 

Process

  • Process - Process are repeatable tasks to achieve something modellable as flow diagrams
  • Metrics - Assigning numbers to test processes including information or motivation; value vs cost; execution, reporting and setup; dysfunction and construct validity
  • Assurance vs Control - Proactive vs reactive approaches to putting good things in and taking bad things out of testing processes to assure and control testing quality

 

Product

 

Coach & Mentor

  • Mentor - Process where seniors help/assist/teach/advise/guide juniors/interns/apprentices to onboard, upskill, achieve goal, get promoted informally/ad-hoc or part of a programme

Biases & Patterns

Things in the minds of testers that influence perceptions and actions in a way that can work for or against them where awareness, acknowledgement and collaboration helps manage their effects.

  • Hindsight Bias (Knew-it-all-along) Perceiving past occurrences as being more predictable than they were
  • In-Group (Groupthink) Favouriting members of one’s own group as opposed to outsiders regardless of actual value or validity
  • Overconfidence Perception of having greater skills, experience, knowledge that actually posses leading to poor judgement, decisions or underestimation of risk
  • Devil’s Advocate Adversarial stance without personal belief, key for testers and infosec/cybersecurity, but too much can inhibit discussion or turn other’s away

 

Glossary

Terms used within this website’s namespace. These may or may not align to similar terms and meanings in other places.

  • Actor A type of favoured stakeholder, such as a user, team member or group of people, that interacts directly or indirectly with the system-under-test to gain value
  • Actor, Threat A type of disfavoured stakeholder, someone who interacts with the software product or project to gain value for themselves and/or cause lost value and harm to other stakeholders in a way that’s undesirable for project stakeholders. Also “bad” or “malicious”.
  • Bug A negative, threatened relationship with the software and a favoured stakeholder due to an exploited vulnerability and risked loss and harm
  • Check An instance of checking.
  • Checking A type of algorithmic testing of a software application to establish certain facts under certain conditions at some point in time
  • Stakeholder, Favoured Anyone with vested interest in the project and its success in a way that delivers value to them or avoids harm (e.g. actors, such as users, or others, including customers, business stakeholders, governments or the public at large
  • Stakeholder, Disfavoured Someone who the project team doesn’t want to have a vested interest in its success, or who has a vested interest in its failure (e.g. such as threat actors or rival companies)
  • Story A narrative used to evaluate and communicate quality of software to stakeholders of value or risk of lost value and harm (story)
  • Test An instance of testing.
  • Tester Anyone on a software project team who performs testing, or a job title of someone on a project team who’s main responsibility is to do testing
  • Testing Applied, qualitative social research to question, investigate, analyse, hypothesise, experiment, evaluate and communicate information on software product quality for project stakeholders (testing)
  • Threat Someone or something (person, event/scenario) that has the potential to exploit a vulnerability and can cause lost value or harm
  • Project The people, processes and product that come together for the shared goal of delivering value to stakeholders
  • Quality A degree of value to some person (quality)
  • Risk Potential for lost value or harm to someone
  • Risk Acceptance Inaction taken by stakeholders based on information from testing to acknowledge the risk but proceed anyway. To “take the risk”.
  • Risk Analysis An assessment on the seriousness of the risk made up of its likeliness to occur, impact should it occur, and how much we know about the risk
  • Risk Avoidance Action taken by stakeholders based on information from testing to avoid the known risk altogether (e.g. stopping the project)
  • Risk Evaluation Someone’s assessment of a risk in the form of a story that describes how someone may come to lost value or harm
  • Risk Mitigation Action taken by stakeholders based on information from testing to reduce the likeliness or impact of known risk (e.g. fixing a bug
  • Risk Rejection Inaction taken by stakeholders following a further evaluation or analysis in which they deem the risk to be false, without warrant, or not serious enough to consider further. See False Positive.
  • Risk, Residual The amount of risk that remains to the project stakeholders once a risk has been mitigated or transferred
  • Risk Transference Action taken by stakeholders based on information from testing to move the risk to another party (e.g. taking out insurance)
  • Value Software’s ability as a social or cognitive tool or prosthesis to help someone solve a problem, complete a task or achieve a goal. Software’s useful purpose.