Simon Mayes - Climber and Software Engineer

It’s been my observation that engineers don’t talk about Software Quality, they get stuck in word-spaghetti soup talking about the numerous types of automated tests covering various characteristics; security, availability, accessibility etc. Each set of characteristics has different stakeholders; InfoSec team, compliance team, UX team etc. and the importance of each tends to be political within the organisation. I want to focus on Software Quality in the holistic sense and suggest some clear language to describe the discipline.

The first issue is; what are we talking about? We are talking about Software Quality a simple, self titled phrase that encapsulates everything. There is no need to rebrand this term for your specific organisation.

mindmap
  root((Software Quality))
    )Engineering Excellence(
    )Service Maturity(
    )Immunisation(
    )Software Standards(
    )Engineering Playbook(
    )Compliance(
    )Checks(
    )Software Health(
    )Service Readiness(

Various organisational terms that might be synonyms for Software Quality.

All this word-spaghetti just complicates things. It then further complicated by external stakeholders. You often find that experienced engineers organically introduce tools over time but once your organisation hits a certain point you spin up InfoSec teams, Compliance teams, Risk teams, UX teams and start caring about Info and Data security, compliance, risk, performance and accessibility as separate disciplines. The list can then be endless. Those teams then speak to vendors, senior leaders and purchase tools, which may not be suitable but are subsequently mandated for use. This introduces politics and distances the concern of software quality from the engineers to the external stakeholders and sometimes results the engineers developing the behaviour of - “it’s a tick box I must fulfil before production”.

The cherry on the cake is the when people start talking about tools - which are brand names and frequently have no meaning by themselves. “Make sure your [Aqua|Wiz|Sonarqube|k6|Lighthouse] scans are passing.” All of this increases the cognitive load on engineers and is mumbo-jumbo and foreign to new engineers. How are they meant to know all the different tools you use and critically, what is the value and importance of each?

Software Quality as a term is important as it is something every engineer should care deeply about. It transcends all technologies, tools and levels of experience. All these organisational behaviours complicate things and have the effect to change engineers’ behaviours to turn software quality (not that you call it that) into a separate tick-box exercise. In some cases as engineers don’t care about some of the checks they think, “how can we bypass this” or “can we get an exception for that”.

So what is the solution? The first step is to adopt precise, universal and relatively plain language based on industry standards.

You then need to define what are these stated and implied needs? A established way to describe this is through requirements - separated out into Functional and Non-functional requirements.

Functional requirements are easy - they are typically what are defined in the user story/ticket: “An actor performs an interaction and then software then responds in a defined way.” Non-functional requirements are frequently implied and specify how the product will perform: “The response will be within 200ms”.

Now to define what software quality characteristics to talk about. I recommend that you follow ISO/IEC 25010:2023 - which specifies the following characteristics:

Functional suitability

capability of a product to provide functions that meet stated and implied needs intended users when it is used under specified conditions

Sub-characteristics: functional completeness, functional correctness, and functional appropriateness.
Testing approaches: unit tests, integration tests, acceptance tests, end-to-end tests
Stakeholders: End-users, product owners, engineers, UX

Performance efficiency

capability of a product to perform its functions within specified time and throughput parameters and be efficient in the use of resources under specified conditions

Sub-characteristics: time behaviour, resource utilisation, and capacity.
Testing approaches: load tests, stress tests, volume tests, spike tests, resource monitoring
Stakeholders: End-users, on-call engineers, DevOps

Compatibility

capability of a product to exchange information with other produces, and/or to perform its required functions while sharing the same common environment and resources

Sub-characteristics: co-existence, and interoperability.
Testing approaches: API tests, contract tests, environment tests, interoperability tests
Stakeholders: System integrators, engineers, DevOps

Interaction capability (aka Usability)

capability of a product to be interacted with by specified users to exchange information between a user and a system via the user interface to complete the intended task

Sub-characteristics: appropriateness recognisability, learnability, operability, user error protection, user engagement, inclusivity, user assistance, and self-descriptiveness.
Testing approaches: usability tests, accessibility tests, user journey tests, A/B tests
Stakeholders: End-users, UX and QA

Reliability

capability of a product to perform specified functions under specified conditions for a specified period of time without interruptions and failures

Sub-characteristics: faultlessness, availability, fault tolerance, and recoverability.
Testing approaches: chaos tests, failover tests, recovery tests, availability monitoring
Stakeholders: End-users, DevOps, QA

Security

capability of a product to protect information and data so that persons or other products have the degree of data access appropriate to their types and levels of authorisation, and to defend against attack patterns by malicious actors

Sub-characteristics: confidentiality, integrity, non-repudiation, accountability, authenticity, and resistance.
Testing approaches: penetration tests, vulnerability scans, security audits, threat modeling
Stakeholders: End-users, security, compliance, risk, engineers

Maintainability

capability of a product to be modified by the intended maintainers with effectiveness and efficiency

Sub-characteristics: modularity, reusability, analysability, modifiability, and testability.
Testing approaches: static analysis, code reviews, complexity analysis, test coverage analysis
Stakeholders: Engineers, DevOps

Flexibility (formerly Portability)

capability of a product to be adapted to changes in its requirements, context of use, or system environment

Sub-characteristics: adaptability, scalability, installability, and replaceability.
Testing approaches: scalability tests, deployment tests, migration tests, configuration tests
Stakeholders: Engineers, DevOps

Safety (new)

capability of a product under defined conditions to avoid a state in which human life, health, property or the environment is endangered

Sub-characteristics: operational constraint, risk identification, fail safe, hazard warning, and safe integration.
Testing approaches: boundary tests, fault injection tests, model verification, hazard analysis
Stakeholders: End-users, safety engineers, compliance, risk

The final steps are to take the characteristics and turn them into measurable Software Quality Attributes (SQAs). SQAs enable you to create a Software Quality framework (SQF) that you can practically use. Industry standards are excellent guidelines but they don’t tell you how to actually execute them. A SQF will factor in software lifecycle, environment, software criticality, etc.

Adopting industry standard language will bring huge benefits across the board. It will empower engineers to understand and own domain of Software Quality. These standards, though formal, provide clear, precise definitions - functional requirements, non-functional requirements and software quality attributes - will improve communication and enable engineers better scope work. They will enable engineers to use open-source resources/documentation and AI-driven tools more effectively to develop their understanding. Clarity of language will speed onboarding, aid design decisions, and reduce miscommunication. Ultimately, improved software quality will reduce incidents, enhances customer satisfaction, and improves the company bottom line.

Over the next few weeks I will continue this software quality deep dive and publish a practical Software Quality Framework using open-source tools.