The Software Testing Jungle: what are the right tests for your company ?

The following post has been co-authored with Claudio Chinosi & Mauro Pessina.

The jungle of testing

What is software testing? In a nutshell, software testing is an investigation to provide stakeholders with information about the quality of the product or service under test.

This definition is the key to a whole world that relies on more than 100 testing types and techniques: a jungle where getting lost is natural. The reason of this complexity can be explained by the ISO/IEC 9126, the international standard for software quality evaluation, which defines 6 quality factors to keep in mind HOW evaluating software quality:

• Functionality
• Portability
• Maintainability
• Usability
• Efficiency
• Reliability

Each quality factor has a full set of sub-characteristics. Each characteristic is divided into attributes and each testing technique is aimed toward the evaluation of one or more attributes. Covering the whole set of attributes without losing the specificity of the single test take a large number of different methodologies.

The objectives bring you to the right decision

To meet project deadlines budget availability and time constrains, many factors occur. How to deal with the clients’ objectives and the right testing to execute? This decision is even more complex where testing lifecycle is not precisely defined. Typically, to identify testing objectives, we conduct a workshop session with open questions to have a brainstorming activity where customers highlight their necessities. Then it is a work for testing experts – together with the customer – to understand customer needs and realize whether objectives are correct and, more important, they can be satisfied.

Objectives can be expressed also as software requirements, also splitting them into at least two different layers  of abstraction: business and IT. The  business one defines the opportunity and states a specific need: the business requirement. while The IT layer splits the same requirement into more detailed pieces understood by IT specialists (e.g. business analysts, developers, project manager and testers). Those are named technical requirements (functional or non-functional requirements).

From a pure management standpoint, the business requirements are high level requirements that would be typically understood, as follows:

• business requirement 1 “We need to establish an online portal to give our users a unique point of access”
• business requirement 2 “The portal should allow connection with existing standalone applications”
• business requirement 3 

  “The portal should be able to sustain the same load coming from previous existing standalone applications”

Functional requirements, on the other hand, are very detailed and outline exactly what needs to be done and would typically be read by business analysts, developers and project manager:

• functional requirement 1 (declining “business requirement 1”) “The portal should show up to no more than 5 widgets per webpage”
• functional requirement 2 (declining “business requirement 2”) “The portal interface should be composed by widgets communicating each one with a specified back-end standalone component by means of HTTP over SSL protocol”

Accordingly, non-functional requirements are very detailed and they outline what needs to be tested and would typically be read by project manager and performance testers:

• non-functional requirement 1 (declining “business requirement 3”) “The portal should be able to sustain the same throughput, concurrency and transaction average response time observed on the standalone applications”
• non-functional requirement 2 (declining “business requirement 3”) “Transaction response time for the login should be less then 10 seconds during the peak hour for the 90 percent of the observed samples”

Doing such an activity, it is important to decline business requirements into technical requirements, this simply because without a correct definition there is no way of effectively communicating to stakeholders, to make sure that both parts have the same understanding about objectives and expected outcomes when the project will end.

How to choose the right test

While caring about correct translation of business requirements into technical requirements, we need to deeply consider at least other 3 factors:

1. Use cases (more related to functional requirements)
2. Usage analysis (more related to non-functional requirements)
3. System requirements (derived from the two previous)

Use cases show typical user interaction with the application deployment, describing the complete flow of  an end user functionality. Each use case can include quantitative estimates about user behaviors, useful to determine system requirements for performance, availability, and other qualities of service. Use cases are also the starting point to design the logical architecture.

Usage analysis determines the usage patterns according to the different users type defined during the deployment design. The collected information provides a feedback on the expected load conditions and is later used to determine performance and other system requirements.

System requirements (e.g. availability, latent capacity, security, performance, etc.) describe the quality of service a deployed system must provide to meet the business requirements through business analysis (use cases and usage analysis).

Obviously, what mentioned above describes the generic case in which a brand new software has to be deployed, and the more level of detail is put when dealing with both business and technical requirements, the lower will be the risk of losing information during the overall project.Taking into account the two non-functional requirements previously stated, among all the existing non-functional tests, a load test – a testing technique that puts demand on a system or device and measures its response – would best verify them.

On the other hand, considering the functional requirements stated above, interface tests (to evaluate whether systems or components pass data and control correctly to one another) and GUI software tests (the process of testing a product that uses a graphical user interface, to ensure it meets its written specifications) would best verify them.

Moreover, when new parts of software are released (e.g. new features inserted in an already existing application) or technology refresh occurs (e.g. migration to a new operating system version) other type of tests (e.g. regression tests – that seeks to uncover software errors after changes to the program have been made, by retesting the program) would be required to ensure system non-regression verification.

…so, what is the right decision ?

Software testing is a big jungle, made of more than a hundred types of tests. Overall, a first categorization can be done dividing types of tests into two main macro-areas:  functional and non-functional testing. Three brief use cases were provided to give an understanding about testing, which vary accordingly to objectives. Identifying which type of test best fits customer need is not trivial.

Basically, reasons for the above are:

• Some environments do not provide testing to properly follow software development life-cycle
• Customers can benefit to receive  support , rationalizing their objectives (business requirements)
• Technical requirements have to be strictly connected to business requirements
• Once identified and addressed technical requirements, a choice about the correct test to perform can be done