The audience of the test plan artifact can vary from organisation to organisation. A test plan with an internal project team as a target audience will be detailed compared to the one produced for external regulator’s consumption (you may decide to keep both artifacts similar, however many times in practice, that isn’t the case). So in this paper, I will explain the details of a test plan and how to make the document useful and efficient.
Irrespective of what methodology is followed within each organisation, a test plan detailing how the overall product or service will be tested and delivered is a key artifact. The delivery can be a combination of internal-only applications OR internal and 3rd party vendor-based applications. In either case, it is important to outline how testing will be conducted for internal applications, 3rd party applications, integration between applications, etc…
We can expand this further by considering the example of agile methodology which an organisation may choose to follow – Agile because this is now the most popular way to develop software.
This methodology, as everyone knows, allows you to deliver in small chunks rather than waiting for a big-bang implementation. Refined user stories are a key part of agile methodology.
Let’s jump into test phases e.g. In Sprint testing. After reviewing scope /user stories, try and define the following as a minimum:
1. Type of Testing required
- Functional & integration testing
- Nonfunctional testing
- User Acceptance Testing
Consider the below points while describing the type of testing:
-
- Scope of testing
- How new functionality being delivered will be tested
- data requirements
2. Clarity of acceptance criteria
- Ensure test acceptance criteria are clearly defined
- Define your test acceptance based on acceptance criteria from user stories
- Ensure acceptance criteria are reviewed and agreed upon by users
3. Test Environment
In many organisations, environments are built by different departments. So, it is vital to define environmental requirements clearly. A meeting to discuss and ensure that your requirements are clear, is always useful:
- Which systems are required to conduct testing?
- What data is required to conduct testing?
- Any other critical activities you expect to be supported (batch runs, monitoring, alerting, etc…)
3rd Party product testing
In case your organisation uses 3rd party vendor products, you will probably be getting releases from vendors. In most cases, you won’t have a say in how ‘vendor sprints’ should operate.
In this case, from a test planning point of view, please refer above points to define the plan first.
As a client of receiving organisation, it is very important to ensure that releases you have received are always working as expected. There has to be a way and criteria the releases should meet for further acceptance into the organisation because you don’t want to spend time/effort in deploying releases across different environments and then find out basics do not work.
One way to deal with this is to ensure there is a pre-defined regression pack that can be executed speedily (in hours) which validates key functionality of the release.
Even better, if such test pack can be executed at vendor side on environment config same as yours. You, together with a vendor can decide what is the minimum quality criteria for release acceptance? These upfront steps will save a lot of time/effort and cost.
Regression testing
For me, this is one of the most important aspects of testing. It is MUST ensure (by validating) previously working ‘un impacted’ functionality is actually working. Any features/user stories/requirements are tested in isolation across different sprints cannot guarantee if previously working functionality is still going to work and hence it is extremely important to have a regression pack that validates at a minimum key pillar of the service. You should invest to automate such a regression pack and ensure it is run as soon as new code is checked-in or at least on a daily basis.
Defining the above key points is applicable for most of the individual test phases.
After coming out of sprint testing or testing of individual releases from 3rd party vendors, the next step would be to integrate the components and validate the functionality, interfaces end to end.
Non-functional testing and Operational Acceptance testing
Non Functional Testing
Many times you are required to validate a product or service from a nonfunctional point of view as soon as key components are sufficiently functionally tested. Typical phases conducted under non-functional testing are:
- Performance testing
- Load testing
- Stress testing
- Security testing
Additional considerations for performance, load, and stress testing are:
- Clarity on what needs measuring
- Ensure load patterns over the time are same as production
- A clear understanding of user pattern
- Infrastructure to be measured and performance monitors
Operational Acceptance Testing
Before the end-to-end service goes live, the IT production / Run team is required to ensure they can ‘run’ the service (familiarisation) and act in case there are failures. so it is important to test monitoring and alerting, validate if issue correction procedures are correct, and other support aspects of the service.
Additionally, many times the following is covered in this phase:
- High availability
- Failover & Disaster recovery
You may want to consider some additional (to the above) key points while considering integration/user acceptance/end to end service testing.
Data requirements across different components
- Lot of time is wasted by not thinking upfront
- What the data requirements for each system is?
- How does that data need to be in sync across systems?
- How to validate the setup to ensure it is ready for detailed testing?
Ensure there is enough time spent on this upfront, else every time integration testing is run, there will be a wastage of efforts, delays in conducting actual testing. Additionally, there could be false defects just because of data rather than functionality.
Test Environment and Support requirements
In many organisations, as code moves from ‘lower environments’ to ‘higher environments’, stricter controls are applied. i.e. only certain groups of people are allowed to do certain things. In such cases, make sure there is an upfront request in place for support.
Run book of daily key activities
This is another powerful artifact. All the key activities to be performed, by who, at what time, etc… can be determined in the runbook. This also can become a status reporting mechanism when such a test phase is running.
Testing sign-off
This is a very important part of project delivery so having an upfront defined and agreed criteria always helps. Liaising with various key stakeholders to define such criteria is key. Reminding the project team of these criteria is also essential as many times during go – no go meetings, such criteria get discussed which sometimes results in unnecessary noise which can easily be avoided.
You should always tweak the overall document as per your organisation, project need. Whatever is listed in this paper should be considered as a guide.
Written by Samir Patki, Head Of QA at European Central Counterparty N.V.