Collectively, everyone in the DevOps chain is responsible for the quality and timeliness of the deliverable.
DevOps is a journey, and while many people focus on tools , culture change to embrace the the DevOps principles often plays a more important role.
Any effort to create CI/CD pipelines should include continuous testing. In a traditional SLDC cycle , QA would get a build which is deployed and would then commence testing. Which may include functional, manual, regression testing, performance, load and UAT testing . The build would ideally sit with the QA, before the QA sign-off on the build.
DevOps is now changing these practices . Collectively, everyone in the chain is responsible for the quality and timeliness of the deliverable.
To achieve such speed and agility, it is important to automate all the testing processes and configure them to run automatically when the deployment is completed. Repetitive testing tasks need to be automated . Automating your organizations integration and acceptance testing can increase confidence in you code while reducing the burden on the QA teams.
What to Automate
Specialized automation testing tools and continuous integration tools are used to achieve this integration. This also necessitates the building of a mature automation testing framework to quickly script new test cases. It i unlikely that you will be able to convert your manual test cases to automated ones overnight.
Unit tests should be a priority as they answer the basic question, of the code intent.
Next would be integration tests of your various subsystems, to test the API’s databases and other systems that do most of the business logic.
Manual , exploratory and UI testing is often the most difficult to automate however the best way to truly test your application end to end.
Some of the changing practices include :
- QA and Dev need to sit together and identify the areas affected due to a particular build and execute those related test cases plus a sanity test pass.
- Unit test automation to achieve high code coverage.
- All the environments required for testing need to be standardized and the deployments must be automated.
- All the pre-testing tasks, clean-ups, post-testing tasks, etc. are automated and aligned with the continuous integration cycle.
- Engineers can add test cases to the QA repository and can configure their automation test cases into the DevOps chain.
- The concept of executing all regression test cases for a test pass is soon becoming obsolete.
- Strategy around testing new features need to be formalized and the interim builds can be supplied to QA who would, in turn, create test scripts and run these automation tests on the interim builds till the code becomes stable enough to be deployed onto the Production environment.
- QA should be able to fire automation testing runs across various cross-platform (and cross-browser in case of web applications) environments.
- Parallel execution of tests helps in reducing time-to-live, which in turn is the crux of a successful DevOps implementation.
- Exit criteria need to be set for each run so that when the results of the tests are fed back to the chain, a go/no-go decision to production is taken.
- Blocker or critical bugs found need to be reported & fixed and passed through the same chain of events before the code is deployed in the production environment
- Treat QA as code gives you a single source of truth
- The test execution should essentially be lean