Creating an effective and portable virtual network function testing framework with end-to-end automation

This Industry Viewpoint was authored by Rajan R N, Sumit Thakur, and Pratik Tidke of Prodapt

As the digital service providers’ (DSPs) ecosystem keeps evolving, the virtual network function (VNF) testing is getting more complex, costlier and time-consuming. Testing activities today consume more than 40% of DSP’s time and efforts in bringing new services online.

Challenges faced by DSPs in testing the virtualized environment

1) Scope of testing has exploded

Variety of network function virtualization (NFV) infrastructure, multi-vendor VNFs and disaggregated layers have dramatically increased the testing scope. Unpredictable interoperability testing results in an explosion of NFV combinations. Moving to a virtualised environment has further increased testing scope with additional test scenarios covering resource allocation, dynamic lifecycle management, multi-tenancy, etc.

2) The complexity of testing is growing with evolving NFV ecosystem

Ensuring similar carrier-grade performance in a virtualized environment comprising of multi-vendor open source and home-grown components is getting more complex. NFV test designs have been evolving to support hybrid architecture. In this scenario, monitoring and evaluating end-to-end testing are more challenging.

3) Frequency of testing has increased exponentially

There are frequent updates in the network functions. This leads to testing and integrating more frequent patches and releases from different VNF vendors. Also, frequent customization of existing test cases is required to suit specific vendor component.

To overcome such challenges, DSPs need to radically change the existing testing methods and shift to a new production model.

VNF testing requires to test entire NFV lifecycle management from concept, integration, launch, operation to retirement. Creating an effective and portable VNF testing framework with end-to-end automation can help DSPs to overcome these complexities.

Major business drivers to create a portable VNF testing framework

• Deliver releases in a safe, timely and agile manner
• Launch new features with stability and without breakages
• Launch services that are performant, even in a heterogeneous environment
• Meet the standards, compliances, and interoperability with other components
• Ensure portability across multiple targets (NFVI + MANO)

Key elements to create an effective and portable VNF testing framework

The below figure highlights the key elements that if embraced, can help DSPs in creating an effective and portable VNF testing framework.

Fig 1: Key elements highlighted in the end-to-end flow of VNF testing framework

These five key elements are further explained in detail below:

1)      Creating a portable and target-independent test automation platform

A portable test automation platform should be agile and flexible enough to fire test cases on different vendor VNFs using different service orchestrators.

Key Steps:

I. Create a plug & play test modeling feature for specific vendor elements

Fig 2: Same set of generic test suite can be fired on multivendor VNFs using different service orchestrators

Every service orchestrator has its own mandatory input list of parameters to fire test case. Creating a plug & play feature requires customized parameter mapping for various vendor’s network service orchestrators. Various middleware tools like Jolt and Kong can be used for parameter mapping in JSON format

II. Create a prebuilt set of master test cases common for all the vendor-specific VNFs

III. Test plan creation option should enable the tester to add or remove test cases as per requirement

These steps can result in creating a fully extensible, reusable and portable test suites, which once built can be run on multiple targets and multiple times. This can be easily integrated with any NFVI MANO stack with minimum changes required to core test suites.

2)      Building keyword-driven test adaptors on top of existing test suites

As the existing NFV ecosystem is evolving rapidly, it requires frequent changes in the test suites. Further, the integration testing comprising of various vendor VNFs and MANO stack is highly complex. Such test steps and logics are not easy to understand and requires higher testing and coding skills resulting in a higher cost to develop, maintain and customize test scripts.

Using a keyword-driven test adaptor can address testing complexities for testers. These test adaptors can be built-in declarative fashion on top of the existing python codes. Tools such as a robot, cucumber, and dovetail can be leveraged to create customized keyword-driven frameworks.

3)      Building test suites compliant to standards specified by a governing body such as ETSI

Ensuring compliance and conformance to certain standards is very crucial while defining the test suites. The ETSI PLUGTEST covers a set of defined standards for interoperability testing and VNFs lifecycle management. These standards specify a set of RESTful protocol specifications and data models fulfilling the requirements specified in ETSI for different interfaces in VNF ecosystem.

4)      Automated CI/CT/CD pipeline to create an agile and reliable testing process

Ensure continuous testing with regular feedback – With the growing complexities in the NFV ecosystem, it is crucial to develop a process of testing early and testing often thereby reducing the wait time for feedback.

Reduce human error factor with E2E automation – Testing of different NFV components, interactions between these components and testing their interfaces need to be completely automated to reduce any manual intervention.

Leveraging tools such as GitLab and Ansible can help in creating automated CI/CD/CT pipeline.

5)      Test monitoring tool to gather key insights for E2E testing

In the end, it is equally important to have monitoring and evaluating a tool that gives a unified view of the end-to-end testing. A holistic user interface should be created that helps to monitor and handle all the basic functions of the testing framework in a single window. The UI should support tabs and functionalities such as test plan, test suites, test execution, test design, reports tab, etc. Also, a report log file must be present for debugging purpose with detailed defect functionality.

Business & operational benefits for DSPs implementing an effective and portable VNF testing framework

• Pays for itself

Cost of testing multiplies given DevOps & release volumes – as many as 10-20 VNFs annually. A well-implemented testing framework saves up to 45% effort on test development & pays for itself within 3-5 release cycles.

• Accelerate time to market

A portable test suite means, same tests can be executed on multiple targets.  i.e. VNFs or orchestrators, cutting testing time & accelerating product launch times in a multi-vendor disaggregated NFV environment.

• The testing team focused on testing

Skills required for testers in evolving NFV environment is reduced significantly. Easy to understand test steps and logics makes testers focused on testing.

• Rapid test development and maintenance

Standardized and easily customizable test suites make the platform fully extensible, reusable, & maintainable

An effective and portable VNF testing framework helps in faster product launch, improved product quality, differentiated product delivery and more reliable product customization resulting in enhanced customer satisfaction.

Authors

• Rajan R N, Head of SDN/NFV Labs & B/OSS Practice, Prodapt
• Sumit Thakur – Manager, Strategic Insights, Prodapt
• Pratik Tidke – Principal Associate, SDN/NFV Practice, Prodapt

If you haven't already, please take our Reader Survey! Just 3 questions to help us better understand who is reading Telecom Ramblings so we can serve you better!

Categories: Industry Viewpoint · NFV