Ansible Network Infrastructure Testing

May 16, 2019

Automated testing is a well understood concept in the world of software development. How could companies like Facebook continuously release new features if every aspect of their platform wasn’t being tested with the use of automation?

With the intersection of DevOps and software development with network engineering, automated testing still isn’t 100% top of mind for all of those embarking on their journey.

Testing the Configuration Build Process

It’s not uncommon to hear network engineers that are now automating forgo writing unit, system, or integration tests to add just one more feature instead (we get these requests from clients). This has to stop. Because there is less software development and more use of tools like Ansible for network automation, it’s not always clear how testing can be accomplished. In this article, we’ll look at how unit testing can be applied to network configuration templating when using Ansible through the use of an open source project called ANIT. Configuration templating has been a common starting point for network automation and now we can apply testing to this phase of generating and building configurations.

Ensuring Your Sending Valid Network Commands

It’s very common to start a network automation journey using YAML files to represent network data, Jinja templates for your configuration templates, and a rendering engine like Python/Ansible, and then a deployment engine like Python/Ansible. If there is one thing we must prevent against in this workflow is sending a wrong or invalid command set to the network device. In order to prevent this, there should be testing done to ensure the YAML data and configuration templates have been fully tested.

For example, you may have VLANs that look like this:

---

vlans:
  - id: 10
    name: web
  - id: 20
    name: app
  - id: 30
    name: db

This data can get used by a template that looks like this:

{% for vlan in vlans %}
vlan {{ vlan['id'] }}
  name {{ vlan['name'] }}
{% endfor %}

This is all good and would generate the following expected configuration:

vlan 10
  name web
vlan 20
  name app
vlan 30
  name db

What if the YAML data looked like this:

---

vlans:
  - id: 10
    name: web
  - id: 20
    name: app
    state: down
  - id: 30
    name: db

The generated configuration would still look like this:

vlan 10
  name web
vlan 20
  name app
vlan 30
  name db

But we really wanted or expected the following:

vlan 10
  name web
vlan 20
  name app
  shutdown
vlan 30
  name db

As you can tell, it’s obvious that the Jinja template didn’t account for state. What if this was interfaces that could have dozens of attributes? How soon would you recognize that the speed, duplex, or multicast attribute was missing?

What if all of your manual testing looked good, but the first person using your template chooses not to use a VLAN name and uses YAML data like this (regardless if you documented what key-value pairs are required or optional):

---

vlans:
  - id: 10
    name: web
  - id: 20
  - id: 30
    name: db

Again, not good. You can tell already from this example, the template would error out with a key error, or in this case with Ansible, an AnsibleUndefinedVariable error. It is telling that even in these basic examples that testing is 100% needed of the commands being generated. This can be done by creating a test case for each data structure in YAML. As a user, you would create the expected commands for each data structure.

This is what ANIT is all about.

Here is a sample summary report from ANIT:

(noxy) ntc@ntc:ansible-build-test (master)$ cat job-summary.txt 
Configuration Build Testing Job Summary
   -> Ansible Version: 2.7.10
   -> Python Version: python3.6
-------------------------------------------------------------

**INFO: FEATURE bgp        TEST: test_01    ---------> PASSED    
**INFO: FEATURE bgp        TEST: test_02    ---------> PASSED    
**INFO: FEATURE vlans      TEST: test_01    ---------> PASSED    
**INFO: FEATURE vlans      TEST: test_02    ---------> FAILED    
--- tests/vlans/test_02/expected_config.cfg
+++ outputs/vlans/test_02/generated_config.cfg
@@ -2,5 +2,5 @@
   name web
 vlan 20
   name app
-vlan 30
+vlan 300
   name db
**INFO: FEATURE vlans      TEST: test_03    ---------> FAILED    
AnsibleUndefinedVariable: 'dict object' has no attribute 'name'

Summary: 5 data files tested

Testing Different Versions of Ansible (and Python)

This is a step in the right direction, but how do you start to certify and test different versions of Ansible (and even Python). Not to go on a tangent here, but we’ve seen very quirky things with this type of testing even with different minor versions of Ansible. ANIT uses nox and makes it super simple to test various versions of Python and Ansible in whatever permutation makes sense for your dev and prod environments.

While ANIT’s focus is around testing the config build process, we’ll look at testing and certifiying playbooks on various versions of Python and Ansible even more in an upcoming blog post.

What is Your Testing Strategy?

Testing needs to be part of any automation strategy. It is critical to be thinking about applying testing in every manner possible regardless of tool at every aspect of a network automation architecture.

For more details on ANIT: https://github.com/networktocode/anit

Happy Automating!

Jason (@jedelman8)