October 5, 2021
A few months ago, Network to Code released two open source projects (more info) to contribute to solving a common problem in modern networks: understand when a circuit is going through a planned maintenance. On one side, the circuit-maintenance-parser
, to parse notifications into a common data structure, and on the other side, the nautobot-circuit-maintenance
plugin, that uses the parser library to automatically populate the data into Nautobot. Following months of development, we are happy to announce the release of version 2.0.0
of the parser, which comes with a lot of improvements based on existing customer deployments and now covers 19 different ISPs!
circuit-maintenance-parser
Library
In the mentioned blog, we acknowledged that we were not the first ones trying to solve this issue. We decided to adopt the format proposed here (using the iCalendar
format) as our gold standard.
Now, you could be wondering: why do we need a parser library when there is a well-defined format? The answer is well-known… being just a recommendation it is not fully adopted by all the Network Service Providers (NSPs). So there is still a need to parse arbitrary data formats in order to obtain something conforming to a standard Maintenance
containing the following attributes:
- provider: identifies the provider of the service that is the subject of the maintenance notification.
- account: identifies an account associated with the service that is the subject of the maintenance notification.
- maintenance_id: contains text that uniquely identifies the maintenance that is the subject of the notification.
- circuits: list of circuits affected by the maintenance notification and their specific impact.
- status: defines the overall status or confirmation for the maintenance.
- start: timestamp that defines the start date of the maintenance in GMT.
- end: timestamp that defines the end date of the maintenance in GMT.
- stamp: timestamp that defines the update date of the maintenance in GMT.
- organizer: defines the contact information included in the original notification.
Please, refer to the BCOP to more details about these attributes.
This library aims to fill the current gap between an ideal representation of a circuit maintenance notification and the current providers’ formats, enabling automation to be built around these notifications.
The first version of the circuit-maintenance-parser
had a simple workflow that eventually became a blocker to solve more complex use-cases
and this required a new middleware that could combine multiple data using custom logic to process composed notifications, and be able to accommodate future new use-cases. More details about the logic workflow is available in the library Readme.
Supported Providers
Obviously, one of the key success indicators of the library is how many Providers
are supported, and thanks to multiple examples seen from early adopters, the supported providers list has increased to 19 providers and it’s growing quickly.
- AquaComms
- AWS
- Cogent
- Colt
- EuNetworks
- GTT
- HGC
- Lumen
- Megaport
- Momentum
- NTT
- PacketFabric
- Seaborn
- Sparkle
- Telia
- Telstra
- Turkcell
- Verizon
- Zayo
Moreover, the gold format is supported by default with the
GenericProvider
, so any NSP that sends the notification with theiCalendar
format is supported by default.
How to Use It?
The circuit_maintenance_parser
library requires two things:
- The
notificationdata
: this is the data that the library will check to extract the maintenance notifications. It can be simple (only one data type and content, such as an iCalendar notification) or more complex (with multiple data parts of different types, such as from an email). - The
provider
identifier: used to select the requiredProvider
. EachProvider
contains the logic to process thenotificationdata
using associated parsers.
Python Library
First step is to define the Provider
that we will use to parse the notifications. By default, the GenericProvider
(used when no other provider type is defined) will support parsing of iCalendar
notifications using the recommended format.
from circuit_maintenance_parser import init_provider
generic_provider = init_provider()
type(generic_provider)
<class 'circuit_maintenance_parser.provider.GenericProvider'>
However, usually some Providers
don’t fully implement the standard. Or perhaps some information is missing, for example the organizer
email. We also support custom defined Providers
that can be used to tailor the data extraction based on the notifications your organization receives :
ntt_provider = init_provider("ntt")
type(ntt_provider)
<class 'circuit_maintenance_parser.provider.NTT'>
Once we have the Provider
ready, we need to initialize the data to process. We call it NotificationData
and can be initialized from a simple content and type or from more complex structures, such as an email.
from circuit_maintenance_parser import NotificationData
raw_data = b"""BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Maint Note//https://github.com/maint-notification//
BEGIN:VEVENT
SUMMARY:Maint Note Example
DTSTART;VALUE=DATE-TIME:20151010T080000Z
DTEND;VALUE=DATE-TIME:20151010T100000Z
DTSTAMP;VALUE=DATE-TIME:20151010T001000Z
UID:42
SEQUENCE:1
X-MAINTNOTE-PROVIDER:example.com
X-MAINTNOTE-ACCOUNT:137.035999173
X-MAINTNOTE-MAINTENANCE-ID:WorkOrder-31415
X-MAINTNOTE-IMPACT:OUTAGE
X-MAINTNOTE-OBJECT-ID;X-MAINTNOTE-OBJECT-IMPACT=NO-IMPACT:acme-widgets-as-a-service
X-MAINTNOTE-OBJECT-ID;X-MAINTNOTE-OBJECT-IMPACT=OUTAGE:acme-widgets-as-a-service-2
X-MAINTNOTE-STATUS:TENTATIVE
ORGANIZER;CN="Example NOC":mailto:noone@example.com
END:VEVENT
END:VCALENDAR
"""
data_to_process = NotificationData.init_from_raw("ical", raw_data)
type(data_to_process)
<class 'circuit_maintenance_parser.data.NotificationData'>
Finally, we retrieve the maintenances (it is a List
because a notification can contain multiple maintenances) from the data calling the get_maintenances
method from the Provider
instance:
maintenances = generic_provider.get_maintenances(data_to_process)
print(maintenances[0].to_json())
{
"account": "137.035999173",
"circuits": [
{
"circuit_id": "acme-widgets-as-a-service",
"impact": "NO-IMPACT"
},
{
"circuit_id": "acme-widgets-as-a-service-2",
"impact": "OUTAGE"
}
],
"end": 1444471200,
"maintenance_id": "WorkOrder-31415",
"organizer": "mailto:noone@example.com",
"provider": "example.com",
"sequence": 1,
"stamp": 1444435800,
"start": 1444464000,
"status": "TENTATIVE",
"summary": "Maint Note Example",
"uid": "42"
}
Notice that, either with the GenericProvider
or NTT
provider, we get the same result from the same parsed data, because they are using exactly the same Processor
and Parser
. The only difference is that NTT Provider
will provide some custom default values for NTT in case the notification doesn’t contain this data. In this case, the notification contains all the information, so the custom defaults for Provider
are not used.
ntt_maintenances = ntt_provider.get_maintenances(data_to_process)
assert ntt_maintenances == maintenances
CLI
There is also a cli
entrypoint circuit-maintenance-parser
which offers easy access to the library using few arguments:
data-file
: file storing the notification.data-type
:ical
,html
oremail
, depending on the data type.provider-type
: to choose the rightProvider
. If empty, theGenericProvider
is used.
circuit-maintenance-parser --data-file "/tmp/___ZAYO TTN-00000000 Planned MAINTENANCE NOTIFICATION___.eml" --data-type email --provider-type zayo
Circuit Maintenance Notification #0
{
"account": "my_account",
"circuits": [
{
"circuit_id": "/OGYX/000000/ /ZYO /",
"impact": "OUTAGE"
}
],
"end": 1601035200,
"maintenance_id": "TTN-00000000",
"organizer": "mr@zayo.com",
"provider": "zayo",
"sequence": 1,
"stamp": 1599436800,
"start": 1601017200,
"status": "CONFIRMED",
"summary": "Zayo will implement planned maintenance to troubleshoot and restore degraded span",
"uid": "0"
}
How to Extend the Library?
Even though the library aims to include support for as many providers as possible, it’s likely that not all the thousands of NSP are supported and you may need to add support for some new one. Adding a new Provider
is quite straightforward, and in the following example we are adding support for an imaginary provider, ABCDE, that uses HTML notifications.
First step is creating a new file: circuit_maintenance_parser/parsers/abcde.py
. This file will contain all the custom parsers needed for the provider and it will import the base classes for each parser type from circuit_maintenance_parser.parser
. In the example, we only need to import Html
and in the child class implement the methods required by the class, in this case parse_html()
which will return a dict
with all the data that this Parser
can extract. In this case we have to helper methods, _parse_bs
and _parse_tables
that implement the logic to navigate the notification data.
from typing import Dict
import bs4 # type: ignore
from bs4.element import ResultSet # type: ignore
from circuit_maintenance_parser.parser import Html
class HtmlParserABCDE1(Html):
def parse_html(self, soup: ResultSet) -> Dict:
data = {}
self._parse_bs(soup.find_all("b"), data)
self._parse_tables(soup.find_all("table"), data)
return [data]
def _parse_bs(self, btags: ResultSet, data: Dict):
...
def _parse_tables(self, tables: ResultSet, data: Dict):
...
Next step is to create the new Provider
by defining a new class in circuit_maintenance_parser/provider.py
. This class that inherits from GenericProvider
only needs to define two attributes:
_processors
: is alist
ofProcessor
instances that uses several dataParsers
. In this example, we don’t need to create a new customProcessor
because the combined logic serves well (the most likely case), and we only need to use the new definedHtmlParserABCDE1
and also the genericEmailDateParser
that extract the email date. Also notice that you could have multipleProcessors
with differentParsers
in this list, supporting several formats._default_organizer
: this is a default helper to fill theorganizer
attribute in theMaintenance
if the information is not part of the original notification._include_filter
: mapping ofdata_types
to a list of regex expressions that if provided have to match to parse the notification. This feature removes noise from notifications that are received from the same provider, but that are not related to circuit maintenance notifications._exclude_filter
: antagonist mapping to define via regex which are the notifications that must not parsed.
class ABCDE(GenericProvider):
_processors: List[GenericProcessor] = [
CombinedProcessor(data_parsers=[EmailDateParser, HtmlParserABCDE1]),
]
_default_organizer = "noc@abcde.com"
_include_filter = {EMAIL_HEADER_SUBJECT: ["Scheduled Maintenance"]}
And expose the new Provider
in circuit_maintenance_parser/__init__.py
:
from .provider import (
GenericProvider,
ABCDE,
...
)
SUPPORTED_PROVIDERS = (
GenericProvider,
ABCDE,
...
)
Last, but not least, you should update the tests!
- Test the new
Parser
intests/unit/test_parsers.py
- Test the new
Provider
logic intests/unit/test_e2e.py
… adding the necessary data samples in tests/unit/data/abcde/
.
What’s Next?
Give it a try!, as the community is growing more and more Providers
are going to be added and you can benefit from all of them. Also, developing a new Provider
or Parser
is straightforward, and you can contribute to the library via Pull Requests
or Issues
, providing notifications samples to develop.
As showed in How to use it?, you can easily integrate it with any automation application only passing the notification data and selecting the Provider
that should be used to parse it, and then do what you want with the structured output.
And don’t forget that you could get this integrated with Nautobot SoT using the Circuit Maintenance Plugin.
-Christian