A few years ago I first heard about a system that let you run code in parallel over thousands of machines. It didn’t involve SSH and relied heavily on facts to discover the nodes you were about to run code on. That system is known as the Marionette Collective or mcollective.

Inspired by the idea of device discovery, I wanted to come up with something similar for network devices on my network. The inability to install arbitrary code on today’s traditional vendor hardware, makes this a bit more challenging than I’d like. However, with a little work, you can get a decent system going.

Note: I’m only going to go over some techniques that I’ve used, rather than point out actual code. Unfortunately, the code behind this needs to be scrubbed before I can put it out there publicly.

Finding the devices

In my network, and I’d assume most others, there is a management subnet that can be used for monitoring your devices and basic management. Typically your device is not considered “online” until it is active here first.

Powered with that knowledge, and assuming you have your baseline config installed, you can use a network scanner to find these devices. I chose nmap since it is the one I’m most familiar with, and there are decent libraries out there for interacting with it.

I’m using the python library libnmap to kick off this scan:

nmap -T4 -sn 10.1.1.0/24 10.2.1.0/24

This is a simple ping scan, so you don’t have to wait too long to get your results in. The python library takes care of converting this data into usable objects.

Gathering facts about the devices

Now that you have a list of devices, it’s easy to start gathering data in a for loop. I’m assuming that you have some sort of provisioning workflow, and your workflow includes setting up things like NETCONF, RADIUS, and SNMP. In this case, the easiest protocol to retrieve facts about your device is SNMP.

I like to mimic the facts outlined by Facter, because this is what I use on Linux machines. When all your facts have similar keys, it makes search queries much easier and allows you to return Linux + Network devices in the same query. Here are the basics I like to grab:

• macaddress — associated with your management interface
• fqdn — fully qualified domain name of the device
• serialnumber — serial number of the device
• interfaces — comma separated string of all device interfaces
• boardmanufacturer — vendor name
• location — snmp location of the device
• ipaddress — IP associated with the management interface
• description — sysDescr output from snmp

Of course there are plenty of other things you can gather and insert into a simple dictionary. For some guidance, I recommend taking a look at python-junos-eznc, though making a full NETCONF connection is a bit much when dealing with hundreds of devices.

Storing the data somewhere for querying later

I have a lot of internal systems that were built around storing Linux node information. So because of that, I’m biased towards using Optopus. However, choose a store that works for your environment. I highly recommend something like Elasticsearch, because it is cluster-able and can be replicated across datacenters for availability.

The storage you choose, should be easy to query. Preferably from a REST based API so that it’s easy to use from any language without much need for a full fledged library. You can always opt for a simple relational database like MySQL or PostgreSQL, but it really depends on how you’re going to use this data.

I personally enjoy being able to search for devices in order to perform actions on them. For example, a query such as:

location:tx01 ex2200

Would return all Juniper EX2200 devices in my “tx01" datacenter.

What are you doing for network device discovery?

I realize there are plenty of for-sale solutions out there, but I’m a firm believer of knowing what happens under the hood. You gain tons of flexibility and integration with other parts of the company by building your own systems. I’d be interested in what other people do to solve this problem, and possibly working to release code I’ve already written if there’s enough interest.

Network device discovery and storage was originally published in Network Operations on Medium, where people are continuing the conversation by highlighting and responding to this story.

During the last year of my college education, I fell in love with Open Solaris. That’s not something you hear everyday, but the guys at Sun were and still are smart people. If you’ve ever had the chance to play with Zones, ZFS, or even their handy admin commands, you’d agree. One of the smaller features at the time which was still fairly new, was the Crossbow project.

Crossbow combined with Zones and even Xen virtual machines, allowed you to create a “network in a box.” This is where I learned a lot about how basic layer 3 connectivity worked. It was easy to create virtual NICs and attach them to a virtual switch. These could then be attached to Zones or VMs which acted as routers between other networks. The commands were relatively easy to to use, and made sense.

I bring this up, because I recently read an article by Doug Hanks about Juniper’s new QFX5100.

Everywhere you look there’s virtualization. Hypervisors, NAS, and Containers. What about networking? Some people say VLANs. Others say MPLS. A topic of recent discussion is overlay technologies and VXLAN. But what if it went deeper than that? What would happen if you apply some of the server virtualization to networking? What would the result be?

Under the hood each networking switch has a control plane, which is basically full blown computer. It has a CPU, memory, and local storage. What would happen if the QFX5100 virtualized its control plane? Well, it did.

The QFX5100 natively boots into Linux and uses KVM as a hypervisor to create virtual machines. Junos, the network operating system, runs inside of a VM. At first glance, one might ask what’s the big deal? You just added to layers of abstraction between the switch and Junos. However with abstraction comes the ability to do more than what was previously possible.

This has me extremely excited. I’m a big fan of KVM and have been using it for years. It’s a bit more heavy handed than Solaris Zones, FreeBSD Jails, or Docker, but it’s a start. Being able to run VMs on your network hardware opens the doors for limitless capabilities.

Imagine a day when all hardware vendors had this functionality, and you were able to drop a lightweight Linux installation on your core routing device. You could run something like mcollective in there to execute NETCONF commands in parallel, or gather metrics directly with collectd and pipe them into graphite. Of course, this is assuming the network operating system running along side this VM provides access.

Virtualization If other vendors are doing something similar, I’d love to know about it. I’m hoping this becomes a standard practice soon.

Virtualization on network hardware was originally published in Network Operations on Medium, where people are continuing the conversation by highlighting and responding to this story.

I’ve been working in the tech industry now for about 7 or 8 years. While that’s not a huge amount of time, I have jumped all over the operations spectrum. Starting in a call center fixing AT&T DSL connections, interned at a small IT outsourcing shop, building “private clouds,” doing the DevOps thing, and now I spend a lot of time playing with routers.

Through out my various positions, I’ve always been able to find free and open source software with amazing communities built around them. It’s been easy to submit patches on GitHub, find advice on Freenode, or keep up with the latest news on Reddit (or hacker news).

Maybe I’m not looking hard enough, but it would appear the community around network automation is scattered through a few different medias. There are a lot of smart people with great aspirations, but there seems to be a lack of communal direction. Everything is spread thin between Twitter, Google+ (yes really), and a few small IRC channels.

You’ve got amazing work being done by Jeremy Schulman to bring network devices (junos at the moment) up to par with a standard Linux box. Meaning he’s got something together for deploying brand new devices, puppet running on junos, and his latest work appears to make using NETCONF as simple as possible.

When you’re trying to work on massive amounts of systems in parallel, I came across Jathan McCollum, who’s got a pretty impressive amount of code exposing the CLI and NETCONF over a variety of different vendors.

Digging a bit deeper into network protocol manipulation, you’re likely to run into Thomas Mangin who makes it fairly trivial to modify BGP routes using software.

To be fair, I’ve only been looking into network automation stuff for the last few months. I know there are plenty of other great projects out there, and people talking about white box networking devices, but we need more collaboration and knowledge share. If you’ve done something cool or exicting with the network, be proud of it and get the word out.

Where’s the community for network automation? was originally published in Network Operations on Medium, where people are continuing the conversation by highlighting and responding to this story.