Are overlays the end of old school networking principles?

Last week I published a post discussing networking principles for hierarchical cloud designs, which prompted a great comment and question from Jason Edelman. He noted how I ignored the impact overlays will have on future cloud architectures, and how I addressed the issue simply in terms of traditional VLANs. In that blog I wanted to focus on fragmentation, but Jason’s question is definitely worth exploring.

First, a couple of definitions, as the word “overlay” is a bit generic. I apply the term “L2 overlay” to any overlay technology that builds virtual L2 networks on top of a physical network. L2 overlays happen to be built on top of L3/L4 “underlays” (i.e. using L3 tunneling), and that leaves enough ambiguity for some marketers to get away with a term like “L3 overlays” to describe them. My issue with stretching the terminology in this manner is that it causes confusion between L2 overlays and a different set of technologies that are more appropriately defined as “L3 overlays.” 

What I call “L3 overlays” are actually technologies that overlay L3 segments on top of a physical network. Real L3 overlays dispense of L2 (i.e. Ethernet) altogether. That makes them more lightweight than L2 overlays: since L3 overlays don’t deal with Ethernet, they don’t need to emulate the Ethernet control and data planes. An L3 overlay virtualizes a point-to-point link between a pair of devices. An L2 overlay typically virtualizes a fully meshed any-to-any L2 network, and yes, it typically attempts to emulate Ethernet, or at least its behavior.

I don’t expect L2 overlays to dramatically change a POD design or to introduce mainstream “POD-less” options. I see how people are trying to shift the balance between L2 and L3 using such technologies, but I do believe that regardless of what technology you use to build subnets, keeping subnets small and using routing across subnets remains the right thing to do. If L3 overlays can enable agile inter-subnet connectivity across PODs, flexibility can be obtained without sacrificing traditional best practices, and subnets (with or without L2 overlays) can be kept small and manageable.

To be practical, we should always justify technology with use cases. I might be too cynical, but I believe that extra flexibility always comes at a cost; does the cost of extra flexibility come with significant enough benefits/ROI? Only use cases can keep us grounded in reality. Sometimes the cost is fully justified, other times it’s not. I completely buy “migration” as a legitimate use case for L2 overlays, but it troubles me that L2 overlays are presented as an “all-or-nothing” alternative to VLANs. They should be able to co-exist with VLANs and be inserted more gently when and where they’re actually needed. Instead, if I want to preserve the ability to do a migration, I need to use VXLAN, NVGRE or STT all the time. To me it’s like forcing me to live in an RV the whole year just because one day I might want to take a road trip. In a way, I wish VXLAN was a bit more like OTV: OTV coexists with VLANs, and doesn’t displace them.

The point I was trying to make in my previous blog is that fragmentation is not the root of all evil, and obsessing about fragmentation isn’t necessary. You just need the right metrics to be in control of the technology choices you make. If someone told you that L2 overlays consume only 2.5% of extra resources on a host, that would probably sound like a negligible amount of overhead. But when one says that a proper POD design can result in 2.5% of resources lost to fragmentation (and only in extreme cases), that same 2.5% suddenly sounds a lot more dramatic. And yet it’s the same amount.

Think about it this way:  when do you stop trying to squeeze more toothpaste out of an empty tube? That’s a fragmentation and ROI problem, too. Sure you might buy a 50-gallon tube of toothpaste so you don’t have to worry about it, but that’s impractical. Or someone one day might invent a machine that sucks the last bit of toothpaste out of a 9 oz. tube, but how much would that be worth? Does the ROI justify using it?

I have to admit I’m still struggling to see the real applicability of L2 overlays in many of the use cases where they’re touted to be better than the old. It might just be a technology maturity problem, but in the back of my mind I’m afraid the industry just might have overreacted, responding to a management plane problem with a {management plane + data plane} solution. We keep hearing about VLANs being the problem that needs fixing (not to pick on anyone, but you can find the latest here). If VLANs were the problem, then VXLAN might be the solution, but here’s a revolutionary thought: what if the problem is Ethernet? If Ethernet is the problem, a good emulation of Ethernet is going to suffer from exactly the same limitations of the original Ethernet. Do VLANs have a control plane and a data plane? No, Ethernet does. (What? VTP? No, seriously…). All VLANs have is an (almost non-existent) management plane. The only significant faults of VLANs are that they’re hard to create/delete/prune and that they’re limited to a maximum of 4,000 partitions of a physical Ethernet network.

The bigger scalability issues are all due to Ethernet, the completely random 6 bytes addresses and the broadcast-based discovery services that have been built on top of it. Anything that emulates Ethernet scales as much as Ethernet does. And if that’s true, IP and routing is what saves us and makes us scale. It helped us scale in the old world, and it still will in this shiny new virtual world. Don’t throw away your old networking books just yet. I bet on IP.