Meet the New Software-Defined Network. Almost the Same as the Old Network.

Occasionally when I wake up, I have some utterly obscure question percolating in my brain. Once, it was about the color of the original, undomesticated carrots (white and purple, it turns out–like their cousins the turnip and the parsnip). I have to go look it up so I can go on about my day without further mental interference, which is why I can tell you, for example, that there is an online Carrot Museum that will tell you everything you want to know about carrots. Also, in Afghanistan they make a fermented beverage from wild carrots. (Thanks, brain!)

Today, I simply had to know what “palimpsest” means. Here’s what it means:

A palimpsest (/ˈpælɪmpsɛst/) is a manuscript page, either from a scroll or a book, from which the text has been either scraped or washed off so that the page can be reused, for another document.[1] …In colloquial usage, the term palimpsest is also used in architecture, archaeology, and geomorphology, to denote an object made or worked upon for one purpose and later reused for another, for example a monumental brass the reverse blank side of which has been re-engraved. [Wikipedia]

SDN is an excellent example of a palimpsest. Let’s take the SDN origin story as gospel: why, indeed, shouldn’t you be able to program a switch the way you can a non-purpose-built computing platform? The short answer is because the networking industry, and the devices it continues to produce and sell, evolved in a certain way, and the world is filled with such devices and vast numbers of people trained to interact with them via communication protocols vs higher-level constructs.

So now we’re elaborating an assortment of higher-level constructs under the common banner of SDN. This does not, however, eliminate existing network concepts, protocols–or the actual networks themselves. We are using all of these things as the foundations upon which software-defined networks will be operated, much as the “Troy” excavated (with dynamite!) by Heinrich Schliemann was, in fact, as many as nine different cities, each of which built upon the decaying foundations of the previous one. And why not? It all works moderately well, and we have lots of people who know how to make it work.

The thing about “adopting” SDN is that it’s a little bit of new technology, but a lot of mindset and process shifts. This, I think, is why SDN is starting to enter a winter of discontent (or if you’re more of a Gartnerian than a Shakespearean–a trough of disillusionment). Networking salespeople and users alike are trained to buy/sell boxes that you plug in and that largely ends the transaction until the support contract comes up for renewal or there’s an opportunity to add in more boxes. SDN controllers today, by contrast, are all at the some-assembly-required stage of evolution. And if your SDN use case is BetterFasterCheaper manageability, it’s very reasonable to question why you should go through a whole bunch of new gymnastics moves to do more or less what you’re already doing in a different way.

It would be a mistake, though, to imagine that the current state of affairs is the de-facto nature of SDN. As controllers mature, they will of course become more plug-and-play. Now that the industry has begun to consolidate around a few leading platforms, we will start to see more packaged SDN applications to run on said controllers. Meanwhile, the bleeding-edge organizations moving towards active deployments now are investing heavily in training NetDevs to build their own applications for proprietary use cases. At Open Networking Summit this spring, and in subsequent media interviews, AT&T indicated that they’re putting tens of thousands of network engineers through specialized coding classes. Some portion of those engineers will eventually go on to jobs at other companies, spreading their new expertise into new soil.

And as I wrote in 2013,

…Today’s discrete controllers will wind up going one of two ways:

  • Down, coupled ever more tightly into existing network operating systems, until eventually they will simply be part of OSes with better northbound APIs than before. This will be especially true of “house” controllers from existing networking vendors.
  • Up, as platforms for emerging ecosystems of network applications. There’s nothing to prevent house controllers from moving in this direction, and for major vendors to develop their own developer ecosystems (well, nothing but mindset and institutional support for such)…

There are those, in fact, who expect that controllers will eventually be packaged with applications such that the (micro)controllers are transparent to the user, as opposed to an independent piece of software to be set up and then integrated with a parade of disparate applications. This scenario will necessitate a truly de facto industry controller platform (we’re far from that yet…) as well as a standardized architecture for peer-to-peer controller communications, and mechanisms for defining order of precedence for operations stemming from different applications.

None of this, however, will change what lies beneath in the foreseeable future. We’ll still have some mix of physical and virtual forwarding devices, managed via some set of protocols–some older, some derivations or extensions of existing ones, some genuinely new ones–because SDN doesn’t necessarily change anything about forwarding architecture, and because of refresh cycles and human operator inertia. It will still be valuable for quite some time to understand how those protocols operate on the devices, even as the preferred method for doing so shifts with controller advances and the general state of administrator skillsets. It’s entirely conceivable that 15-20 years down the road, the idea of monkeying around with networking protocols themselves will seem as arcane to most as being fluent in the inner workings of the systems bus in your PC. But we’ve got a long way to go as an industry before that state of affairs appears on the horizon, and most of that shift will come well after we have some semblance of controller maturity and an SDN application ecosystem in place.

Meanwhile, I’ll apparently be mulling over things like the origins of carrots. Stupid brain.

Stop Hiding. Just Ask.

I came across an exchange on Twitter the other day that made me sad. In part:

Opensource newbie

It made me sad because I spent longer than I should have in a similar place: curious, interested, wanting to know more and also to have input, but being surrounded by people who’d been doing “this stuff’ for 20+ years. I was at a loss as to how I’d ever “catch up”, never mind be in a position to provide value.

Here’s the thing: I wound up in that place because I’d invested a good 20-25 years of my life by that point in making sure I was really competent. Competence and knowing what you’re talking about are things that are valued very highly in my family. Moreover, I grew up surrounded by engineers, who tend to enjoy arguing and sometimes pounce with delight on logical lapses or other types of weak arguments. So from an early age, I shied away from spending a lot of time on things that didn’t come naturally. Early in my career, I did the same: I carefully stayed in roles that didn’t require me to have any specialized knowledge. By my mid-30s, I was in a rather small box–one entirely of my own making.

Now, had anyone else told me I just didn’t have what it takes to do any particular kind of job, I most likely would have furiously plunged into proving them wrong. Instead, I took great care to avoid fulfilling the stereotype of the clueless (female) marketeer: I asked no questions that could be “dumb” questions. I’d quietly go read up on things on my own, but a lot of the time I didn’t have enough context to make total sense of what I read. Still, I avoided asking questions.

What finally did it was screaming boredom. I just didn’t want to keep doing what I’d been doing any more. I decided I’d rather be dumb than bored.

It was terrifying.

Eventually I found a colleague who was working in a similar area, but was coming into marketing after a long time as a engineer. We’ll call him Fred. Every so often I’d walk over to Fred’s cube and announce (pre-emptively, in my mind), “Hi, I have another stupid question,” to which he would helpfully reply, “There are no stupid questions, only stupid people.” And then proceed to explain both the theory and the depressing reality of whatever I was asking about. I have since learned to spot Freds fairly quickly, and over the last few years I’ve built up a wonderful network of Freds. Thanks to social media, most of them are not in a nearby cube, but several states or even a continent away–but that means I get a much broader perspective on whatever I’m curious about than if I just asked the person in the next cube whose vantage point on the world likely only differs from mine by a few degrees.

This brings me to a wonderful talk I unexpectedly attended a few months back. My daughter had been taking a Scratch class through Coder Dojo, and the last night of the session, the parents were all asked to gather in a different room for a pitch from one of the Coder Dojo cofounders, Bill Liao (@liaonet).

Liao opened by stating that coding is poetry, which won me over right there. To be good at poetry, he said, you have to be deeply fluent in a language, in a way you can be only if you speak it from a very young age. It goes without saying, perhaps, that when you learn to speak a language in a fluent, natural way, you do it by interacting with other people. Or as Liao put it,  “I learned to code in my bedroom, and I think working alone in your bedroom is a terrible way to learn to code. I think working with others on a project is a great way to learn to code.”  You can watch an abbreviated version of his talk below.

And this, in turn, brings me to a lesson taught to me by my friend Brent Salisbury (@networkstatic), who became active in OpenDaylight in its very early days. He was trying to convince me to get involved. “But Brent,” I protested, “I haven’t coded in 20 years, and I never got good at it even then.” He told me it didn’t matter. ODL desperately needed documentation. QA. “Play around with Python,” he said. “Hell, I’m dyslexic–if I can code anyone can.” The point being that any project in which people are doing things mostly on their own time–which is most open source projects–desperately needs help of all kinds, and will usually gladly welcome a range of skills, not just narrow technical ones. “I’m just learning how to code, but this is really interesting to me–how can I help?” is as good a door-opener as any.

And of course the way to make book-learning–or any learning–really stick is to try to use it. At Brocade, we have a couple of SEs who have spent most of their careers as network engineers, but are really excited about SDN. They’ve been delving into the documentation–both on the ODL site as well as the materials for my company’s distribution–and making notes about how it works in ways that make more sense to someone accustomed to running a network. I myself had learned about SDN and ODL from a more classical software perspective, so reading what they’ve written was really helpful to me, as they naturally point out very practical nits and concerns that hadn’t occurred to me. Now we’re in the process of converting their work into papers the bigger world can access, which should be helpful to a lot of other network engineers trying to get their heads around this stuff. Note: none of us is writing Java code. All of us are contributing to the growth of an open source effort. It takes a village.

Becoming an expert also takes a village. It’s why throughout history, you find innovation happens in clusters, where talented people gather, get to know each other, and exchange ideas. The lone genius is a myth. Mozart was mentored by Haydn, among others.

Ask for help, and ask how you can help.

Why Open SDN – In 6 Words

In Maslowian terms:

Open SDN by Maslow

Maslow’s SDN – by Lisa Caywood

Interoperability of different device types from different vendors.

Optimization of device selection–for price and performance independently of services features.

Continuous visibility of flows from source to destination.

Informed, centralized manageability for all devices.

Programmability to shape network and network tools according to users’ needs.

Automation of and by policy.

So, What Are WE Going to Do About It?

Well, I hardly imagined there was anything out of the ordinary with my last post; just another conference write-up. But apparently I hit a nerve with regard to my complaint about how most conferences are structured. It seems LOTS of people are fed up—vendors and users alike.

I’ve since had various conversations about this across multiple platforms, so I thought I’d consolidate my views here, in one place, along with my thoughts about what we can do better.

I mean WE in the broadest sense. There’s plenty of unhappiness with the current state of affairs, but for an industry that prides itself on “innovation”, we—sponsors, attendees and event organizers—are mostly just going with the flow laid down for industry conferences ages ago, rather than thinking about how to make them better instead of just bigger.

First, I have no quarrel with vendor sponsorship as such: it costs money to host a large event, and for some organizations, it’s an actual revenue stream. For some (Upperside Conferences, Layer123), it’s their raison d’être. For others, it’s a codified part of the revenue mix (Gartner, for example, reported that 11% of their revenues came from conference sales in 2014). If a tech vendor hosts its own event (eg VMworld, SAPworld, etc), it’s primarily viewed as a marketing activity: less of a revenue stream than an awareness and customer/partner contact event—but still not a small budget item.

As any arts organization can tell you, ticket sales only cover a portion of the cost of an event. The rest has to be made up by deep-pocketed benefactors. And so event organizers, by and large, are just giving their biggest customers what those customers have been trained to want.

And that’s the problem. Not because it’s inherently evil or corrupt—it’s just standard business practice to make your biggest customers happy. The problem is that the deliverables sponsors have been trained to expect, and therefore want, don’t necessarily deliver the kind of value that sponsors actually want or could get. Instead, there are probably better ways to provide value to the largest customers and do more for smaller ones (attendees) at the same time. In other words, I’d like to see more creative sponsorship packages.

For example:

  • Sponsors spend lots of money to obtain booth space, and then lots more money on big fancy booths. Some of us then book juggling or unicycle or magic acts to draw people to our booth, and offer up giveaways as an incentive for badge scans. The sponsor scans some number of badges of people interested in their giveaway. Actual qualified leads that can be used by Sales are typically under 1%. Where there’s genuine interest, ie the attendee has had a meaningful conversation with a qualified booth staffer, the staffer usually gets the attendee’s business card and promptly gets in touch with the relevant salesperson and/or product people at corporate, thereby circumventing the whole lead-gen flow.

Now, events are absolutely a good place for users to meet with a lot of current or potential suppliers in rapid succession. Sometimes good sales reps are proactive about arranging meetings at events. But maybe event organizers could help facilitate the process: all attendees fitting a certain profile automatically get offered meeting matching services with up to 3 vendors of their choice, plus a “wild card” in an area in which they have a general interest (“DevOps tools”, say) which could provide luck-of-the-draw opportunities for small, potentially interesting startups without a lot of name recognition. Or heck, maybe a vendor-user speed dating event.

  • Speaking slots are probably the most valuable portion of any standard sponsorship package, but there’s definitely a right way and a wrong way to do it. More often than not,unfortunately, it’s pure advertising. I’ve even known of individual speakers who tried to do something more interesting than straightforwardly “push the message”, only to be “corrected” by their corporate people. Some events try to encourage or even mandate a user speaker in their sponsor sessions, but even when that’s possible (and it can be remarkably difficult even for a large vendor), the user speaker knows they’re there to help the vendor, and the end result is the same.

On the other hand, event organizers who are in touch with users themselves are in a good position to find good speakers with interesting stories. For example, the 2011 Gartner Data Center conference featured several “How we got to private cloud” sessions, which were all packed. Because the speakers weren’t naming any of their vendors, they were less constrained by internal rules against discussing their deployments. And the knowledge-sharing was widely appreciated. The speakers were typically surrounded 5-deep after their talks.

  • What’s wrong with working sessions?? This is absolutely possible, even at a big event. I expressed by boredom and frustration at the last OpenStack Summit to a friend, who told me he’d been spending the week in an ancillary session that was apparently open to all attendees but completely unadvertised. “It’s awesome! I don’t know why every product person isn’t there. I mean, it’s actual users sitting there telling devs, ‘I want it to do something like this,’ and the dev quickly scripts out a rudimentary idea and they go from there.” People actually talking to each other in a collaborative context to figure out real solutions—wouldn’t that be useful to everyone involved?

What about a little competition? A panel of users poses a problem they’ve been grappling with, gives relevant vendor teams 30-60 mins to draft an idea for a solution, and 3-5 minutes apiece to pitch them. Judging criteria should include points for complete solutions which demonstrate an understanding of the entirety of the customer deployment experience—not just how to build a better mousetrap.

As I said in my last post, I’d even be good with gripe sessions. Customer Advisory Boards tend to be all very polite and not terribly useful because no one wants to be rude to their hosts. But at an independent event? Put together a panel on a given topic (“Identity Management”), invite vendors, and have them sit in the audience where they have to listen and not talk while the panel members talk about what they love, hate and need from their IAM solutions. A slightly less intimate version of the OpenStack hackathon, but also perhaps a bit more scalable. And certainly a refreshing change from the current state of affairs, in which vendors mostly lecture at users.

In general, I’m not fond of things being structured to further the distance between users and vendors. It’s no secret that many user orgs deeply dislike working with a number of their suppliers, but often those same suppliers don’t fully grasp the intensity of dislike, much less the reasons behind it. Wouldn’t it be better if they did, so they could do something about it? Instead, event organizers enable and perpetuate bad marketing habits which are all about unidirectional shouting, instead of a mix of talking and listening. Which make users dislike vendors all the more and further deepens the adversarial, predator/prey dynamic you find in many interactions—especially on the conference show floor.

This isn’t healthy. Nobody really likes working this way. But sponsor expectations will need to be reset. Event organizers will need to show that alternative approaches really can deliver value, and be prepared to articulate the kind of value that is—beyond lead generation. Attendees can be far more vocal to organizers about what they do and don’t want to experience more of.

So—what would work for you? And what can you do about it?

ONUG Spring 2015 & the Cancer of Vendor Sponsorship

Based on feedback from attendees of earlier ONUG events, I was looking forward to this conference. Right now, most of my customer conversations are at the “1a” level (“What is SDN? What does my company mean when we say SDN?”). There are certainly some that are more sophisticated (we have paying customers for our controller, after all), but I was looking forward to a concentrated gathering of users to hear what’s on their minds.

Here are my observations from Day 1:

Speaking of vendors–most of the vendor attendees were small startups, many in the SD-WAN space. Of the established players, Juniper was notably absent. And despite the clear interest in white-box with this group, Cumulus opted not to invest in this event.

There were some bright spots, notably Adrian Cockcroft’s talk about how to DevOps. You can view his slides here. Ted Turner from Intuit (Ananda Rajagopal’s guest) later in the day talked about how Intuit is taking 15-year-old apps and breaking them into microservices, and then leveraging AWS for the non-core portions of the app workflow. The reliance on AWS for a significant portion of one’s IT operations was actually a fairly common theme in both speaker and offline comments.

Unfortunately, the rest of Day 1 was given over to “Working Group test results presentations”, aka the aforementioned vendor panels. Or as Andre Kindness of Forrester put it:

Today the morning sessions were closed to us vendors–fair enough, I suppose…on the one hand, I can appreciate users’ desire for frank discussions without getting aggressive phone calls to their management from certain suppliers. On the other hand, hearing what users are really struggling with (which isn’t always the same thing as engineering’s neat ideas) is always instructive for those of us who might be in a position to help provide solutions. I’d be good with seats in the peanut gallery and a strict behind-the-red-velvet-rope, listen-only policy for vendors for these sessions–especially since there was no bar on media attendees.

The first open session of today featured Najam Ahmad of Facebook. Facebook is doing some very interesting things with their infrastructure, and although it’s certainly not, in its entirety, a blueprint for most other companies, there are always interesting nuggets in their presentations that can be applied in other contexts. For example, when an audience member asked about getting the right skillsets in the networking organization, here’s what he said (slightly paraphrasing):

“We have a free-hacking month once a year. Anyone can go work on any team they’re interested in, and at the end of the month, if they want, they can stay with that team, or go back to their old team. For a while we weren’t getting any of the software engineers. Networking just seemed boring and weird to them. Then we started framing it as a big distributed systems project, and that started drawing them in. Then you have the problem of developers not knowing much about networking and vice versa, so we paired networking people with software devs and had them work together on projects, with the expectation that each would learn from the other. Now everyone on our team codes, to some extent. Some more, some less, but they all have an appreciation for each others’ knowledge and skills.”

There were some luncheon sessions that I couldn’t stay focused on because they mostly had the same problem as Day 1’s panels. This was followed by an investor panel, which mostly focused on two questions:

  1. Will SDN/NFV/whitebox hurt Cisco?
  2. What are the prospects for startups in these areas?

On #1, the view was that if Cisco and other established networking vendors lay down strong strategies now, once we start seeing active deployments a year or so from now, it shouldn’t make a significant difference for them. The hype around SDN of a year or so ago, which appeared to be hurting Cisco’s stock price, has died down as the reality of the length of this transition (10 years, one speaker posited, which seems very likely) has set in, and there’s plenty of time to get the go-to-market execution in place. Interestingly, the institutional investor commented that he didn’t see VMware NSX breaking out beyond the security segmentation use case it’s currently positioning. NSX’s architecture as a vertically integrated software stack anchored to a specific hypervisor probably does limit its use cases more than other, more general-purpose controllers, but given the very long runway the same speaker gave other vendors in the space to work out their strategies, I was left wondering what other knowledge and commentary was behind the comment.

On #2, no one expected any significant moves to IPOs or strong exits in the next year, owing to the expected leadtime on real SDN deployments. One speaker also commented that the bar would likely be higher for IPOs than in past cycles, with an expectation that the company actually be making money before entering into the public markets. That would be refreshing, wouldn’t it?

The final session today was a discussion on DevOps, which was covered very adequately by other bloggers. In fact, here are links to several other live blogs posted by my Networking Field Day friends:

ONUG Spring 2015 Live Blog – Day 1 (Ethan Banks)

ONUG Spring 2015 Live Blog – Day 2 (Ethan Banks)

Tom Hollingsworth/John Herbert – Day 1

Liveblog from ONUG Day 2 (John Herbert)

Now, about the “cancer” comment in the title of this post: I had gone all the way over to the other side of the world (Paris, not horrible) in March to the MPLS/SDN/NFV World Congress, having heard good things about it from past attendees. I heard from the same people (vendors) I’ve heard from many times at conferences in the US. I already know my colleagues’ and competitors’ schticks. I don’t need to hear it again. ONUG turned out to be much the same. I said this to an analyst friend at Gartner Data Center a couple of years ago. I’m dreading ONS at this point. I get that events are revenue opportunities (and for Mr. Lippis of ONUG, a primary one), and also that when vendors pony up large chunks of money to help cover the costs of the events, the marketing departments expect something (visibility, since the value of leads is honestly very negligible) in return. But–YUCK. I know I’m not the only one who’s tired of hearing the same thing at every event. There have to be other, better ways to do this.

Update: Some ideas to get the ball rolling here.

First, Let’s Define the Problem

So, Women in Tech. It’s become a hot topic in the last year. That’s a good thing in many ways. It also means, as happens with most hot topics, that lots of ideas and statistics get thrown around, increasingly willy-nilly and context-free. This is less good, because it tends to make the conversation more heated without producing anything useful as a result, and because it means that while more people can recite depressing stats and thereby convince themselves they’re enlightened, there isn’t actually any deeper understanding of the problem for most.

Which brings me to my next point: which problem, exactly? The real answer of course, is that there are many, and often they’re linked to one another, but far too many articles on the Women in Tech topic recite a litany of statistics (most of which are from studies about women workers in all industries, or write-ups of polls of college students) that relate to a bunch of different problems and then conclude “Tech Hates Women!”

I take issue with this. Continue reading

Books I Read in 2014

Often when you read interviews with business leaders, they get asked about their favorite book, or what they’re currently reading. Invariably the answer is some business tome, which led me to make this observation one day:

Continue reading

Deploying Out of Context: Behavioral Findings

I conducted a practical anthropology experiment recently: I wore an 18th century dress with a long train out to a masquerade-themed event, and then to a private party. Some of the things I learned about how unfamiliar things affect behavior were unexpected, and applicable outside the realm of retro fashion. I detail my findings below.


You get dressed every day. You’ve been doing it for years. You might give some thought in the morning to the design of how you will be presenting yourself that day, but probably almost none to the mechanics of getting dressed and moving about in your clothes.

You can do that because the clothes you own are designed for the social context AND the physical environment in which you operate. But change any of those things and you quickly find you have to adapt to all of them. Continue reading

If content is created, does it have to make a sound?

Amy Lewis just published a post I like a lot, “Consume and Be Thankful“. The executive summary is “Shut up and listen”. In other words, publishing content should not be the sole focus of being a community member, much less a leader. This is exactly the opposite approach advocated lately by Om Malik, Greg Ferro and others, challenging themselves and others to publish 30 blogs in 30 days. Continue reading

Reflections on Armistice Day

I’m just off the plane from France, where I combined the OpenStack Summit with a family vacation. As we landed, the American flight attendant explained to the plane that it was Veteran’s Day in the U.S., which means that we thank our veterans for their service. I was bitterly amused.


In Europe November 11th is Armistice Day. It celebrates the day peace was made at the end of the “War To End All Wars”–WWI. When I lived in the Pas de Calais, where most of it was fought, it was still living memory for old folks, a somber day of wearing black and visiting graves, and remembering the horrors of that time, and being thankful for peace. I found it horribly ironic that this woman, who clearly had no notion of the history of this particular date, felt she was positioned to teach half the plane about it. Continue reading