Flow based architecture in data centers(more specifically Telco Clouds)
Hi, Are most of the Telco Cloud deployments envisioned to be modeled on a flow based or a non flow based architecture? I am presuming that for deeper insights into the traffic one would need a flow based architecture, but that can have scale issues (# of flows, flow setup rates, etc) and was hence checking. Thanks, Glen
Please explain for us dumb sales guys the distinction between flow and non-flow. My question is the fundamental architecture of these clouds. We all know that Amazon is buying dark fiber and building a network based on lighting 100 and 10 gig waves on IRU and titled fiber. Same for Microsoft (I sold them in a past life some waves) and other large players. But there appear to be quite a few cloud players that rely heavily on Layer 3 purchased from Level3 (CenturyLink) and other members of the august Tier 1 club. And many CDN players are really transit + real estate operations as was Akamai until recently. It seems the threshold for moving from purchased transit plus peering to a Layer 1 and 2 network has risen over time. Many former Tier 2 ISPs pretty much gutted their private line networks as transit prices continued inexorable declines. Best, Roderick. ________________________________ From: NANOG <nanog-bounces@nanog.org> on behalf of Glen Kent <glen.kent@gmail.com> Sent: Sunday, February 9, 2020 11:02 AM To: nanog@nanog.org <nanog@nanog.org> Subject: Flow based architecture in data centers(more specifically Telco Clouds) Hi, Are most of the Telco Cloud deployments envisioned to be modeled on a flow based or a non flow based architecture? I am presuming that for deeper insights into the traffic one would need a flow based architecture, but that can have scale issues (# of flows, flow setup rates, etc) and was hence checking. Thanks, Glen
(caution, I'm just a chemical engineer, but) You appear to ask one question: "What is the difference between flow and non-flow architectures?" then sideline in some discussion about fiber/waves vs layer-3/transit/peering/x-connect I don't think the second part really relates to the first part of your message. (I didn't put this content in-line because .. it's mostly trying to clarify what you are asking Rod" On Sun, Feb 9, 2020 at 3:19 AM Rod Beck <rod.beck@unitedcablecompany.com> wrote:
Please explain for us dumb sales guys the distinction between flow and non-flow. My question is the fundamental architecture of these clouds. We all know that Amazon is buying dark fiber and building a network based on lighting 100 and 10 gig waves on IRU and titled fiber. Same for Microsoft (I sold them in a past life some waves) and other large players.
But there appear to be quite a few cloud players that rely heavily on Layer 3 purchased from Level3 (CenturyLink) and other members of the august Tier 1 club. And many CDN players are really transit + real estate operations as was Akamai until recently.
It seems the threshold for moving from purchased transit plus peering to a Layer 1 and 2 network has risen over time. Many former Tier 2 ISPs pretty much gutted their private line networks as transit prices continued inexorable declines.
Best,
Roderick.
________________________________ From: NANOG <nanog-bounces@nanog.org> on behalf of Glen Kent <glen.kent@gmail.com> Sent: Sunday, February 9, 2020 11:02 AM To: nanog@nanog.org <nanog@nanog.org> Subject: Flow based architecture in data centers(more specifically Telco Clouds)
Hi,
Are most of the Telco Cloud deployments envisioned to be modeled on a flow based or a non flow based architecture? I am presuming that for deeper insights into the traffic one would need a flow based architecture, but that can have scale issues (# of flows, flow setup rates, etc) and was hence checking.
Thanks, Glen
They don't have to be related. 🙂 I am curious about the distinction about the flow versus non-flow architecture for data centers and I am also fascinated by the separate issue of WAN architecture for these clouds. Regards, Roderick. ________________________________ From: Christopher Morrow <morrowc.lists@gmail.com> Sent: Sunday, February 9, 2020 9:24 PM To: Rod Beck <rod.beck@unitedcablecompany.com> Cc: Glen Kent <glen.kent@gmail.com>; nanog@nanog.org <nanog@nanog.org> Subject: Re: Flow based architecture in data centers(more specifically Telco Clouds) (caution, I'm just a chemical engineer, but) You appear to ask one question: "What is the difference between flow and non-flow architectures?" then sideline in some discussion about fiber/waves vs layer-3/transit/peering/x-connect I don't think the second part really relates to the first part of your message. (I didn't put this content in-line because .. it's mostly trying to clarify what you are asking Rod" On Sun, Feb 9, 2020 at 3:19 AM Rod Beck <rod.beck@unitedcablecompany.com> wrote:
Please explain for us dumb sales guys the distinction between flow and non-flow. My question is the fundamental architecture of these clouds. We all know that Amazon is buying dark fiber and building a network based on lighting 100 and 10 gig waves on IRU and titled fiber. Same for Microsoft (I sold them in a past life some waves) and other large players.
But there appear to be quite a few cloud players that rely heavily on Layer 3 purchased from Level3 (CenturyLink) and other members of the august Tier 1 club. And many CDN players are really transit + real estate operations as was Akamai until recently.
It seems the threshold for moving from purchased transit plus peering to a Layer 1 and 2 network has risen over time. Many former Tier 2 ISPs pretty much gutted their private line networks as transit prices continued inexorable declines.
Best,
Roderick.
________________________________ From: NANOG <nanog-bounces@nanog.org> on behalf of Glen Kent <glen.kent@gmail.com> Sent: Sunday, February 9, 2020 11:02 AM To: nanog@nanog.org <nanog@nanog.org> Subject: Flow based architecture in data centers(more specifically Telco Clouds)
Hi,
Are most of the Telco Cloud deployments envisioned to be modeled on a flow based or a non flow based architecture? I am presuming that for deeper insights into the traffic one would need a flow based architecture, but that can have scale issues (# of flows, flow setup rates, etc) and was hence checking.
Thanks, Glen
On Sun, Feb 9, 2020 at 1:06 PM Rod Beck <rod.beck@unitedcablecompany.com> wrote:
They don't have to be related. 🙂
makes a cogent conversation harder :)
I am curious about the distinction about the flow versus non-flow architecture for data centers and I am also fascinated by the separate issue of WAN architecture for these clouds.
WAN is probably: "least expensive option form A to B" plus some effort to standardize across your deployment. Right? Akamai is probably a good example, from what I can tell they were 'transit/peering only' until they realized their product was sending 'more bits' between deployments than to customers (in some cases). So, pushing the 'between our deployments' bits over dedicated links (be that dark, waves, other L3 transport) made sense budget-wise. (again.. just a chemical engineer and not a peering engineer, but...)
Regards,
Roderick.
------------------------------ *From:* Christopher Morrow <morrowc.lists@gmail.com> *Sent:* Sunday, February 9, 2020 9:24 PM *To:* Rod Beck <rod.beck@unitedcablecompany.com> *Cc:* Glen Kent <glen.kent@gmail.com>; nanog@nanog.org <nanog@nanog.org> *Subject:* Re: Flow based architecture in data centers(more specifically Telco Clouds)
(caution, I'm just a chemical engineer, but)
You appear to ask one question: "What is the difference between flow and non-flow architectures?" then sideline in some discussion about fiber/waves vs layer-3/transit/peering/x-connect
I don't think the second part really relates to the first part of your message. (I didn't put this content in-line because .. it's mostly trying to clarify what you are asking Rod"
On Sun, Feb 9, 2020 at 3:19 AM Rod Beck <rod.beck@unitedcablecompany.com> wrote:
Please explain for us dumb sales guys the distinction between flow and
non-flow. My question is the fundamental architecture of these clouds. We all know that Amazon is buying dark fiber and building a network based on lighting 100 and 10 gig waves on IRU and titled fiber. Same for Microsoft (I sold them in a past life some waves) and other large players.
But there appear to be quite a few cloud players that rely heavily on
Layer 3 purchased from Level3 (CenturyLink) and other members of the august Tier 1 club. And many CDN players are really transit + real estate operations as was Akamai until recently.
It seems the threshold for moving from purchased transit plus peering to
a Layer 1 and 2 network has risen over time. Many former Tier 2 ISPs pretty much gutted their private line networks as transit prices continued inexorable declines.
Best,
Roderick.
________________________________ From: NANOG <nanog-bounces@nanog.org> on behalf of Glen Kent <
Sent: Sunday, February 9, 2020 11:02 AM To: nanog@nanog.org <nanog@nanog.org> Subject: Flow based architecture in data centers(more specifically Telco Clouds)
Hi,
Are most of the Telco Cloud deployments envisioned to be modeled on a flow based or a non flow based architecture? I am presuming that for deeper insights into the traffic one would need a flow based architecture, but
glen.kent@gmail.com> that can have scale issues (# of flows, flow setup rates, etc) and was hence checking.
Thanks, Glen
On Sun, Feb 9, 2020 at 4:15 PM Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Sun, Feb 9, 2020 at 1:06 PM Rod Beck <rod.beck@unitedcablecompany.com> wrote:
They don't have to be related.
makes a cogent conversation harder :)
Srsly?! Any conversation including Cogent is harder.... W (Sorry, couldn't resist. I tried, but failed...)
I am curious about the distinction about the flow versus non-flow architecture for data centers and I am also fascinated by the separate issue of WAN architecture for these clouds.
WAN is probably: "least expensive option form A to B" plus some effort to standardize across your deployment. Right?
Akamai is probably a good example, from what I can tell they were 'transit/peering only' until they realized their product was sending 'more bits' between deployments than to customers (in some cases). So, pushing the 'between our deployments' bits over dedicated links (be that dark, waves, other L3 transport) made sense budget-wise.
(again.. just a chemical engineer and not a peering engineer, but...)
Regards,
Roderick.
________________________________ From: Christopher Morrow <morrowc.lists@gmail.com> Sent: Sunday, February 9, 2020 9:24 PM To: Rod Beck <rod.beck@unitedcablecompany.com> Cc: Glen Kent <glen.kent@gmail.com>; nanog@nanog.org <nanog@nanog.org> Subject: Re: Flow based architecture in data centers(more specifically Telco Clouds)
(caution, I'm just a chemical engineer, but)
You appear to ask one question: "What is the difference between flow and non-flow architectures?" then sideline in some discussion about fiber/waves vs layer-3/transit/peering/x-connect
I don't think the second part really relates to the first part of your message. (I didn't put this content in-line because .. it's mostly trying to clarify what you are asking Rod"
On Sun, Feb 9, 2020 at 3:19 AM Rod Beck <rod.beck@unitedcablecompany.com> wrote:
Please explain for us dumb sales guys the distinction between flow and non-flow. My question is the fundamental architecture of these clouds. We all know that Amazon is buying dark fiber and building a network based on lighting 100 and 10 gig waves on IRU and titled fiber. Same for Microsoft (I sold them in a past life some waves) and other large players.
But there appear to be quite a few cloud players that rely heavily on Layer 3 purchased from Level3 (CenturyLink) and other members of the august Tier 1 club. And many CDN players are really transit + real estate operations as was Akamai until recently.
It seems the threshold for moving from purchased transit plus peering to a Layer 1 and 2 network has risen over time. Many former Tier 2 ISPs pretty much gutted their private line networks as transit prices continued inexorable declines.
Best,
Roderick.
________________________________ From: NANOG <nanog-bounces@nanog.org> on behalf of Glen Kent <glen.kent@gmail.com> Sent: Sunday, February 9, 2020 11:02 AM To: nanog@nanog.org <nanog@nanog.org> Subject: Flow based architecture in data centers(more specifically Telco Clouds)
Hi,
Are most of the Telco Cloud deployments envisioned to be modeled on a flow based or a non flow based architecture? I am presuming that for deeper insights into the traffic one would need a flow based architecture, but that can have scale issues (# of flows, flow setup rates, etc) and was hence checking.
Thanks, Glen
-- I don't think the execution is relevant when it was obviously a bad idea in the first place. This is like putting rabid weasels in your pants, and later expressing regret at having chosen those particular rabid weasels and that pair of pants. ---maf
On Sun, 9 Feb 2020 at 23:09, Rod Beck <rod.beck@unitedcablecompany.com> wrote:
I am curious about the distinction about the flow versus non-flow architecture for data centers and I am also fascinated by the separate issue of WAN architecture for these
Based on the context of the OP's question, he is talking about architecture where some components, potentially network devices, are flow-aware, instead of doing LPM lookup per packet, they are doing LPM lookup per flow. This comes up every few years in various formats, because with flow-lookup you have one expensive LPM lookup per flow and multiple cheap LEM lookups. However the LEM table size is unbounded and easily abusable leading to a set of very complex problems. There are of course a lot of variation what OP might mean. Network might be for example entirely LEM lookup with extremely small table, by using stack of MPLS labels, zero LPM lookups. This architecture could be made so that when server needs to send something say video to a client, it asks orchestration for permission, telling I need to send x GB to DADDR K with rate at least Z and no more than Y, orchestration could then tell the server to start sending at time T0 and impose MPLS label stack of [l1, l2, l3, l4, l5] Orchestration would know exactly which links traffic traverses, how long will it be utilised and how much free capacity there is. Network would be extremely dumb, no IP lookups ever, only thousands of MPLS labels in FIB, so entirely on-chip lookups of trivial cost. -- ++ytti
Hi, On Mon, Feb 10, 2020 at 3:22 PM Saku Ytti <saku@ytti.fi> wrote:
On Sun, 9 Feb 2020 at 23:09, Rod Beck <rod.beck@unitedcablecompany.com> wrote:
I am curious about the distinction about the flow versus non-flow architecture for data centers and I am also fascinated by the separate issue of WAN architecture for these
Based on the context of the OP's question, he is talking about architecture where some components, potentially network devices, are flow-aware, instead of doing LPM lookup per packet, they are doing LPM lookup per flow.
This was exactly my question.
This comes up every few years in various formats, because with flow-lookup you have one expensive LPM lookup per flow and multiple cheap LEM lookups. However the LEM table size is unbounded and easily abusable leading to a set of very complex problems.
There are of course a lot of variation what OP might mean. Network might be for example entirely LEM lookup with extremely small table, by using stack of MPLS labels, zero LPM lookups. This architecture could be made so that when server needs to send something say video to a client, it asks orchestration for permission, telling I need to send x GB to DADDR K with rate at least Z and no more than Y, orchestration could then tell the server to start sending at time T0 and impose MPLS label stack of [l1, l2, l3, l4, l5]
Orchestration would know exactly which links traffic traverses, how long will it be utilised and how much free capacity there is. Network would be extremely dumb, no IP lookups ever, only thousands of MPLS labels in FIB, so entirely on-chip lookups of trivial cost.
My question was rather simple. Many cloud operators use Open Vswitch (OVS) based dataplanes wherein each packet results in a new flow in the system. The first packet does a lookup in the slow path which causes the fast path (either OVS-DPDK or a smartNIC or VPP-like paradigm or something entirely different) to be programmed. All subsequent packets hit the fast path and reach the VM (which is hosting the VNF). The advantage of this scheme is that the operator knows the exact flows existing in their data center and can run some sort of analytics on that. This obviously becomes harder once you start aggregating the flows or with mega flows, but you hopefully get the drift. The other architecture is based on LPM and LEM lookups. BTW, when i spoke about the Telco Cloud i had meant pure software based routing. NO hardware. No baremetals and physical network functions. I had pure VNFs in my mind, I see that Mellanox (smartNIC) is also programmed using flows. Hence is it fair to say that most of the current telco cloud architectures are built around OVS style flow based network devices? Glen
-- ++ytti
participants (5)
-
Christopher Morrow -
Glen Kent -
Rod Beck -
Saku Ytti -
Warren Kumari