Confirming source-routed multicast is dead on the public Internet
Its tought to prove a negative. I'm extremely confident the answer is yes, public internet multicast is not viable. I did all the google searches, check all the usual CAIDA and ISP sites. IP Multicast is used on private enterprise networks, and some ISPs use it for some closed services. I got sent back with a random comment from a senior official saying "but I heard different." I bit my tongue, and said I would double (now quadruple) check. If any ISPs have working IP source-routed multicast on the public Internet that I missed, or what I got wrong. That's what content distribution networks (cdn's) are for instead.
On Jul 31, 2018, at 2:28 PM, Sean Donelan <sean@donelan.com> wrote:
Its tough to prove a negative. I'm extremely confident the answer is yes, public internet multicast is not viable.
From a technical perspective, yeah, that’s right, but as you say, tough to prove a negative. If you want to give them a “why” it doesn’t work, Zhi-Li Zhang and I wrote that up fifteen years ago, when it became evident that it wasn’t going to happen. https://www.pch.net/resources/Papers/multicast-billing/multicast-billing-v10... -Bill
On Jul 31, 2018, at 2:28 PM, Sean Donelan <sean@donelan.com> wrote:
Its tough to prove a negative. I'm extremely confident the answer is yes, public internet multicast is not viable.
From a technical perspective, yeah, that’s right, but as you say, tough to prove a negative. If you want to give them a “why” it doesn’t work, Zhi-Li Zhang and I wrote that up fifteen years ago, when it became evident that it wasn’t going to happen. https://www.pch.net/resources/Papers/multicast-billing/multicast-billing-v10... -Bill
other than billing problem, is there any other reasons why multicast would not be viable for public internet ? Mankamana
On Jul 31, 2018, at 2:36 PM, Bill Woodcock <woody@pch.net> wrote:
On Jul 31, 2018, at 2:28 PM, Sean Donelan <sean@donelan.com> wrote:
Its tough to prove a negative. I'm extremely confident the answer is yes, public internet multicast is not viable.
From a technical perspective, yeah, that’s right, but as you say, tough to prove a negative. If you want to give them a “why” it doesn’t work, Zhi-Li Zhang and I wrote that up fifteen years ago, when it became evident that it wasn’t going to happen.
https://www.pch.net/resources/Papers/multicast-billing/multicast-billing-v10...
-Bill
Thus spake Mankamana Mishra (mankamis) via NANOG (nanog@nanog.org) on Wed, Aug 01, 2018 at 02:43:10AM +0000:
other than billing problem, is there any other reasons why multicast would not be viable for public internet ?
Hi Mankamana, You can find a reasonable overview here with respect to ASM: https://tools.ietf.org/html/draft-acg-mboned-deprecate-interdomain-asm-02 Dale
Hey Mankamana,
other than billing problem, is there any other reasons why multicast would not be viable for public internet ?
Imagine someone like youtube or netflix would like to use multicast, instead of caches. They'd need to start new multicast stream for every content with small delay (to get more viewers on given stream), how much delay would consumer tolerate before content starts? 1min? 5min? So every minute or every 5 minute new stream of movie would be sent, except it would need to be sent many times, for each bitrate supported. Each of these streams is wide (wider than unicast) HW state that needs to be stored on every device on path, for unicast we only store 1 narrow HW state per destination, for multicast we store 1 wide HW state per flow/stream, we don't have the hardware to do that, if there would be any significant demand for multicast. It only works when there is no use-case for it, and even then, it's insecure DoS vector. -- ++ytti
What if... Bear with me for a moment here, we don't try to force VoD onto a multicast setup? Multicast is used extensively by all major ISPs(if they have the rights) to deliver IPTV. One issue you brought up is people unwillin to wait 1 or 5 mins for a show, well before the days of youtube people waited weeks for OTA programming that started with or without delay, depending on how many relays you were going through. As a use case of multicast over the internet, a Real time TV rebroadcaster would be a really good use case. The federal govt already subsidises super expensive energy hogging TV broadcast towers, who's to say they wouldn't prefer it to just go over the interwebs? Bit rate's not a problem, a 720i stream takes 1 or 2 mbps, which is a fraction of a home broadband connection(25mbps down 3 up, last time i checked). I think we all on nanog would agree internet is more important than TV. Govt money might better be spent on a better internet than TV radios. Of course that might mean some internet backbone upgrades(maybe even govt subsidised upgrade), but i would never say that there isn't a commercial use case for it. On 1 August 2018 at 10:24, Saku Ytti <saku@ytti.fi> wrote:
Hey Mankamana,
other than billing problem, is there any other reasons why multicast would not be viable for public internet ?
Imagine someone like youtube or netflix would like to use multicast, instead of caches. They'd need to start new multicast stream for every content with small delay (to get more viewers on given stream), how much delay would consumer tolerate before content starts? 1min? 5min? So every minute or every 5 minute new stream of movie would be sent, except it would need to be sent many times, for each bitrate supported. Each of these streams is wide (wider than unicast) HW state that needs to be stored on every device on path, for unicast we only store 1 narrow HW state per destination, for multicast we store 1 wide HW state per flow/stream, we don't have the hardware to do that, if there would be any significant demand for multicast. It only works when there is no use-case for it, and even then, it's insecure DoS vector.
-- ++ytti
hey,
What if... Bear with me for a moment here, we don't try to force VoD onto a multicast setup? Multicast is used extensively by all major ISPs(if they have the rights) to deliver IPTV.
We are an IPTV provider in europe and we definetly see share of linear TV (that we are delivering via intra-AS multicast today) decreasing YOY. OTT plays a big part but even more customers use our own on-demand services including network PVR. Numbers don't add up yet but in 3-4 years even the intra-AS multicast will not make sense for us any more, easier/better to deliver everything via unicast. -- tarko
On 1/Aug/18 19:43, Tarko Tikan wrote:
We are an IPTV provider in europe and we definetly see share of linear TV (that we are delivering via intra-AS multicast today) decreasing YOY.
OTT plays a big part but even more customers use our own on-demand services including network PVR.
Numbers don't add up yet but in 3-4 years even the intra-AS multicast will not make sense for us any more, easier/better to deliver everything via unicast.
I fully expect this to trend. In Africa, linear (pay) TV is on the down every month, and the only reason it's mostly still seeing some patronage is because of the sports channels which "cannot be" unbundled from the full package. The other reason it's still marginally alive is because of customers that can't spare enough "data" to do the majority of their entertainment online. I know several power users that have deliberately cut the linear TV cord, have gone fully VoD, and have managed to make arrangements to stream sports shows in other ways. The leading pay TV operator in sub-Saharan Africa, Multichoice (DStv) are thinking about their future and are providing the ability for customers to stream all of their channels online, either in real-time or as post-aired VoD content. They've also launched ShowMax, which is their offering to compete with Netflix, Hulu, e.t.c. I was in Malaysia the other day, and Astro are definitely struggling with the declining demand in their pay TV service (both satellite- and IPTV-based). I'm with Tarko when I say in 5 or so years, Multicast for IPTV will begin to die off because customers will be moving more to VoD use-cases. Mark.
On 8/1/18 12:24 PM, Saku Ytti wrote:
Hey Mankamana,
other than billing problem, is there any other reasons why multicast would not be viable for public internet ? Imagine someone like youtube or netflix would like to use multicast, instead of caches. They'd need to start new multicast stream for every content with small delay (to get more viewers on given stream), how much delay would consumer tolerate before content starts? 1min? 5min? So every minute or every 5 minute new stream of movie would be sent, except it would need to be sent many times, for each bitrate supported. Each of these streams is wide (wider than unicast) HW state that needs to be stored on every device on path, for unicast we only store 1 narrow HW state per destination, for multicast we store 1 wide HW state per flow/stream, we don't have the hardware to do that, if there would be any significant demand for multicast. It only works when there is no use-case for it, and even then, it's insecure DoS vector.
Personally, I think the use case is a lot stronger for multi-person video conferencing. Miles Fidelman -- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra
Hey Miles and Michael, It is entirely fair to debate what the use-case would be, the underlaying technical problem remains the same, if it becomes useful (globally) we don't have the hardware to cater for it. I'm sure both of your use cases are used extensively in internal network. I've worked for company who distributed broadcast TV on their MPLS IP backbone, two-plane network, red and blue, one copy for each tv channel on both planes and far-end drops redundant copy. Very attractive solution commercially for client and provider. But no potential for global scale. On Wed, 1 Aug 2018 at 20:44, Miles Fidelman <mfidelman@meetinghouse.net> wrote:
On 8/1/18 12:24 PM, Saku Ytti wrote:
Hey Mankamana,
other than billing problem, is there any other reasons why multicast would not be viable for public internet ? Imagine someone like youtube or netflix would like to use multicast, instead of caches. They'd need to start new multicast stream for every content with small delay (to get more viewers on given stream), how much delay would consumer tolerate before content starts? 1min? 5min? So every minute or every 5 minute new stream of movie would be sent, except it would need to be sent many times, for each bitrate supported. Each of these streams is wide (wider than unicast) HW state that needs to be stored on every device on path, for unicast we only store 1 narrow HW state per destination, for multicast we store 1 wide HW state per flow/stream, we don't have the hardware to do that, if there would be any significant demand for multicast. It only works when there is no use-case for it, and even then, it's insecure DoS vector.
Personally, I think the use case is a lot stronger for multi-person video conferencing.
Miles Fidelman
-- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra
-- ++ytti
On Wed, 1 Aug 2018 at 20:47, Saku Ytti <saku@ytti.fi> wrote:
I'm sure both of your use cases are used extensively in internal network. I've worked for company who distributed broadcast TV on their MPLS IP backbone, two-plane network, red and blue, one copy for each tv channel on both planes and far-end drops redundant copy. Very attractive solution commercially for client and provider. But no potential for global scale.
Just to clarify what I mean by 'broadcast TV', these streams were not received by set-top boxes receiving IP packets. End-users were cable/antennae users, 'client' was receiving the content at antennae site to be transmitted over air. Classically these are separate purpose built networks, which are very expensive, so in this particular case everyone won, except the legacy provider who was billing for the purpose built network. -- ++ytti
On Tue, Jul 31, 2018 at 5:28 PM, Sean Donelan <sean@donelan.com> wrote:
Its tought to prove a negative. I'm extremely confident the answer is yes, public internet multicast is not viable. I did all the google searches, check all the usual CAIDA and ISP sites. IP Multicast is used on private enterprise networks, and some ISPs use it for some closed services.
I got sent back with a random comment from a senior official saying "but I heard different." I bit my tongue, and said I would double (now quadruple) check.
If any ISPs have working IP source-routed multicast on the public Internet that I missed, or what I got wrong. That's what content distribution networks (cdn's) are for instead.
Hi Sean, I'm sure that transit providers with whom you have no commercial relationship are happy to let you program their routers to forward your multicast packets. Not just happy, ecstatic I'd say. Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Dirtside Systems ......... Web: <http://www.dirtside.com/>
On Tue, 31 Jul 2018 at 23:29, Sean Donelan <sean@donelan.com> wrote:
Its tought to prove a negative. I'm extremely confident the answer is yes, public internet multicast is not viable. I did all the google searches, check all the usual CAIDA and ISP sites. IP Multicast is used on private enterprise networks, and some ISPs use it for some closed services.
I got sent back with a random comment from a senior official saying "but I heard different." I bit my tongue, and said I would double (now quadruple) check.
If any ISPs have working IP source-routed multicast on the public Internet that I missed, or what I got wrong. That's what content distribution networks (cdn's) are for instead.
AS 2914 is working to fully dismantle all its Internet multicast related infrastructure and configs. All MSDP sessions have been turned off, we have deny-all filters for the multicast AFI, and the RPs have been shut down. For years we haven’t seen actual legit multicast traffic. Also the multicast “Default-Free Zone” has always been severely partitioned. Not all the players were peering with each other, which led to significant complexity for any potential multicast source. Reasoning behind turning it off is that it limits the attack surface (multicast can bring quite some state to the core), reduces the things we need to test and qualify, and by taking this off the RFPs we can perhaps consider more vendors. However, as you noted; multicast within a single administrative domain (such as an access network distributing linear TV), or confined to purpose-built L3VPNs very much is a thing. On the public Internet multicast seems dead. Kind regards, Job
It is hard to prove a negative. So let’s prove a positive. One of the largest (2nd largest?) transit networks on the planet just affirmatively stated they filter at their border. It is now possible to state that multicast is not ubiquitous on the Internet. If any other large transit network (L3, GTT, HE, Cogent, etc.) would like to confirm they filter at their borders as well, that would put the final nail in the coffin. -- TTFN, patrick
On Jul 31, 2018, at 15:15, Job Snijders <job@ntt.net> wrote:
On Tue, 31 Jul 2018 at 23:29, Sean Donelan <sean@donelan.com> wrote:
Its tought to prove a negative. I'm extremely confident the answer is yes, public internet multicast is not viable. I did all the google searches, check all the usual CAIDA and ISP sites. IP Multicast is used on private enterprise networks, and some ISPs use it for some closed services.
I got sent back with a random comment from a senior official saying "but I heard different." I bit my tongue, and said I would double (now quadruple) check.
If any ISPs have working IP source-routed multicast on the public Internet that I missed, or what I got wrong. That's what content distribution networks (cdn's) are for instead.
AS 2914 is working to fully dismantle all its Internet multicast related infrastructure and configs. All MSDP sessions have been turned off, we have deny-all filters for the multicast AFI, and the RPs have been shut down.
For years we haven’t seen actual legit multicast traffic. Also the multicast “Default-Free Zone” has always been severely partitioned. Not all the players were peering with each other, which led to significant complexity for any potential multicast source.
Reasoning behind turning it off is that it limits the attack surface (multicast can bring quite some state to the core), reduces the things we need to test and qualify, and by taking this off the RFPs we can perhaps consider more vendors.
However, as you noted; multicast within a single administrative domain (such as an access network distributing linear TV), or confined to purpose-built L3VPNs very much is a thing. On the public Internet multicast seems dead.
Kind regards,
Job
I can confirm that GTT does indeed filter IP sourced from 224.0.0.0/4 at its edge. On 7/31/2018 6:44 PM, Patrick W. Gilmore wrote:
It is hard to prove a negative.
So let’s prove a positive. One of the largest (2nd largest?) transit networks on the planet just affirmatively stated they filter at their border. It is now possible to state that multicast is not ubiquitous on the Internet.
If any other large transit network (L3, GTT, HE, Cogent, etc.) would like to confirm they filter at their borders as well, that would put the final nail in the coffin.
As you all have said, to confirm, I use ssm Mcast to distribute TV from satellite down links in the headend, out to a few different remote head ends. From there it's converted back to RF video and sent to subscribers via cable or hfc plant Aaron
On Jul 31, 2018, at 5:15 PM, Job Snijders <job@ntt.net> wrote:
On Tue, 31 Jul 2018 at 23:29, Sean Donelan <sean@donelan.com> wrote:
Its tought to prove a negative. I'm extremely confident the answer is yes, public internet multicast is not viable. I did all the google searches, check all the usual CAIDA and ISP sites. IP Multicast is used on private enterprise networks, and some ISPs use it for some closed services.
I got sent back with a random comment from a senior official saying "but I heard different." I bit my tongue, and said I would double (now quadruple) check.
If any ISPs have working IP source-routed multicast on the public Internet that I missed, or what I got wrong. That's what content distribution networks (cdn's) are for instead.
AS 2914 is working to fully dismantle all its Internet multicast related infrastructure and configs. All MSDP sessions have been turned off, we have deny-all filters for the multicast AFI, and the RPs have been shut down.
For years we haven’t seen actual legit multicast traffic. Also the multicast “Default-Free Zone” has always been severely partitioned. Not all the players were peering with each other, which led to significant complexity for any potential multicast source.
Reasoning behind turning it off is that it limits the attack surface (multicast can bring quite some state to the core), reduces the things we need to test and qualify, and by taking this off the RFPs we can perhaps consider more vendors.
However, as you noted; multicast within a single administrative domain (such as an access network distributing linear TV), or confined to purpose-built L3VPNs very much is a thing. On the public Internet multicast seems dead.
Kind regards,
Job
Can your hfc customers do an igmp join? No? Then it's probably not considered "public". -Steve On Wed, Aug 1, 2018 at 5:21 AM Aaron Gould <aaron1@gvtc.com> wrote:
As you all have said, to confirm, I use ssm Mcast to distribute TV from satellite down links in the headend, out to a few different remote head ends. From there it's converted back to RF video and sent to subscribers via cable or hfc plant
Aaron
On Jul 31, 2018, at 5:15 PM, Job Snijders <job@ntt.net> wrote:
On Tue, 31 Jul 2018 at 23:29, Sean Donelan <sean@donelan.com> wrote:
Its tought to prove a negative. I'm extremely confident the answer is yes, public internet multicast is not viable. I did all the google searches, check all the usual CAIDA and ISP sites. IP Multicast is used on private enterprise networks, and some ISPs use it for some closed services.
I got sent back with a random comment from a senior official saying "but I heard different." I bit my tongue, and said I would double (now quadruple) check.
If any ISPs have working IP source-routed multicast on the public Internet that I missed, or what I got wrong. That's what content distribution networks (cdn's) are for instead.
AS 2914 is working to fully dismantle all its Internet multicast related infrastructure and configs. All MSDP sessions have been turned off, we have deny-all filters for the multicast AFI, and the RPs have been shut down.
For years we haven’t seen actual legit multicast traffic. Also the multicast “Default-Free Zone” has always been severely partitioned. Not all the players were peering with each other, which led to significant complexity for any potential multicast source.
Reasoning behind turning it off is that it limits the attack surface (multicast can bring quite some state to the core), reduces the things we need to test and qualify, and by taking this off the RFPs we can perhaps consider more vendors.
However, as you noted; multicast within a single administrative domain (such as an access network distributing linear TV), or confined to purpose-built L3VPNs very much is a thing. On the public Internet multicast seems dead.
Kind regards,
Job
On Wed, 1 Aug 2018, Aaron Gould wrote:
As you all have said, to confirm, I use ssm Mcast to distribute TV from satellite down links in the headend, out to a few different remote head ends. From there it's converted back to RF video and sent to subscribers via cable or hfc plant
I'm aware that multicast is used extensively for "closed enterprise" networks in the financial and media industries. It seems to work well when a single organization is paying for the entire network. My executive official came from that background, so I get why someone from that world would think multicast is widely used. Asking enterprise network sales people, they keep saying Yes, of course we support multicast. That's why I wanted to hear from public Internet engineers if multicast was still viable on the *public Internet*.
On Wed, Aug 1, 2018 at 11:27 Sean Donelan <sean@donelan.com> wrote:
On Wed, 1 Aug 2018, Aaron Gould wrote:
As you all have said, to confirm, I use ssm Mcast to distribute TV from satellite down links in the headend, out to a few different remote head ends. From there it's converted back to RF video and sent to subscribers via cable or hfc plant
I'm aware that multicast is used extensively for "closed enterprise" networks in the financial and media industries. It seems to work well when a single organization is paying for the entire network.
My executive official came from that background, so I get why someone from that world would think multicast is widely used. Asking enterprise network sales people, they keep saying Yes, of course we support multicast.
That's why I wanted to hear from public Internet engineers if multicast was still viable on the *public Internet*.
I'd say no - even though we have done inter-AS multicast before, it's only been with our direct peers.
On 1/Aug/18 00:15, Job Snijders wrote:
However, as you noted; multicast within a single administrative domain (such as an access network distributing linear TV), or confined to purpose-built L3VPNs very much is a thing. On the public Internet multicast seems dead.
I'd concur. Mark.
Thanks to everyone that helped. Off-list I heard from network engineers at several global Internet providers. They all confirmed that multicast is no longer supported on their public Internet backbones, no matter what their sales people might say. If someone opened a multicast trouble ticket, likely no one in their NOC would know what to do with it. They acknowledged there might be some old multicast configs on Internet facing routers, and eventually they'll clean those out. Backbone network engineers are fairly blunt. Multicast is being used in various private IP networks. It seems to work very well for satellite content distribution because multicast doesn't require ack's. Enterprise networks also use multicast.
In article <nycvar.OFS.7.76.4444.1808021118080.22714@cnex.qbaryna.pbz> you write:
Multicast is being used in various private IP networks. It seems to work very well for satellite content distribution because multicast doesn't require ack's. Enterprise networks also use multicast.
I would think it'd work fine on private networks, but since there's no authentication, on the public Internet how could you tell the multicast you want from random malicious junk on the same IP address?
On Thu, 2 Aug 2018, John Levine wrote:
In article <nycvar.OFS.7.76.4444.1808021118080.22714@cnex.qbaryna.pbz> you write:
Multicast is being used in various private IP networks. It seems to work very well for satellite content distribution because multicast doesn't require ack's. Enterprise networks also use multicast.
I would think it'd work fine on private networks, but since there's no authentication, on the public Internet how could you tell the multicast you want from random malicious junk on the same IP address?
They use some type of encryption to authenticate the data. Satellite distribution networks usually encrypt both the satellite signal so only authorized receivers get the download. The multicast data files are also separately encrypted/signed/checked. On private/enterprise networks, I guess they just trust its a controlled network. On the public Internet. Gosh darn, I don't know, shrug?
participants (17)
-
Aaron Gould -
Adam Davenport -
Bill Woodcock -
Dale W. Carder -
Hunter Fuller -
Job Snijders -
John Levine -
Mankamana Mishra (mankamis) -
Mark Tinka -
Michael Crapse -
Miles Fidelman -
Patrick W. Gilmore -
Saku Ytti -
Sean Donelan -
Steve Meuse -
Tarko Tikan -
William Herrin