At some point over night on 30th September (i.e. the night going into 1st October), we saw a number of Spectrum (Charter) customers stop handling fragmented UDP packets. This has manifested itself in such that the phones of affected customers are no longer receiving UDP SIP INVITE packets which exceed whatever their WAN side MTU is. We've so far had 6 customers report the issue - we can see that the last call on 30th September worked and the first and subsequent calls on 1st October failed. Is anyone aware of an update to CPE devices pushed out that night which may have broken their ability to handle IP fragmentation?
Hey Phil,
At some point over night on 30th September (i.e. the night going into 1st October), we saw a number of Spectrum (Charter) customers stop handling fragmented UDP packets. This has manifested itself in such that the phones of affected customers are no longer receiving UDP SIP INVITE packets which exceed whatever their WAN side MTU is. We've so far had 6 customers report the issue - we can see that the last call on 30th September worked and the first and subsequent calls on 1st October failed.
Is anyone aware of an update to CPE devices pushed out that night which may have broken their ability to handle IP fragmentation?
I don't know anything specific to this case, but you'd serve your best interest to send small enough packets that do not need fragmentation, particularly in the backbone. Even devices often considered SP quality, such as ASR9k, fragment in the linecard CPU, giving very poor service quality compared to sending two packets not fragmented. While we can say this should just work, the reality is, it's not very reliably true and I would not build product or business on the assumption that it works well. -- ++ytti
While we can say this should just work, the reality is, it's not very reliably true and I would not build product or business on the assumption that it works well.
Yup. Understood. We can't get away from sending multi-packet messages. We try our best to keep SIP messages as small as possible though sometimes certain optional features required by customers push it beyond their MTU. We're also starting to see decreasing MTUs as customers deploy various SD-WAN solutions and it's tough to keep up with these when you're already teetering on the edge of what used to be considered a fairly common minimum MTU value. We can, of course, get away from using UDP. We can and do run SIP over TCP and indeed over TLS on TCP though the stateful nature of TCP often makes this undesirable. We see a lot of SIP phone implementations that do not handle TCP connection failures very well and result in a loss of calls for a period of minutes if this happens, as they take a while to notice the connection has dropped and should be re-established. Pros and cons of each. If anyone has any specific information about Spectrum CPE changes or indeed any contacts who may be able to interrogate this internally within Spectrum that would be appreciated.
Hi Phil, Contact me off list with the locations impacted and I will look into it. Jim On 10/2/19, 7:00 AM, "NANOG on behalf of Phil Lavin" <nanog-bounces@nanog.org on behalf of phil.lavin@cloudcall.com> wrote: > While we can say this should just work, the reality is, it's not very reliably true and I would not build product or business on the assumption that it works well. Yup. Understood. We can't get away from sending multi-packet messages. We try our best to keep SIP messages as small as possible though sometimes certain optional features required by customers push it beyond their MTU. We're also starting to see decreasing MTUs as customers deploy various SD-WAN solutions and it's tough to keep up with these when you're already teetering on the edge of what used to be considered a fairly common minimum MTU value. We can, of course, get away from using UDP. We can and do run SIP over TCP and indeed over TLS on TCP though the stateful nature of TCP often makes this undesirable. We see a lot of SIP phone implementations that do not handle TCP connection failures very well and result in a loss of calls for a period of minutes if this happens, as they take a while to notice the connection has dropped and should be re-established. Pros and cons of each. If anyone has any specific information about Spectrum CPE changes or indeed any contacts who may be able to interrogate this internally within Spectrum that would be appreciated. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
hey,
I don't know anything specific to this case, but you'd serve your best interest to send small enough packets that do not need fragmentation, particularly in the backbone.
In this case the SIP invite is already sent fragmented from the source and no fragmentation is required in transit. Someone probably thought "we see a lot of DDOSes with UDP fragments, better drop them altogether". -- tarko
Wait till STIR/SHAKEN is enabled. Were going to see real quickly who isn't handling fragmentation correctly....... On Wed, Oct 2, 2019 at 8:34 AM Saku Ytti <saku@ytti.fi> wrote:
Hey Phil,
At some point over night on 30th September (i.e. the night going into 1st October), we saw a number of Spectrum (Charter) customers stop handling fragmented UDP packets. This has manifested itself in such that the phones of affected customers are no longer receiving UDP SIP INVITE packets which exceed whatever their WAN side MTU is. We've so far had 6 customers report the issue - we can see that the last call on 30th September worked and the first and subsequent calls on 1st October failed.
Is anyone aware of an update to CPE devices pushed out that night which may have broken their ability to handle IP fragmentation?
I don't know anything specific to this case, but you'd serve your best interest to send small enough packets that do not need fragmentation, particularly in the backbone. Even devices often considered SP quality, such as ASR9k, fragment in the linecard CPU, giving very poor service quality compared to sending two packets not fragmented.
While we can say this should just work, the reality is, it's not very reliably true and I would not build product or business on the assumption that it works well. -- ++ytti
At some point over night on 30th September (i.e. the night going into 1st October), we saw a number of Spectrum (Charter) customers stop handling fragmented UDP packets
To bring this thread to a close, Charter kindly investigated and fixed the issue. It was caused by a change to their network that occurred on 1st October. Thanks to those involved for their help.
participants (5)
-
Dovid Bender -
Phil Lavin -
Rampley, Jim F -
Saku Ytti -
Tarko Tikan