Hey all, anyone aware of issues with Arelion this morning? We have a bunch of end users on at least Cox and Cogeco who are having serious issues with service access, and the problem appears to be on the return path where it traverses Arelion. Source net is Lumen/L3 3356 but loss/latency doesn’t appear to creep up until already within Arelion’s network. Imperva mentioned outage/degradation due to a national ISP issue too, so I suspect it’s affecting more than just certain peerings. Thanks, David
We had the same issue with Arelion this morning, a huge increase in the latency and jitter was happening across the US and of course for any traffic sourced from the EU flowing to the US. Within the US we noticed a latency spike from 20-50ms to ~200ms. From the EU to our US facilities, we noticed a spike from ~130ms to ~400ms. We ended up shifting traffic from Arelion to a different carrier, and will stay there until we are confident that they fixed any issue that they had in their network. To be honest, we've been experiencing quite similar issues with unstable latency on Arelion over the past couple of months, and it's becoming quite frustrating *Andrian Visnevschi* VP of Network & Security +373 68374133 andrian@acreto.io | https://acreto.io On Tue, Oct 24, 2023 at 6:03 PM David Hubbard <dhubbard@dino.hostasaurus.com> wrote:
Hey all, anyone aware of issues with Arelion this morning? We have a bunch of end users on at least Cox and Cogeco who are having serious issues with service access, and the problem appears to be on the return path where it traverses Arelion. Source net is Lumen/L3 3356 but loss/latency doesn’t appear to creep up until already within Arelion’s network.
Imperva mentioned outage/degradation due to a national ISP issue too, so I suspect it’s affecting more than just certain peerings.
Thanks,
David
We also observed this today, UTC morning, esp. across the pond, and our alternative paths from Europe to the US were suffering from the still unfixed fibre cut, so I was a bit unhappy with 200 extra ms, and 60% loss. 1299 seems to've found an alternative path in the meantime, looks good to us. Elmar. nanog@nanog.org (Andrian Visnevschi via NANOG) wrote:
We had the same issue with Arelion this morning, a huge increase in the latency and jitter was happening across the US and of course for any traffic sourced from the EU flowing to the US. Within the US we noticed a latency spike from 20-50ms to ~200ms. From the EU to our US facilities, we noticed a spike from ~130ms to ~400ms.
We ended up shifting traffic from Arelion to a different carrier, and will stay there until we are confident that they fixed any issue that they had in their network. To be honest, we've been experiencing quite similar issues with unstable latency on Arelion over the past couple of months, and it's becoming quite frustrating
*Andrian Visnevschi*
VP of Network & Security
+373 68374133
andrian@acreto.io | https://acreto.io
On Tue, Oct 24, 2023 at 6:03 PM David Hubbard <dhubbard@dino.hostasaurus.com> wrote:
Hey all, anyone aware of issues with Arelion this morning? We have a bunch of end users on at least Cox and Cogeco who are having serious issues with service access, and the problem appears to be on the return path where it traverses Arelion. Source net is Lumen/L3 3356 but loss/latency doesn’t appear to creep up until already within Arelion’s network.
Imperva mentioned outage/degradation due to a national ISP issue too, so I suspect it’s affecting more than just certain peerings.
Thanks,
David
I'm experiencing this in the southeast US region. Unfortunately, return traffic from a data center that we have a VPN connection to. Is routed first over Level 3 who is then handing it off to Arelion in Miami where we then see the 200 to 300 millisecond latency and 50% plus packet loss. On Tue, Oct 24, 2023, 11:22 Elmar K. Bins <elmi@4ever.de> wrote:
We also observed this today, UTC morning, esp. across the pond, and our alternative paths from Europe to the US were suffering from the still unfixed fibre cut, so I was a bit unhappy with 200 extra ms, and 60% loss.
1299 seems to've found an alternative path in the meantime, looks good to us.
Elmar.
nanog@nanog.org (Andrian Visnevschi via NANOG) wrote:
We had the same issue with Arelion this morning, a huge increase in the latency and jitter was happening across the US and of course for any traffic sourced from the EU flowing to the US. Within the US we noticed a latency spike from 20-50ms to ~200ms. From the EU to our US facilities, we noticed a spike from ~130ms to ~400ms.
We ended up shifting traffic from Arelion to a different carrier, and will stay there until we are confident that they fixed any issue that they had in their network. To be honest, we've been experiencing quite similar issues with unstable latency on Arelion over the past couple of months, and it's becoming quite frustrating
*Andrian Visnevschi*
VP of Network & Security
+373 68374133
andrian@acreto.io | https://acreto.io
On Tue, Oct 24, 2023 at 6:03 PM David Hubbard < dhubbard@dino.hostasaurus.com> wrote:
Hey all, anyone aware of issues with Arelion this morning? We have a bunch of end users on at least Cox and Cogeco who are having serious issues with service access, and the problem appears to be on the return path where it traverses Arelion. Source net is Lumen/L3 3356 but loss/latency doesn’t appear to creep up until already within Arelion’s network.
Imperva mentioned outage/degradation due to a national ISP issue too, so I suspect it’s affecting more than just certain peerings.
Thanks,
David
$dayjob has transit from 1299 in Miami. We’ve been seeing packet loss nearby in their network since around 06:00 UTC, still ongoing. Steve
On Oct 24, 2023, at 8:20 AM, Elmar K. Bins <elmi@4ever.de> wrote:
We also observed this today, UTC morning, esp. across the pond, and our alternative paths from Europe to the US were suffering from the still unfixed fibre cut, so I was a bit unhappy with 200 extra ms, and 60% loss.
1299 seems to've found an alternative path in the meantime, looks good to us.
Elmar.
nanog@nanog.org (Andrian Visnevschi via NANOG) wrote:
We had the same issue with Arelion this morning, a huge increase in the latency and jitter was happening across the US and of course for any traffic sourced from the EU flowing to the US. Within the US we noticed a latency spike from 20-50ms to ~200ms. From the EU to our US facilities, we noticed a spike from ~130ms to ~400ms.
We ended up shifting traffic from Arelion to a different carrier, and will stay there until we are confident that they fixed any issue that they had in their network. To be honest, we've been experiencing quite similar issues with unstable latency on Arelion over the past couple of months, and it's becoming quite frustrating
*Andrian Visnevschi*
VP of Network & Security
+373 68374133
andrian@acreto.io | https://acreto.io
On Tue, Oct 24, 2023 at 6:03 PM David Hubbard <dhubbard@dino.hostasaurus.com> wrote:
Hey all, anyone aware of issues with Arelion this morning? We have a bunch of end users on at least Cox and Cogeco who are having serious issues with service access, and the problem appears to be on the return path where it traverses Arelion. Source net is Lumen/L3 3356 but loss/latency doesn’t appear to creep up until already within Arelion’s network.
Imperva mentioned outage/degradation due to a national ISP issue too, so I suspect it’s affecting more than just certain peerings.
Thanks,
David
About as ugly as a graph as ive seen! Reverse path for this site is cogent, mtr and forward ISP failing me a bit not returning the transition hop IP I need to verify, but im assuming its Arelion. This site is in the southeast also. [cid:image001.png@01DA0667.5563C150] From: NANOG <nanog-bounces+brent.neader=druryhotels.com@nanog.org> On Behalf Of Steve Feldman Sent: Tuesday, October 24, 2023 10:37 AM To: Elmar K. Bins <elmi@4ever.de> Cc: NANOG <nanog@nanog.org> Subject: [EXTERNAL] Re: Arelion/Telia AS1299 issues? $dayjob has transit from 1299 in Miami. We’ve been seeing packet loss nearby in their network since around 06: 00 UTC, still ongoing. Steve > On Oct 24, 2023, at 8: 20 AM, Elmar K. Bins <elmi@ 4ever. de> wrote: > > We also observed ZjQcmQRYFpfptBannerStart This Message Is From an Untrusted Sender You have not previously corresponded with this sender. Report Suspicious <https://us-phishalarm-ewt.proofpoint.com/EWT/v1/MQS9YPUatEJj!i4ZDkHftmwe3H0JvedmHHhW_2t4DY3hWkyfYuBLeQ3bHEB3ja6kIgXJzwWCVRZuBlVEkFmZSEzdnPRSuKhsqw8XY6Ke8Hg551lyf7iTALgB-nw$> ZjQcmQRYFpfptBannerEnd $dayjob has transit from 1299 in Miami. We’ve been seeing packet loss nearby in their network since around 06:00 UTC, still ongoing. Steve
On Oct 24, 2023, at 8:20 AM, Elmar K. Bins <elmi@4ever.de<mailto:elmi@4ever.de>> wrote:
We also observed this today, UTC morning, esp. across the pond, and our
alternative paths from Europe to the US were suffering from the still unfixed
fibre cut, so I was a bit unhappy with 200 extra ms, and 60% loss.
1299 seems to've found an alternative path in the meantime, looks good to us.
Elmar.
nanog@nanog.org<mailto:nanog@nanog.org> (Andrian Visnevschi via NANOG) wrote:
We had the same issue with Arelion this morning, a huge increase in the
latency and jitter was happening across the US and of course for any
traffic sourced from the EU flowing to the US.
Within the US we noticed a latency spike from 20-50ms to ~200ms. From the
EU to our US facilities, we noticed a spike from ~130ms to ~400ms.
We ended up shifting traffic from Arelion to a different carrier, and will
stay there until we are confident that they fixed any issue that they had
in their network. To be honest, we've been experiencing quite similar
issues with unstable latency on Arelion over the past couple of months, and
it's becoming quite frustrating
*Andrian Visnevschi*
VP of Network & Security
+373 68374133
andrian@acreto.io<mailto:andrian@acreto.io> | https://urldefense.com/v3/__https://acreto.io__;!!MQS9YPUatEJj!iZ7j3VWHoxkh3S3Y9nSVzutat2MUlem16ycD6ZkjZEKDIfzdtZuY1hZ8YlspD9zJ_9Go4lCTes737Rtqe1SEPX-_sA$<https://urldefense.com/v3/__https:/acreto.io__;!!MQS9YPUatEJj!iZ7j3VWHoxkh3S3Y9nSVzutat2MUlem16ycD6ZkjZEKDIfzdtZuY1hZ8YlspD9zJ_9Go4lCTes737Rtqe1SEPX-_sA$>
On Tue, Oct 24, 2023 at 6:03 PM David Hubbard <dhubbard@dino.hostasaurus.com<mailto:dhubbard@dino.hostasaurus.com>>
wrote:
Hey all, anyone aware of issues with Arelion this morning? We have a
bunch of end users on at least Cox and Cogeco who are having serious issues
with service access, and the problem appears to be on the return path where
it traverses Arelion. Source net is Lumen/L3 3356 but loss/latency doesn’t
appear to creep up until already within Arelion’s network.
Imperva mentioned outage/degradation due to a national ISP issue too, so I
suspect it’s affecting more than just certain peerings.
Thanks,
David
It’s definitely been an annoying day. Cogent’s “don’t advertise to telia” BGP community doesn’t work, so we can’t route around this either. Then again, my bad for using the “Wal-mart” of the internet. Eric
Cogent doesn’t have any peer specific action communities. Brendan On Wed, 25 Oct 2023 at 9:01 am, Eric C. Miller <eric@ericheather.com> wrote:
It’s definitely been an annoying day. Cogent’s “don’t advertise to telia” BGP community doesn’t work, so we can’t route around this either. Then again, my bad for using the “Wal-mart” of the internet.
Eric
I've got a small single-homed site in Central Europe with an Arelion transit circuit and it had some brief connectivity issues around 0830-0900 Central US time. Cleared up relatively quickly. Also have connectivity with Arelion in NYC at 60 Hudson, but have not experienced any issues related to that transit circuit which is part of a significantly larger multi-homed network. - mdh On Tue, Oct 24, 2023 at 10:37 AM Steve Feldman <feldman@twincreeks.net> wrote:
$dayjob has transit from 1299 in Miami. We’ve been seeing packet loss nearby in their network since around 06:00 UTC, still ongoing. Steve
Matt Harris VP OF INFRASTRUCTURE Follow us on LinkedIn! matt.harris@netfire.net 816-256-5446 www.netfire.com
On Oct 24, 2023, at 8:20 AM, Elmar K. Bins <elmi@4ever.de> wrote:
We also observed this today, UTC morning, esp. across the pond, and our alternative paths from Europe to the US were suffering from the still
fibre cut, so I was a bit unhappy with 200 extra ms, and 60% loss.
1299 seems to've found an alternative path in the meantime, looks good to us.
Elmar.
nanog@nanog.org (Andrian Visnevschi via NANOG) wrote:
We had the same issue with Arelion this morning, a huge increase in the latency and jitter was happening across the US and of course for any traffic sourced from the EU flowing to the US. Within the US we noticed a latency spike from 20-50ms to ~200ms. From
unfixed the
EU to our US facilities, we noticed a spike from ~130ms to ~400ms.
We ended up shifting traffic from Arelion to a different carrier, and will stay there until we are confident that they fixed any issue that they had in their network. To be honest, we've been experiencing quite similar issues with unstable latency on Arelion over the past couple of months, and it's becoming quite frustrating
*Andrian Visnevschi*
VP of Network & Security
+373 68374133
andrian@acreto.io | https://acreto.io
On Tue, Oct 24, 2023 at 6:03 PM David Hubbard < dhubbard@dino.hostasaurus.com> wrote:
Hey all, anyone aware of issues with Arelion this morning? We have a bunch of end users on at least Cox and Cogeco who are having serious issues with service access, and the problem appears to be on the return path where it traverses Arelion. Source net is Lumen/L3 3356 but loss/latency doesn’t appear to creep up until already within Arelion’s network.
Imperva mentioned outage/degradation due to a national ISP issue too, so I suspect it’s affecting more than just certain peerings.
Thanks,
David
participants (9)
-
Andrian Visnevschi -
Brandon Jackson -
Brendan Halley -
David Hubbard -
Elmar K. Bins -
Eric C. Miller -
Matt Harris -
Neader, Brent -
Steve Feldman