Dear All Just wondering if anyone else saw this yesterday afternoon ? Jun 20 16:57:29:E:BGP: From Peer 38.X.X.X received Long AS_PATH= AS_SEQ(2) 174 12956 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 ... attribute length (567) More than configured MAXAS-LIMIT Jun 20 16:15:26:E:BGP: From Peer 78.X.X.X received Long AS_PATH= AS_SEQ(2) 5580 3257 12956 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 ... attribute length (568) More than configured MAXAS-LIMIT Someone is having fun, creating weird and wonderful long AS paths based around AS 23456, we saw the same pattern of data from numerous upstream providers. Kindest Regards, James Braunegg [cid:image001.png@01D280A4.01865B60] 1300 769 972<tel:1300%20769%20972> / 0488 997 207<tel:1300%20769%20972> james@micron21.com<mailto:james@micron21.com> www.micron21.com/<http://www.micron21.com/> [cid:image002.png@01D280A4.01865B60]<http://www.micron21.com/> [cid:image003.png@01D280A4.01865B60]<https://www.facebook.com/micron21/> [cid:image004.png@01D280A4.01865B60]<https://twitter.com/micron21> Follow us on Twitter<https://twitter.com/micron21> for important service and system updates. This message is intended for the addressee named above. It may contain privileged or confidential information. If you are not the intended recipient of this message you must not use, copy, distribute or disclose it to anyone other than the addressee. If you have received this message in error please return the message to the sender by replying to it and then delete the message from your computer.
Yes, we had this kind of stuff in our logs: Jun 20 08:15:25 cr-co-01-pareq2-re0 rpd[9656]: %DAEMON-3: Prefix Send failed ! x:x:186.177.176.0/23 (label 19) bgp_rt_trace_too_big_message:1213 path attribute too big. Cannot build update. The AS path we have here is currently 12956 262206 262206 262197...
On 21 june 2017 at 01:12, James Braunegg <james.braunegg@micron21.com> wrote :
Just wondering if anyone else saw this yesterday afternoon ?
Jun 20 16:57:29:E:BGP: From Peer 38.X.X.X received Long AS_PATH= AS_SEQ(2) 174 12956 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 ... attribute length (567) More than configured MAXAS-LIMIT
Someone is having fun, creating weird and wonderful long AS paths based around AS 23456, we saw the same pattern of data from numerous upstream providers.
On 2017-06-20 23:12, James Braunegg wrote:
Dear All
Just wondering if anyone else saw this yesterday afternoon ?
Jun 20 16:57:29:E:BGP: From Peer 38.X.X.X received Long AS_PATH= AS_SEQ(2) 174 12956 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 ... attribute length (567) More than configured MAXAS-LIMIT
Jun 20 16:15:26:E:BGP: From Peer 78.X.X.X received Long AS_PATH= AS_SEQ(2) 5580 3257 12956 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 ... attribute length (568) More than configured MAXAS-LIMIT
Someone is having fun, creating weird and wonderful long AS paths based around AS 23456, we saw the same pattern of data from numerous upstream providers.
Kindest Regards,
James Braunegg
[cid:image001.png@01D280A4.01865B60]
1300 769 972<tel:1300%20769%20972> / 0488 997 207<tel:1300%20769%20972>
james@micron21.com<mailto:james@micron21.com>
www.micron21.com/<http://www.micron21.com/>
[cid:image002.png@01D280A4.01865B60]<http://www.micron21.com/>
[cid:image003.png@01D280A4.01865B60]<https://www.facebook.com/micron21/>
[cid:image004.png@01D280A4.01865B60]<https://twitter.com/micron21>
Follow us on Twitter<https://twitter.com/micron21> for important service and system updates.
This message is intended for the addressee named above. It may contain privileged or confidential information. If you are not the intended recipient of this message you must not use, copy, distribute or disclose it to anyone other than the addressee. If you have received this message in error please return the message to the sender by replying to it and then delete the message from your computer.
Could be this: http://blog.ipspace.net/2009/02/root-cause-analysis-oversized-as-paths.html
Just wondering if anyone else saw this yesterday afternoon ?
Jun 20 16:57:29:E:BGP: From Peer 38.X.X.X received Long AS_PATH=3D AS_SEQ(2= ) 174 12956 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 234= 56 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 = 23456 23456 23456 23456 23456 ... attribute length (567) More than configur= ed MAXAS-LIMIT
There are quite a few examples of people using stupidly long AS paths. For instance 177.23.232.0/24 *[BGP/170] 00:52:40, MED 0, localpref 105 AS path: 6939 16735 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262949 52938 52938 52938 52938 52938 52938 52938 52938 52938 52938 52938 I I currently have 27 prefixes in my Internet routing table with 40 or more ASes in the AS path (show route aspath-regex ".{40,}"). I see no valid reason for such long AS paths. Time to update filters here. I'm tempted to set the cutoff at 30 - can anybody see a good reason to permit longer AS paths? Steinar Haug, Nethelp consulting, sthaug@nethelp.no
177.23.232.0/24 *[BGP/170] 00:52:40, MED 0, localpref 105 ... ... I see no valid reason for such long AS paths.
[ assuming it is not the microtik thing ] the /24 can not be sliced to steer inbound, so they're desperately trying to push it away with prepend. of course, their upstreams all prefer customers, so they keep adding prepends in some vain hope. randy
On 06/21/2017 12:56 AM, sthaug@nethelp.no wrote:
I see no valid reason for such long AS paths. Time to update filters here. I'm tempted to set the cutoff at 30 - can anybody see a good reason to permit longer AS paths?
Well, as I mentioned in my Net Neutrality filing to the FCC, a TTL of 30 is OK for intra-planet routing, but when you start leaving the big blue marble you will need to allow the packets to live longer. Your network, your decisions <g,d&rlh>
I see no valid reason for such long AS paths. Time to update filters here. I'm tempted to set the cutoff at 30 - can anybody see a good reason to permit longer AS paths?
Well, as I mentioned in my Net Neutrality filing to the FCC, a TTL of 30 is OK for intra-planet routing, but when you start leaving the big blue marble you will need to allow the packets to live longer.
TTL != AS path length I can certainly see the use for a TTL of 30. I cannot see the use for an AS path length greater than 30 (especially not when 2 ASes are each repeated 16 times). Steinar Haug, Nethelp consulting, sthaug@nethelp.no
Why not ask the operator why they are pretending this path? Perhaps they have a good explanation that you haven't thought of. Blindly limiting otherwise legal path lengths is not a defensible practice, in my opinion. -mel beckman On Jun 21, 2017, at 1:36 PM, "sthaug@nethelp.no" <sthaug@nethelp.no> wrote:
I see no valid reason for such long AS paths. Time to update filters here. I'm tempted to set the cutoff at 30 - can anybody see a good reason to permit longer AS paths?
Well, as I mentioned in my Net Neutrality filing to the FCC, a TTL of 30 is OK for intra-planet routing, but when you start leaving the big blue marble you will need to allow the packets to live longer.
TTL != AS path length
I can certainly see the use for a TTL of 30. I cannot see the use for an AS path length greater than 30 (especially not when 2 ASes are each repeated 16 times).
Steinar Haug, Nethelp consulting, sthaug@nethelp.no
"Mel" == Mel Beckman <mel@beckman.org> writes:
Mel> Why not ask the operator why they are pretending this path? Perhaps Mel> they have a good explanation that you haven't thought of. Blindly Mel> limiting otherwise legal path lengths is not a defensible practice, in Mel> my opinion. Mel> -mel beckman A prepend like that is usually the result of someone using the IOS syntax on a XR or Junos router. Long ago, someone accidentally prepending 255 times hit a bug (or was it a too strict bgp implementation? I don't remember) resulting in several networks across the globe dropping neighbors. One has to protect against these things somehow. As a data point, here is how many prefixes I see on my network for each as-path length, after removing prepends: aspath length count ------------------------- 0: 340 1: 47522 2: 292879 3: 227822 4: 58390 5: 10217 6: 2123 7: 638 8: 48 9: 58 11: 20 12: 2 So, does your customer have a legitimate reason to prepend more than 5 times? Maybe. I still think that anyone that does should have their BGP driving licence revoked, though. Pf -- Pierfrancesco Caci, ik5pvx
I see 5+ prepends as maybe not reason to have your "BGP driving license revoked" but if I can continue with the concept that you have your BGP learners permit. If I think back to when I learned to code or when making ACL's, we still used line number and practice would be to give ourselves lots of space 5 or 10 numbers in case we have to insert something in the middle. ie I need 2 sets of prepends, I'm still learning this stuff so I'll go with 5 and 10. We all started somewhere, we all did dumb stuff, hopefully, we all learned. 12AS hops, I have to go see how they are connected now, maybe someone in that chain needs to be invited by an IX to a NANOG or GPF or some such, that can't be super efficient. -jim On Thu, Jun 22, 2017 at 3:09 AM, Pierfrancesco Caci <pf@tippete.net> wrote:
"Mel" == Mel Beckman <mel@beckman.org> writes:
Mel> Why not ask the operator why they are pretending this path? Perhaps Mel> they have a good explanation that you haven't thought of. Blindly Mel> limiting otherwise legal path lengths is not a defensible practice, in Mel> my opinion.
Mel> -mel beckman
A prepend like that is usually the result of someone using the IOS syntax on a XR or Junos router.
Long ago, someone accidentally prepending 255 times hit a bug (or was it a too strict bgp implementation? I don't remember) resulting in several networks across the globe dropping neighbors. One has to protect against these things somehow.
As a data point, here is how many prefixes I see on my network for each as-path length, after removing prepends:
aspath length count ------------------------- 0: 340 1: 47522 2: 292879 3: 227822 4: 58390 5: 10217 6: 2123 7: 638 8: 48 9: 58 11: 20 12: 2
So, does your customer have a legitimate reason to prepend more than 5 times? Maybe. I still think that anyone that does should have their BGP driving licence revoked, though.
Pf
-- Pierfrancesco Caci, ik5pvx
You don't have to wonder. You can call and ask them. -mel via cell
On Jun 22, 2017, at 5:47 AM, jim deleskie <deleskie@gmail.com> wrote:
I see 5+ prepends as maybe not reason to have your "BGP driving license revoked" but if I can continue with the concept that you have your BGP learners permit. If I think back to when I learned to code or when making ACL's, we still used line number and practice would be to give ourselves lots of space 5 or 10 numbers in case we have to insert something in the middle. ie I need 2 sets of prepends, I'm still learning this stuff so I'll go with 5 and 10. We all started somewhere, we all did dumb stuff, hopefully, we all learned.
12AS hops, I have to go see how they are connected now, maybe someone in that chain needs to be invited by an IX to a NANOG or GPF or some such, that can't be super efficient.
-jim
On Thu, Jun 22, 2017 at 3:09 AM, Pierfrancesco Caci <pf@tippete.net> wrote:
> "Mel" == Mel Beckman <mel@beckman.org> writes:
Mel> Why not ask the operator why they are pretending this path? Perhaps Mel> they have a good explanation that you haven't thought of. Blindly Mel> limiting otherwise legal path lengths is not a defensible practice, in Mel> my opinion.
Mel> -mel beckman
A prepend like that is usually the result of someone using the IOS syntax on a XR or Junos router.
Long ago, someone accidentally prepending 255 times hit a bug (or was it a too strict bgp implementation? I don't remember) resulting in several networks across the globe dropping neighbors. One has to protect against these things somehow.
As a data point, here is how many prefixes I see on my network for each as-path length, after removing prepends:
aspath length count ------------------------- 0: 340 1: 47522 2: 292879 3: 227822 4: 58390 5: 10217 6: 2123 7: 638 8: 48 9: 58 11: 20 12: 2
So, does your customer have a legitimate reason to prepend more than 5 times? Maybe. I still think that anyone that does should have their BGP driving licence revoked, though.
Pf
-- Pierfrancesco Caci, ik5pvx
Steinar, What reason is there to filter them? They are not a significant fraction of BGP paths. They cause no harm. It's just your sense of tidiness. You might consider contacting one of the operators to see if they do have a good reason you haven't considered. But absent a good reason *to* filter them, I would let BGP mechanics work as intended. -mel beckman On Jun 21, 2017, at 12:57 AM, "sthaug@nethelp.no" <sthaug@nethelp.no> wrote:
Just wondering if anyone else saw this yesterday afternoon ?
Jun 20 16:57:29:E:BGP: From Peer 38.X.X.X received Long AS_PATH=3D AS_SEQ(2= ) 174 12956 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 234= 56 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 = 23456 23456 23456 23456 23456 ... attribute length (567) More than configur= ed MAXAS-LIMIT
There are quite a few examples of people using stupidly long AS paths. For instance
177.23.232.0/24 *[BGP/170] 00:52:40, MED 0, localpref 105 AS path: 6939 16735 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262949 52938 52938 52938 52938 52938 52938 52938 52938 52938 52938 52938 I
I currently have 27 prefixes in my Internet routing table with 40 or more ASes in the AS path (show route aspath-regex ".{40,}").
I see no valid reason for such long AS paths. Time to update filters here. I'm tempted to set the cutoff at 30 - can anybody see a good reason to permit longer AS paths?
Steinar Haug, Nethelp consulting, sthaug@nethelp.no
My cut off is 6 ASNs - more than 6 and it never makes it to the FIB. However, for this to be viable with plenty of unique prefixes to maintain a large table, we have lots and lots of direct big and small peers and much more than the usual amount of transit neighbors in our network. Silicon Valley companies are very demanding for the fasted path with the lowest number of router hops. ASN hops almost always lead to more router hops in the trace. We have customers that call us if everything is fine and they want to shave off milliseconds to favorite destinations. Picky, picky, picky. I am wondering how may other networks get requests (more like demands) from customers wanting you to speed packets up to and from a specific office in India or China. Customers knowing nothing about their office ISP overseas. BTW, it's almost always they have the cheapest congested shared office connection in the building overseas (especially in India). So they can't do anything there except "pretend" about the bandwidth available. About all they know is the IP address of the VPN and they were told they have a full gig connection. Sure they have a gig port, but it's on a switch together with 10 building neighbors that all also have a gig port on a circuit to the building that no one can maintain a gig for more than 3 ms. Go ahead try and fix that latency packet dropping issue with a firewall on both ends with SPI turned on in both directions. It's your fault if you cant make it better. After all their VPN from London to Bangalore works fine. And the ones in China all work fine to and from Australia. Anyways, I always wondered is it just me or do others get these kind of requests? Thank You Bob Evans CTO
Steinar,
What reason is there to filter them? They are not a significant fraction of BGP paths. They cause no harm. It's just your sense of tidiness.
You might consider contacting one of the operators to see if they do have a good reason you haven't considered. But absent a good reason *to* filter them, I would let BGP mechanics work as intended.
-mel beckman
On Jun 21, 2017, at 12:57 AM, "sthaug@nethelp.no" <sthaug@nethelp.no> wrote:
Just wondering if anyone else saw this yesterday afternoon ?
Jun 20 16:57:29:E:BGP: From Peer 38.X.X.X received Long AS_PATH=3D AS_SEQ(2= ) 174 12956 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 234= 56 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 = 23456 23456 23456 23456 23456 ... attribute length (567) More than configur= ed MAXAS-LIMIT
There are quite a few examples of people using stupidly long AS paths. For instance
177.23.232.0/24 *[BGP/170] 00:52:40, MED 0, localpref 105 AS path: 6939 16735 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262949 52938 52938 52938 52938 52938 52938 52938 52938 52938 52938 52938 I
I currently have 27 prefixes in my Internet routing table with 40 or more ASes in the AS path (show route aspath-regex ".{40,}").
I see no valid reason for such long AS paths. Time to update filters here. I'm tempted to set the cutoff at 30 - can anybody see a good reason to permit longer AS paths?
Steinar Haug, Nethelp consulting, sthaug@nethelp.no
Hey, Uou're saying, you drop long AS_PATH, to improve customer observed latency? Implication being, because you dropped the long AS_PATH prefixes, you're now selecting shorter AS_PATH prefixes to the FIB? Absent of this policy, in which scenario would you have inserted the filtered longer AS_PATH prefix to the FIB? I assume only scenario where this happens is where there is larger aggregate route, which is lower latency than the more specific route with longer as_path. Do you have data how many of such paths exist and what is the latency delta? I'm extremely skeptical that long AS_PATH rejection has any measurable impact on aggregate path latencies. On 21 June 2017 at 19:42, Bob Evans <bob@fiberinternetcenter.com> wrote:
My cut off is 6 ASNs - more than 6 and it never makes it to the FIB.
However, for this to be viable with plenty of unique prefixes to maintain a large table, we have lots and lots of direct big and small peers and much more than the usual amount of transit neighbors in our network. Silicon Valley companies are very demanding for the fasted path with the lowest number of router hops. ASN hops almost always lead to more router hops in the trace. We have customers that call us if everything is fine and they want to shave off milliseconds to favorite destinations. Picky, picky, picky.
I am wondering how may other networks get requests (more like demands) from customers wanting you to speed packets up to and from a specific office in India or China. Customers knowing nothing about their office ISP overseas. BTW, it's almost always they have the cheapest congested shared office connection in the building overseas (especially in India). So they can't do anything there except "pretend" about the bandwidth available. About all they know is the IP address of the VPN and they were told they have a full gig connection. Sure they have a gig port, but it's on a switch together with 10 building neighbors that all also have a gig port on a circuit to the building that no one can maintain a gig for more than 3 ms. Go ahead try and fix that latency packet dropping issue with a firewall on both ends with SPI turned on in both directions. It's your fault if you cant make it better. After all their VPN from London to Bangalore works fine. And the ones in China all work fine to and from Australia.
Anyways, I always wondered is it just me or do others get these kind of requests?
Thank You Bob Evans CTO
Steinar,
What reason is there to filter them? They are not a significant fraction of BGP paths. They cause no harm. It's just your sense of tidiness.
You might consider contacting one of the operators to see if they do have a good reason you haven't considered. But absent a good reason *to* filter them, I would let BGP mechanics work as intended.
-mel beckman
On Jun 21, 2017, at 12:57 AM, "sthaug@nethelp.no" <sthaug@nethelp.no> wrote:
Just wondering if anyone else saw this yesterday afternoon ?
Jun 20 16:57:29:E:BGP: From Peer 38.X.X.X received Long AS_PATH=3D AS_SEQ(2= ) 174 12956 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 234= 56 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 = 23456 23456 23456 23456 23456 ... attribute length (567) More than configur= ed MAXAS-LIMIT
There are quite a few examples of people using stupidly long AS paths. For instance
177.23.232.0/24 *[BGP/170] 00:52:40, MED 0, localpref 105 AS path: 6939 16735 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262949 52938 52938 52938 52938 52938 52938 52938 52938 52938 52938 52938 I
I currently have 27 prefixes in my Internet routing table with 40 or more ASes in the AS path (show route aspath-regex ".{40,}").
I see no valid reason for such long AS paths. Time to update filters here. I'm tempted to set the cutoff at 30 - can anybody see a good reason to permit longer AS paths?
Steinar Haug, Nethelp consulting, sthaug@nethelp.no
-- ++ytti
On Wed, 21 Jun 2017, Saku Ytti wrote:
Hey,
Uou're saying, you drop long AS_PATH, to improve customer observed latency? Implication being, because you dropped the long AS_PATH prefixes, you're now selecting shorter AS_PATH prefixes to the FIB?
Absent of this policy, in which scenario would you have inserted the filtered longer AS_PATH prefix to the FIB? I assume only scenario where this happens is where there is larger aggregate route, which is lower latency than the more specific route with longer as_path.
You do have to wonder, what was the thought process that resulted in 35 being the right number of prepends "accomplish" whatever TE they were shooting for? AS path: 10026 9498 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 45271 I don't mean to single out 55644. It's just the first/most obnoxiously long as-path I found when looking for long ones. I seriously doubt provider/customer/customer-of-customer relationships ever get much deeper than a handful or so of ASNs...so if prepending a few times doesn't get it done, 10-20-30 prepends are unlikely to help. In the above case, that long path is actually our best path. We localpref peering above transit. So, it doesn't matter how many prepends, they add, we're never going to not use this path if its available. We have transit paths to these routes that are only a single handful of ASNs long. ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route | therefore you are _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
On 06/22/2017 04:27 AM, Jon Lewis wrote:
You do have to wonder, what was the thought process that resulted in 35 being the right number of prepends "accomplish" whatever TE they were shooting for?
AS path: 10026 9498 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 45271
I don't mean to single out 55644. It's just the first/most obnoxiously long as-path I found when looking for long ones.
I seriously doubt provider/customer/customer-of-customer relationships ever get much deeper than a handful or so of ASNs...so if prepending a few times doesn't get it done, 10-20-30 prepends are unlikely to help.
In the above case, that long path is actually our best path. We localpref peering above transit. So, it doesn't matter how many prepends, they add, we're never going to not use this path if its available. We have transit paths to these routes that are only a single handful of ASNs long.
I think I understand the problem, and now I understand why prepends didn't do much for me. Over the years, I tended two multi-homed sites. In both cases, the two uplinks had different speeds. When I used prepend to try to get the outside world to prefer the faster link, some traffic was stubborn about coming in the slow one. Difference in speeds? In the first network it was 45 mbps and 10 mbps. In the second network it was 16 mbps and 1.5 mbps. Both network owners were too stingy at the time to opt for harmonized rates. Question: how could communities be used to "force" preference for one uplink over another by the peers? I'm long past needing this, but others might benefit. (And when you stop learning, you're dead.)
I didn't see anyone answer (sorry if I missed it and this is redundant) ... In the path selection algorithm, local preference is processed before AS-PATH. Within your provider's AS, your prefixes could be a default localpref of 100, and learned prefixes from their peers 85, for example. In this case, Intra-AS will always be preferred due to higher lpref. You may prepend a handful of times out of your connection that is within the provider's AS, thus making the overall AS-PATH longer, but localpref still remains 100 vs. peer 85, so intra-AS still preferred. Some providers allow you to send community attributes with your prefixes to modify the localpref within the provider AS and make it lower than their peer localpref. Solves this particular conundrum. Couple examples: https://onestep.net/communities/as3356/ https://onestep.net/communities/as1299/ Depending on your allocation size and your needs, if you wanted to force all traffic over the "fast" link and use the "slow" link as an emergency backup, it could be easier to just announce more specific routes out the faster connection and send an aggregate out the backup one. No communities needed in that scenario. All depends on what you need to do, of course. HTH, Ryan On Thu, Jun 22, 2017 at 9:29 AM, Stephen Satchell <list@satchell.net> wrote:
On 06/22/2017 04:27 AM, Jon Lewis wrote:
You do have to wonder, what was the thought process that resulted in 35 being the right number of prepends "accomplish" whatever TE they were shooting for?
AS path: 10026 9498 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 55644 45271
I don't mean to single out 55644. It's just the first/most obnoxiously long as-path I found when looking for long ones.
I seriously doubt provider/customer/customer-of-customer relationships ever get much deeper than a handful or so of ASNs...so if prepending a few times doesn't get it done, 10-20-30 prepends are unlikely to help.
In the above case, that long path is actually our best path. We localpref peering above transit. So, it doesn't matter how many prepends, they add, we're never going to not use this path if its available. We have transit paths to these routes that are only a single handful of ASNs long.
I think I understand the problem, and now I understand why prepends didn't do much for me. Over the years, I tended two multi-homed sites. In both cases, the two uplinks had different speeds. When I used prepend to try to get the outside world to prefer the faster link, some traffic was stubborn about coming in the slow one.
Difference in speeds? In the first network it was 45 mbps and 10 mbps. In the second network it was 16 mbps and 1.5 mbps. Both network owners were too stingy at the time to opt for harmonized rates.
Question: how could communities be used to "force" preference for one uplink over another by the peers? I'm long past needing this, but others might benefit. (And when you stop learning, you're dead.)
Usually when someone starts griping about RTT between destinations more than about 6 time zones apart, I start to talk to them about refraction indicies, platform specific switching delay differences, stuff like that. Normally I can chase them away or put them to sleep well before getting to 'I can break a lot of things, but physics is not one of them.' A longer ASN path is not an automatic signifier of a longer latency path. It can just as easily be a lower latency path that happens to traverse a expensive link that AS doesn't like to use normally, but wants a backup. Or maybe they have congestion inside their AS from that way in and want to influence traffic away from it. Or maybe they just read that chapter and thought it was a cool setting without knowing what it did. On Wed, Jun 21, 2017 at 12:42 PM, Bob Evans <bob@fiberinternetcenter.com> wrote:
My cut off is 6 ASNs - more than 6 and it never makes it to the FIB.
However, for this to be viable with plenty of unique prefixes to maintain a large table, we have lots and lots of direct big and small peers and much more than the usual amount of transit neighbors in our network. Silicon Valley companies are very demanding for the fasted path with the lowest number of router hops. ASN hops almost always lead to more router hops in the trace. We have customers that call us if everything is fine and they want to shave off milliseconds to favorite destinations. Picky, picky, picky.
I am wondering how may other networks get requests (more like demands) from customers wanting you to speed packets up to and from a specific office in India or China. Customers knowing nothing about their office ISP overseas. BTW, it's almost always they have the cheapest congested shared office connection in the building overseas (especially in India). So they can't do anything there except "pretend" about the bandwidth available. About all they know is the IP address of the VPN and they were told they have a full gig connection. Sure they have a gig port, but it's on a switch together with 10 building neighbors that all also have a gig port on a circuit to the building that no one can maintain a gig for more than 3 ms. Go ahead try and fix that latency packet dropping issue with a firewall on both ends with SPI turned on in both directions. It's your fault if you cant make it better. After all their VPN from London to Bangalore works fine. And the ones in China all work fine to and from Australia.
Anyways, I always wondered is it just me or do others get these kind of requests?
Thank You Bob Evans CTO
Steinar,
What reason is there to filter them? They are not a significant fraction of BGP paths. They cause no harm. It's just your sense of tidiness.
You might consider contacting one of the operators to see if they do have a good reason you haven't considered. But absent a good reason *to* filter them, I would let BGP mechanics work as intended.
-mel beckman
On Jun 21, 2017, at 12:57 AM, "sthaug@nethelp.no" <sthaug@nethelp.no> wrote:
Just wondering if anyone else saw this yesterday afternoon ?
Jun 20 16:57:29:E:BGP: From Peer 38.X.X.X received Long AS_PATH=3D AS_SEQ(2= ) 174 12956 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 234= 56 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 = 23456 23456 23456 23456 23456 ... attribute length (567) More than configur= ed MAXAS-LIMIT
There are quite a few examples of people using stupidly long AS paths. For instance
177.23.232.0/24 *[BGP/170] 00:52:40, MED 0, localpref 105 AS path: 6939 16735 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262949 52938 52938 52938 52938 52938 52938 52938 52938 52938 52938 52938 I
I currently have 27 prefixes in my Internet routing table with 40 or more ASes in the AS path (show route aspath-regex ".{40,}").
I see no valid reason for such long AS paths. Time to update filters here. I'm tempted to set the cutoff at 30 - can anybody see a good reason to permit longer AS paths?
Steinar Haug, Nethelp consulting, sthaug@nethelp.no
Mel, There was a Cisco bug many years ago that caused lots of issues. Since then we have limited max-as to 50 and it has not caused any reported issues yet. Link that does not require a CCO login to view. http://blog.ipspace.net/2009/02/oversized-as-paths-cisco-ios-bug.html Regards Steve Lalonde
On 21 Jun 2017, at 16:49, Mel Beckman <mel@beckman.org> wrote:
Steinar,
What reason is there to filter them? They are not a significant fraction of BGP paths. They cause no harm. It's just your sense of tidiness.
You might consider contacting one of the operators to see if they do have a good reason you haven't considered. But absent a good reason *to* filter them, I would let BGP mechanics work as intended.
-mel beckman
On Jun 21, 2017, at 12:57 AM, "sthaug@nethelp.no" <sthaug@nethelp.no> wrote:
Just wondering if anyone else saw this yesterday afternoon ?
Jun 20 16:57:29:E:BGP: From Peer 38.X.X.X received Long AS_PATH=3D AS_SEQ(2= ) 174 12956 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 234= 56 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 23456 = 23456 23456 23456 23456 23456 ... attribute length (567) More than configur= ed MAXAS-LIMIT
There are quite a few examples of people using stupidly long AS paths. For instance
177.23.232.0/24 *[BGP/170] 00:52:40, MED 0, localpref 105 AS path: 6939 16735 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 28163 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262401 262949 52938 52938 52938 52938 52938 52938 52938 52938 52938 52938 52938 I
I currently have 27 prefixes in my Internet routing table with 40 or more ASes in the AS path (show route aspath-regex ".{40,}").
I see no valid reason for such long AS paths. Time to update filters here. I'm tempted to set the cutoff at 30 - can anybody see a good reason to permit longer AS paths?
Steinar Haug, Nethelp consulting, sthaug@nethelp.no
On 21 Jun 2017 17:51, "Mel Beckman" <mel@beckman.org> wrote: Steinar, What reason is there to filter them? The main reason I know of is this: On 22 Jun 2017 17:17, "Steve Lalonde" <steve@enta.net> wrote: Mel, There was a Cisco bug many years ago that caused lots of issues. Since then we have limited max-as to 50 and it has not caused any reported issues yet. Link that does not require a CCO login to view. http://blog.ipspace.net/2009/02/oversized-as-paths-cisco-ios-bug.html Like many providers we do still have legacy kit tucked away with shameful firmware versions running and also multiple vendors in play so I can't be sure we don't have devices still susceptible to such a bug in view of the DFZ. On 21 Jun 2017 18:45, "Bob Evans" <bob@fiberinternetcenter.com> wrote: My cut off is 6 ASNs - more than 6 and it never makes it to the FIB. So for the above reason we do have an AS_PATH limit but it is quite high like 63 or 120 ASNs (on mobile can't remember and right now). I don't want to get near a ^2 boundary like 255 or 128 but also don't want to drop prefixes if possible like a last resort fail-safe, so it's a very high number and will remove it at some point. On 22 Jun 2017 14:49, "jim deleskie" <deleskie@gmail.com> wrote: I see 5+ prepends as maybe not reason to have your "BGP driving license revoked" but if I can continue with the concept that you have your BGP learners permit. That (6) seems pretty low to me. Apart from a potential bug the other reason we thought of to block routes with a massive AS_PATH is that it could be a sign that the operator of a network doesn't know what their doing or doesn't have good processes in place (YMMV). If you have a path twice in BGP, once with a "giant" path length and once with a "normal" length the provider offering the "longer" path may have any manner of problems internally if they can't even manage their BGP routing policies appropriately. I have seen genuine reasons for up to about 6 pre-prends and beyond so you're probably dropping a decent amount of routes. At the time I set our filter I think we were dropping one single route. No one has complained so its still in place. Giant AS_PATHs can be debated in a similar fashion to AS numbers used/restricted by RFCs. We have AS number filters in place to block prefixes with a private ASN in the path, any transit provider should block such routes, again if they aren't it's a sign to me they're not doing a really great job. But it's very hard to know if you can drop those routes without affecting your customer base or that a suitable alternative exists. Great care must be taken when doing this. Otherwise the following issue arises: On 21 Jun 2017 19:13, "Saku Ytti" <saku@ytti.fi> wrote: Hey, Uou're saying, you drop long AS_PATH, to improve customer observed latency? Implication being, because you dropped the long AS_PATH prefixes, you're now selecting shorter AS_PATH prefixes to the FIB? Absent of this policy, in which scenario would you have inserted the filtered longer AS_PATH prefix to the FIB? I assume only scenario where this happens is where there is larger aggregate route, which is lower latency than the more specific route with longer as_path. So we drop "giant" AS_PATH length routes and routes with certain ASNs in the path, but we are fairly certain we don't need them / our customers don't need them etc. Not because we believe we are getting better (lower latency routes) routes from somewhere else as Saku said above, we can't feasibly "test" and compare the performance of every route we receive or need/want, but to protect our infrastructure. On 22 Jun 2017 16:54, "Jakob Heitz (jheitz)" <jheitz@cisco.com> wrote: 23456 is AS_TRANS. Either your router does not support 4 byte AS or there is a bug at AS 12956 or AS 12956 is intentionally prepending 23456. This ties in with my point about specific ASN filtering. Its not a problem seeing 23456 in your table but again perhaps an indicator of a problem or someone using legacy kit. No problem with 23456 routes like AS_PATH length of up to say 50 but beyond that, I can't thing of a genuine reasons and could be a sign that along the path there may be stability issues for example. Again, too difficult to measure. Depending on your customer base and requirements it can be safe to drop giant paths but care is needed, and if you do it, I think a number like 6 is way too low. Cheers, James.
James, The question is whether you would actually hear of any problems. Chances are that the problem would be experienced by somebody else, who has no idea that your filtering was causing it. -mel beckman
On Jun 23, 2017, at 4:33 AM, James Bensley <jwbensley@gmail.com> wrote:
On 21 Jun 2017 17:51, "Mel Beckman" <mel@beckman.org> wrote:
Steinar,
What reason is there to filter them?
The main reason I know of is this:
On 22 Jun 2017 17:17, "Steve Lalonde" <steve@enta.net> wrote:
Mel,
There was a Cisco bug many years ago that caused lots of issues. Since then we have limited max-as to 50 and it has not caused any reported issues yet.
Link that does not require a CCO login to view. http://blog.ipspace.net/2009/02/oversized-as-paths-cisco-ios-bug.html
Like many providers we do still have legacy kit tucked away with shameful firmware versions running and also multiple vendors in play so I can't be sure we don't have devices still susceptible to such a bug in view of the DFZ.
On 21 Jun 2017 18:45, "Bob Evans" <bob@fiberinternetcenter.com> wrote:
My cut off is 6 ASNs - more than 6 and it never makes it to the FIB.
So for the above reason we do have an AS_PATH limit but it is quite high like 63 or 120 ASNs (on mobile can't remember and right now). I don't want to get near a ^2 boundary like 255 or 128 but also don't want to drop prefixes if possible like a last resort fail-safe, so it's a very high number and will remove it at some point.
On 22 Jun 2017 14:49, "jim deleskie" <deleskie@gmail.com> wrote:
I see 5+ prepends as maybe not reason to have your "BGP driving license revoked" but if I can continue with the concept that you have your BGP learners permit.
That (6) seems pretty low to me. Apart from a potential bug the other reason we thought of to block routes with a massive AS_PATH is that it could be a sign that the operator of a network doesn't know what their doing or doesn't have good processes in place (YMMV). If you have a path twice in BGP, once with a "giant" path length and once with a "normal" length the provider offering the "longer" path may have any manner of problems internally if they can't even manage their BGP routing policies appropriately.
I have seen genuine reasons for up to about 6 pre-prends and beyond so you're probably dropping a decent amount of routes. At the time I set our filter I think we were dropping one single route. No one has complained so its still in place.
Giant AS_PATHs can be debated in a similar fashion to AS numbers used/restricted by RFCs. We have AS number filters in place to block prefixes with a private ASN in the path, any transit provider should block such routes, again if they aren't it's a sign to me they're not doing a really great job. But it's very hard to know if you can drop those routes without affecting your customer base or that a suitable alternative exists. Great care must be taken when doing this.
Otherwise the following issue arises:
On 21 Jun 2017 19:13, "Saku Ytti" <saku@ytti.fi> wrote:
Hey,
Uou're saying, you drop long AS_PATH, to improve customer observed latency? Implication being, because you dropped the long AS_PATH prefixes, you're now selecting shorter AS_PATH prefixes to the FIB?
Absent of this policy, in which scenario would you have inserted the filtered longer AS_PATH prefix to the FIB? I assume only scenario where this happens is where there is larger aggregate route, which is lower latency than the more specific route with longer as_path.
So we drop "giant" AS_PATH length routes and routes with certain ASNs in the path, but we are fairly certain we don't need them / our customers don't need them etc. Not because we believe we are getting better (lower latency routes) routes from somewhere else as Saku said above, we can't feasibly "test" and compare the performance of every route we receive or need/want, but to protect our infrastructure.
On 22 Jun 2017 16:54, "Jakob Heitz (jheitz)" <jheitz@cisco.com> wrote:
23456 is AS_TRANS. Either your router does not support 4 byte AS or there is a bug at AS 12956 or AS 12956 is intentionally prepending 23456.
This ties in with my point about specific ASN filtering. Its not a problem seeing 23456 in your table but again perhaps an indicator of a problem or someone using legacy kit. No problem with 23456 routes like AS_PATH length of up to say 50 but beyond that, I can't thing of a genuine reasons and could be a sign that along the path there may be stability issues for example. Again, too difficult to measure.
Depending on your customer base and requirements it can be safe to drop giant paths but care is needed, and if you do it, I think a number like 6 is way too low.
Cheers, James.
On 23 Jun 2017 17:03, "Mel Beckman" <mel@beckman.org> wrote: James, The question is whether you would actually hear of any problems. Chances are that the problem would be experienced by somebody else, who has no idea that your filtering was causing it. -mel beckman Hi Mel, For us this the answer is almost definitely a yes. We are an MSP (managed service provider) as opportunities to a traditional ISP, so our customers know they can open a ticket with us for pretty much anything and we'll try and look into it. We have had some weird issues with far away sites, first line can't find any issue, it works it's way up to somebody who knows how to check if we would be filtering a route on our transit and peering sessions. Earlier when I said that care is required when filtering long AS_PATH routes or certain AS numbers, we looked at the BGP table to see exactly which routes we'd drop before hand and communicated out these changes. I think for an MSP this shouldn't be hard to implement and manage, I can appreciate for a "flat" ISP ("he's some transit, help yourself") it could be more challenging. In relation to the OPs question, long AS_PATH routes can be filtered I just wouldn't bother except for very long paths to drop as little as possible and be sure of whY you drop/filter. Cheers, James.
James, By "experienced by someone else" I mean someone who is not one of your customers. The better strategy, I think, is to not filter long paths unless you have a reason to see their creating a problem. Otherwise you're just operating on superstition, no? -mel via cell
On Jun 24, 2017, at 12:42 AM, James Bensley <jwbensley@gmail.com> wrote:
On 23 Jun 2017 17:03, "Mel Beckman" <mel@beckman.org> wrote:
James,
The question is whether you would actually hear of any problems. Chances are that the problem would be experienced by somebody else, who has no idea that your filtering was causing it.
-mel beckman
Hi Mel,
For us this the answer is almost definitely a yes. We are an MSP (managed service provider) as opportunities to a traditional ISP, so our customers know they can open a ticket with us for pretty much anything and we'll try and look into it.
We have had some weird issues with far away sites, first line can't find any issue, it works it's way up to somebody who knows how to check if we would be filtering a route on our transit and peering sessions.
Earlier when I said that care is required when filtering long AS_PATH routes or certain AS numbers, we looked at the BGP table to see exactly which routes we'd drop before hand and communicated out these changes. I think for an MSP this shouldn't be hard to implement and manage, I can appreciate for a "flat" ISP ("he's some transit, help yourself") it could be more challenging.
In relation to the OPs question, long AS_PATH routes can be filtered I just wouldn't bother except for very long paths to drop as little as possible and be sure of whY you drop/filter.
Cheers, James.
On 24 June 2017 at 13:10, Mel Beckman <mel@beckman.org> wrote:
James,
By "experienced by someone else" I mean someone who is not one of your customers.
The better strategy, I think, is to not filter long paths unless you have a reason to see their creating a problem. Otherwise you're just operating on superstition, no?
-mel via cell
Hi Mel, I mean this as a rhetorical question as we could talk until the end of time about this; what is the difference between operating on superstition and trying to be pro-active? Both for me fall under the category of "risk management". Cheers, James.
Superstition has no basis in reality (i.e. black cat walks past DC door) Pro-Active is based on experience and knowledge (i.e. when disk space is 90% full for a regularly growing volume, we need to clean or add more before it hits 100%) I mean this as a rhetorical question as we could talk until the end of time about this; what is the difference between operating on superstition and trying to be pro-active? Both for me fall under the category of "risk management". Cheers, James.
This could just be ignorance, but based on this thread, I'm not sure what risk we would be managing, as DFZ router operators, by filtering those paths. They seem silly, but harmless (similar to, for instance, painting a nyan cat on a graph by announcing prefixes at certain times). On Sun, Jun 25, 2017 at 6:32 AM James Bensley <jwbensley@gmail.com> wrote:
On 24 June 2017 at 13:10, Mel Beckman <mel@beckman.org> wrote:
James,
By "experienced by someone else" I mean someone who is not one of your customers.
The better strategy, I think, is to not filter long paths unless you have a reason to see their creating a problem. Otherwise you're just operating on superstition, no?
-mel via cell
Hi Mel,
I mean this as a rhetorical question as we could talk until the end of time about this; what is the difference between operating on superstition and trying to be pro-active? Both for me fall under the category of "risk management".
Cheers, James.
-- -- Hunter Fuller Network Engineer VBH Annex B-5 +1 256 824 5331 Office of Information Technology The University of Alabama in Huntsville Systems and Infrastructure
Couldn't one make the same argument with respect to filtering private ASNs from the global table? Unlike filtering of RFC1918 and the like a private ASN in the path isn't likely to leak RFC1918 like traffic, yet I believe several major ISPs have done just that. This topic was discussed ~1 year ago on NANOG. I do filter private ASNs but have not yet filtered long AS paths. Before I did it I had to contact a major CDN because I would have dropped their route, in the end costing me money (choosing transit vs peering). In the end, it is indeed risk vs reward, with risk being undefined behavior. It's plausible to speculate that not every path length bug has been squashed (or might not be re-introduced). -Michael
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Hunter Fuller Sent: Monday, June 26, 2017 9:40 AM To: James Bensley <jwbensley@gmail.com>; nanog@nanog.org Subject: Re: Long AS Path
This could just be ignorance, but based on this thread, I'm not sure what risk we would be managing, as DFZ router operators, by filtering those paths. They seem silly, but harmless (similar to, for instance, painting a nyan cat on a graph by announcing prefixes at certain times).
On Sun, Jun 25, 2017 at 6:32 AM James Bensley <jwbensley@gmail.com> wrote:
On 24 June 2017 at 13:10, Mel Beckman <mel@beckman.org> wrote:
James,
By "experienced by someone else" I mean someone who is not one of your customers.
The better strategy, I think, is to not filter long paths unless you have a reason to see their creating a problem. Otherwise you're just operating on superstition, no?
-mel via cell
Hi Mel,
I mean this as a rhetorical question as we could talk until the end of time about this; what is the difference between operating on superstition and trying to be pro-active? Both for me fall under the category of "risk management".
Cheers, James.
--
-- Hunter Fuller Network Engineer VBH Annex B-5 +1 256 824 5331
Office of Information Technology The University of Alabama in Huntsville Systems and Infrastructure
Michael, Filtering private ASNs is actually part of the standard. It's intrinsic in the term "private ASN". A private ASN in the public routing table is a clear error, so filtering them is reasonable. Long AS paths are not a clear error.' I'm surprised nobody here who complains about long paths is has followed my suggestion: call the ASN operator and ask them why they do it, and report the results here. Until somebody does that, I don't see long path filtering as morally defensible :) -mel beckman
On Jun 26, 2017, at 8:09 AM, Michael Hare <michael.hare@wisc.edu> wrote:
Couldn't one make the same argument with respect to filtering private ASNs from the global table? Unlike filtering of RFC1918 and the like a private ASN in the path isn't likely to leak RFC1918 like traffic, yet I believe several major ISPs have done just that. This topic was discussed ~1 year ago on NANOG.
I do filter private ASNs but have not yet filtered long AS paths. Before I did it I had to contact a major CDN because I would have dropped their route, in the end costing me money (choosing transit vs peering).
In the end, it is indeed risk vs reward, with risk being undefined behavior. It's plausible to speculate that not every path length bug has been squashed (or might not be re-introduced).
-Michael
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Hunter Fuller Sent: Monday, June 26, 2017 9:40 AM To: James Bensley <jwbensley@gmail.com>; nanog@nanog.org Subject: Re: Long AS Path
This could just be ignorance, but based on this thread, I'm not sure what risk we would be managing, as DFZ router operators, by filtering those paths. They seem silly, but harmless (similar to, for instance, painting a nyan cat on a graph by announcing prefixes at certain times).
On Sun, Jun 25, 2017 at 6:32 AM James Bensley <jwbensley@gmail.com> wrote:
On 24 June 2017 at 13:10, Mel Beckman <mel@beckman.org> wrote: James,
By "experienced by someone else" I mean someone who is not one of your customers.
The better strategy, I think, is to not filter long paths unless you have a reason to see their creating a problem. Otherwise you're just operating on superstition, no?
-mel via cell
Hi Mel,
I mean this as a rhetorical question as we could talk until the end of time about this; what is the difference between operating on superstition and trying to be pro-active? Both for me fall under the category of "risk management".
Cheers, James. --
-- Hunter Fuller Network Engineer VBH Annex B-5 +1 256 824 5331
Office of Information Technology The University of Alabama in Huntsville Systems and Infrastructure
participants (19)
-
Bob Evans -
Hunter Fuller -
James Bensley -
James Braunegg -
Jerry Cloe -
jim deleskie -
Jon Lewis -
Laszlo Hanyecz -
Mel Beckman -
Michael Hare -
Olivier Benghozi -
Pierfrancesco Caci -
Randy Bush -
Ryan L -
Saku Ytti -
Stephen Satchell -
Steve Lalonde -
sthaug＠nethelp.no -
Tom Beecher