Hello, we have multiple internet connections of which one is a research network where many medical institutions and universities are also connected to threw out the country. This research network (ORION) also has internet access but is not meant to be used as a primary path to the internet by its customers. Connected to the ORION network are many sites we exchange email with daily who also have multiple internet connections. One of these sites is not reachable by us. After investigating, it was discovered this site is dropping our connections as the path back to use would use a different interface on the firewall ( a Fortinet device) than that which it arrived upon. The admins at this university claim this is by design and for security reasons.. My response was the entire internet is asymmetrical and while this may of been a legitimate concern in the 90's, I don't think its a real concern anymore if things are set up correctly. They suggested we add static routes to our equipment to address this… This seems like a bad idea and I am not comfortable adjusting my routing table to address one site's issues on the internet due to their (not ours) routing/security policies. am I correct here? any comments on this would be greatly appreciated as I'll be called into a meeting to discuss this further (they are digging in their heals in on this, and higher ups are getting involved now). I'd like to arm myself with a few perspectives. thanks very much for your time again, greg -- This message and any attachments may contain confidential and/or privileged information for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this message in error, please contact the sender and delete all copies. Opinions, conclusions or other information contained in this message may not be that of the organization.
The admins at this university claim this is by design and for security reasons.. My response was the entire internet is asymmetrical and while this may of been a legitimate concern in the 90's, I don't think its a real concern anymore if things are set up correctly. They suggested we add static routes to our equipment to address this… This seems like a bad idea and I am not comfortable adjusting my routing table to address one site's issues on the internet due to their (not ours) routing/security policies.
Working in a university environment like you, we do have connectivity to some of those high-speed R&E networks, and or routing policy generally prefers to use those paths if they are available, for reasons of performance (offloading traffic from more traditional transit paths) and cost/cost avoidance, as others have mentioned. Asymmetric routing is always a possibility between two multi-homed networks. I still occasionally have to wrestle with the notion that many people have that asymmetric routing is bad... If the organization at the far end is doing stateful firewalling at the borders of their multi-homed network, then they are probably accustomed to things 'just breaking' more often then they're willing to admit ;) jms
On Fri, 7 Jan 2011 12:40:32 -0500 Greg Whynott <Greg.Whynott@oicr.on.ca> wrote:
we have multiple internet connections of which one is a research network where many medical institutions and universities are also connected to threw out the country. This research network (ORION) also has internet access but is not meant to be used as a primary path to the internet by its customers. Connected to the ORION network are many sites we exchange email with daily who also have multiple internet connections. One of these sites is not reachable by us. After investigating, it was discovered this site is dropping our connections as the path back to use would use a different interface on the firewall ( a Fortinet device) than that which it arrived upon.
Correct me if I'm wrong, I'm not very familiar with ORION, but if it's like some of the research networks in the U.S. have been built in the past, ORION is dedicated high speed, low latency network that interconnects research institutions together. The way these are often used is that you localpref routes you learn from ORION participants so that traffic between each of you goes over the research network. You'd typically want this since the performance is good and there is plenty of capacity available, but it is also paid for, probably through some research grant, helping to reduce the use and expense of your commercial transit. You should be sending your traffic to them via ORION and they likewise. However, if that path is down, then it would make sense for it to go via another route. Hence, asymmetry may happen. Are you not sending the traffic via ORION? If so, then I'd suggest you both have something to fix. :-) John
Thanks John for your input. You are correct, ORION is a dedicated high speed research network. Based on the fact that we access ORION via one of our ISPs (3rd party, we don't BGP/directly peer with ORION), I'm not sure if i can use this solution here. I could do that for the routes learned from that ISP, but we receive the entire internet routing table from them… I'd have to understand things more before I went down that road. perhaps I shouldn't be accepting the full table from them. the localpref is something I'll look at, thanks for that. I'm not a BGP expert by any stretch, and our requirements here are "simple". we are not a transit. I've only attempted to make the config safe, not efficient. i'd like to hear what you have to say about the original question, is there good reason in this day and age to drop traffic as described in the original post in your opinion? -g On Jan 7, 2011, at 1:15 PM, John Kristoff wrote:
On Fri, 7 Jan 2011 12:40:32 -0500 Greg Whynott <Greg.Whynott@oicr.on.ca> wrote:
we have multiple internet connections of which one is a research network where many medical institutions and universities are also connected to threw out the country. This research network (ORION) also has internet access but is not meant to be used as a primary path to the internet by its customers. Connected to the ORION network are many sites we exchange email with daily who also have multiple internet connections. One of these sites is not reachable by us. After investigating, it was discovered this site is dropping our connections as the path back to use would use a different interface on the firewall ( a Fortinet device) than that which it arrived upon.
Correct me if I'm wrong, I'm not very familiar with ORION, but if it's like some of the research networks in the U.S. have been built in the past, ORION is dedicated high speed, low latency network that interconnects research institutions together. The way these are often used is that you localpref routes you learn from ORION participants so that traffic between each of you goes over the research network. You'd typically want this since the performance is good and there is plenty of capacity available, but it is also paid for, probably through some research grant, helping to reduce the use and expense of your commercial transit.
You should be sending your traffic to them via ORION and they likewise. However, if that path is down, then it would make sense for it to go via another route. Hence, asymmetry may happen.
Are you not sending the traffic via ORION? If so, then I'd suggest you both have something to fix. :-)
John
-- This message and any attachments may contain confidential and/or privileged information for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this message in error, please contact the sender and delete all copies. Opinions, conclusions or other information contained in this message may not be that of the organization.
Based on the fact that we access ORION via one of our ISPs (3rd party, we don't BGP/directly peer with ORION), I'm not sure if i can use this solution here. I could do that for the routes learned from that ISP, but we receive
On Fri, Jan 07, 2011 at 01:56:00PM -0500, Greg Whynott said: the entire internet routing table from them? I'd have to understand things more before I went down that road. perhaps I shouldn't be accepting the full table from them.
the localpref is something I'll look at, thanks for that. I'm not a BGP
expert by any stretch, and our requirements here are "simple". we are not a transit. I've only attempted to make the config safe, not efficient.
i'd like to hear what you have to say about the original question, is
there good reason in this day and age to drop traffic as described in the original post in your opinion? It sounds like the target site has a possible misconfiguration if this is a long term issue. If they're using the open internet to get back to you and not ORION (when your packets arrived from ORION-based connection), then something is misconfigured or down. The problem is a conflict in the way BGP works and how people assume it works :) BGP is designed to get packets to where they want to go, not drop them if they're going the wrong way. The route back to you via ORION might not be available temporarily for a legitimate reason (outtage, etc), or due to other unintentional side effects of preference configurations. They are likely not seeing a route to you being preferred via ORION. Try some traceroutes from your end and from their end and compare. They're likely different paths. However, that shouldnt be suprising - or a reason to filter traffic, really. Symmetric routes across any chunk (big or small) of the internet are a mythological thing of the past. Don't rely on that being the case at any time. As for your getting a full table from your upstream, you would likely expect and want that. Mixed in would be ORION's prefixes, and if things are working right, you are using an ORION path to get to your target. That's up to the upstream's config preferences for packets destined to go through ORION, so if you are the one using the open internet to get to the target (check your traceroutes), then you need to talk to your upstream and get them to fix their route preferences or whatever link or peering session is down. As for dropping traffic, that's a strong fail-fast signal there. If you want to ensure you are getting the intended bandwidth, say if you needed a 100mbps path guaranteed that ORION can easily give you but the open internet/your respective ISPs cant or wont (or you didnt pay for nor want to), then having it fail immediately instead of using a slow backup path might be what you want. There's a balance to be struck between failing fast thus generating sufficient complaints that pressure to fix the best (and only) path that has the required capacity is done ASAP, but then you are also left with no alternate connectivity to the target in the meantime. Which scenario you prefer is up to you and dependant on your organizations' needs. An alternative is to generate sufficient alarms on the best connection's failure, fail over to slower alternates, alert users to the degraded quality (and modify their expectations and behaviour), and have your respective tech teams respond promptly. This all requires awareness, training and a more sophisticated failure-handling system. (How do you automatically alert all users that the alternate degraded path is in use for eg? Email? Pager?) Having alternate connectivity tends to dilute responce urgency from my experience. A question of discipline/(dis)incentives. :) As for using an outtage as a security measure, yes you will reduce the opportunities for the open internet to be a source of spoofed packets from other 'semi-trusted' entities that you expect to only go across ORION for. However, it sounds like you have no opportunity to determine such packets' arrivals/departures, as it all goes through your upstream(s). The other end however seems to have a firewall system that does determine this disparity of paths. This is a minor security benefit, IMHO, which may be worth it to you, depending on how important some connectivity is vs the increased risk of spoofed packets from the general internet during the primary link via ORION's downtime. And, it seems this is the other org's decision to make, not yours, since you dont directly control your own ORION peering, and you are getting a full routing table from your upstream. If you wanted to control your ORION traffic, you would likely get a second BGP feed and link from your upstream if you cant connect to ORION directly somehow. Are you not on TORIX? If so you could connect to ORION directly as we at Heavy Computing do. /kc
-g
On Jan 7, 2011, at 1:15 PM, John Kristoff wrote:
On Fri, 7 Jan 2011 12:40:32 -0500 Greg Whynott <Greg.Whynott@oicr.on.ca> wrote:
we have multiple internet connections of which one is a research network where many medical institutions and universities are also connected to threw out the country. This research network (ORION) also has internet access but is not meant to be used as a primary path to the internet by its customers. Connected to the ORION network are many sites we exchange email with daily who also have multiple internet connections. One of these sites is not reachable by us. After investigating, it was discovered this site is dropping our connections as the path back to use would use a different interface on the firewall ( a Fortinet device) than that which it arrived upon.
Correct me if I'm wrong, I'm not very familiar with ORION, but if it's like some of the research networks in the U.S. have been built in the past, ORION is dedicated high speed, low latency network that interconnects research institutions together. The way these are often used is that you localpref routes you learn from ORION participants so that traffic between each of you goes over the research network. You'd typically want this since the performance is good and there is plenty of capacity available, but it is also paid for, probably through some research grant, helping to reduce the use and expense of your commercial transit.
You should be sending your traffic to them via ORION and they likewise. However, if that path is down, then it would make sense for it to go via another route. Hence, asymmetry may happen.
Are you not sending the traffic via ORION? If so, then I'd suggest you both have something to fix. :-)
John
--
This message and any attachments may contain confidential and/or privileged information for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this message in error, please contact the sender and delete all copies. Opinions, conclusions or other information contained in this message may not be that of the organization.
-- Ken Chase - ken@heavycomputing.ca - +1 416 897 6284 - Toronto CANADA Heavy Computing - Clued bandwidth, colocation and managed linux VPS @151 Front St. W.
Thanks Ken, Some good stuff there, thanks. Since my original email, i think i've come up with a partial solution not requiring the far end's involvement. If not, at least it would get us into a better position to utilize the ORION network when possible. We peer over a L2 tunnel with a router down in the states threw one of our ISP's 10G links, I'm going to see if ORION will do the same with us. This would allow us to establish a BGP session directly with the ORION router, then I could use the localpref options, which may help. this problem is intermitting, most of the time things are fine. doing the above isn't going to help if path/route conditions change, but at least we'll have done all we could within reason and have a proper config. I didn't consider the reasons you mentioned related to 'fail fast', that does make a lot of sense. this is not the reason they claim this policy is in place, it is for security reasons. we access ORION via GTAnet, they are within/part of/something to do with the UoT, and we are across the street. take care, greg @Anthony Pardini <tony@pardini.org> On Jan 7, 2011, at 2:45 PM, Anthony Pardini wrote:
Firewalls aren't routers and pretty much all of them behave in the similar manner.
oh! thanks. 8) On Jan 7, 2011, at 2:37 PM, Ken Chase wrote:
It sounds like the target site has a possible misconfiguration if this is a long term issue. If they're using the open internet to get back to you and not ORION (when your packets arrived from ORION-based connection), then something is misconfigured or down. The problem is a conflict in the way BGP works and how people assume it works :) BGP is designed to get packets to where they want to go, not drop them if they're going the wrong way.
-- This message and any attachments may contain confidential and/or privileged information for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this message in error, please contact the sender and delete all copies. Opinions, conclusions or other information contained in this message may not be that of the organization.
On Fri, Jan 07, 2011 at 03:13:02PM -0500, Greg Whynott said:
Thanks Ken,
Some good stuff there, thanks.
Since my original email, i think i've come up with a partial solution not requiring the far end's involvement. If not, at least it would get us into a better position to utilize the ORION network when possible. We peer over a L2 tunnel with a router down in the states threw one of our ISP's 10G links, I'm going to see if ORION will do the same with us. This would allow us to establish a BGP session directly with the ORION router, then I could use the localpref options, which may help.
this problem is intermitting, most of the time things are fine. doing the above isn't going to help if path/route conditions change, but at least we'll have done all we could within reason and have a proper config.
I didn't consider the reasons you mentioned related to 'fail fast', that does make a lot of sense. this is not the reason they claim this policy is in place, it is for security reasons.
we access ORION via GTAnet, they are within/part of/something to do with the UoT, and we are across the street.
Intermittent sounds exactly like what's happening - and alternate routes are being chosen when the main link or peering session is down. And their firewall isnt liking the alternate route and blocking packets. (oddly if their policy is simple enough, you sending packets to them also across the open internet so you both are using it to eachother, might make things work - with a further reduction in (perceived?) security. :) Yes, peering directly with ORION would allow you some control of outgoing packets if that's the issue - ie the issue is you sending via open internet. But if the issue is you receiving via open internet (ie the far side is sending via open internet because their ORION routes are not being preferred to you due to outtage, etc), then you'll have to work with them on their side of the issue. Localpref and other big hammer approaches to preferences are effective but indelicate, and only work on outgoing traffic. Engineering incoming traffic is a black art (there are several vendors that will sell you gear and access to help you of course :) If you are going through an upstream that is also on ORION, their concerns for routing to your target should be convergent with yours. Im suspecting that the issue is at the far end with their firewalling policy and link/session, which are harder for you to fix directly. You could also get a lan extension between your site and theirs directly if you wanted to peer with that entity directly, if signficant/frequent valuable traffic between you is sufficient, and you do not want it to ever pass over the open internet. good luck. /kc
take care, greg
@Anthony Pardini <tony@pardini.org> On Jan 7, 2011, at 2:45 PM, Anthony Pardini wrote:
Firewalls aren't routers and pretty much all of them behave in the similar manner.
oh! thanks. 8)
On Jan 7, 2011, at 2:37 PM, Ken Chase wrote:
It sounds like the target site has a possible misconfiguration if this is a long term issue. If they're using the open internet to get back to you and not ORION (when your packets arrived from ORION-based connection), then something is misconfigured or down. The problem is a conflict in the way BGP works and how people assume it works :) BGP is designed to get packets to where they want to go, not drop them if they're going the wrong way.
--
This message and any attachments may contain confidential and/or privileged information for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this message in error, please contact the sender and delete all copies. Opinions, conclusions or other information contained in this message may not be that of the organization.
-- Ken Chase - ken@heavycomputing.ca - +1 416 897 6284 - Toronto CANADA Heavy Computing - Clued bandwidth, colocation and managed linux VPS @151 Front St. W.
On Fri, 7 Jan 2011 13:56:00 -0500 Greg Whynott <Greg.Whynott@oicr.on.ca> wrote:
the localpref is something I'll look at, thanks for that. I'm not a BGP expert by any stretch, and our requirements here are "simple". we are not a transit. I've only attempted to make the config safe, not efficient.
I'm not quite sure I understand what the paths look like, but you could also append your ASN once or twice for your routes on the less preferred path to make the other institution use the more preferred one as long as it is available.
i'd like to hear what you have to say about the original question, is there good reason in this day and age to drop traffic as described in the original post in your opinion?
Depends on who you ask. I think it clearly shows a mismatch in the assumptions of security devices, engineers and the actual behavior of some deployed networks. Since you're both part of ORION, ideally packets would be following the same path in both directions. I suggest you endeavor to make that the common case. However, to answer your question, dropping packets because the path is asymmetrical would not be something I'd want my university network to be doing. I'd love for them to tell me it's happening though. John
You can allow asymmetric traffic on the Fortinet, but you lose some functionality. Firewalls aren't routers and pretty much all of them behave in the similar manner. On Fri, Jan 7, 2011 at 11:40 AM, Greg Whynott <Greg.Whynott@oicr.on.ca> wrote:
Hello,
we have multiple internet connections of which one is a research network where many medical institutions and universities are also connected to threw out the country. This research network (ORION) also has internet access but is not meant to be used as a primary path to the internet by its customers. Connected to the ORION network are many sites we exchange email with daily who also have multiple internet connections. One of these sites is not reachable by us. After investigating, it was discovered this site is dropping our connections as the path back to use would use a different interface on the firewall ( a Fortinet device) than that which it arrived upon.
The admins at this university claim this is by design and for security reasons.. My response was the entire internet is asymmetrical and while this may of been a legitimate concern in the 90's, I don't think its a real concern anymore if things are set up correctly. They suggested we add static routes to our equipment to address this… This seems like a bad idea and I am not comfortable adjusting my routing table to address one site's issues on the internet due to their (not ours) routing/security policies.
am I correct here? any comments on this would be greatly appreciated as I'll be called into a meeting to discuss this further (they are digging in their heals in on this, and higher ups are getting involved now). I'd like to arm myself with a few perspectives.
thanks very much for your time again,
greg
--
This message and any attachments may contain confidential and/or privileged information for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this message in error, please contact the sender and delete all copies. Opinions, conclusions or other information contained in this message may not be that of the organization.
Tarig Yassin Ahmed On Jan 7, 2011, at 10:45 PM, Anthony Pardini <tony@pardini.org> wrote:
You can allow asymmetric traffic on the Fortinet, but you lose some functionality. Firewalls aren't routers and pretty much all of them behave in the similar manner.
Hi I think u can solve this issue only by adding router between the firewall and the Internet. in multihoming metwork, Internet connections should be connect to routers then afterthat come the the firewall to avoid such problems. Thanks
On Fri, Jan 7, 2011 at 11:40 AM, Greg Whynott <Greg.Whynott@oicr.on.ca> wrote:
Hello,
we have multiple internet connections of which one is a research network where many medical institutions and universities are also connected to threw out the country. This research network (ORION) also has internet access but is not meant to be used as a primary path to the internet by its customers. Connected to the ORION network are many sites we exchange email with daily who also have multiple internet connections. One of these sites is not reachable by us. After investigating, it was discovered this site is dropping our connections as the path back to use would use a different interface on the firewall ( a Fortinet device) than that which it arrived upon.
The admins at this university claim this is by design and for security reasons.. My response was the entire internet is asymmetrical and while this may of been a legitimate concern in the 90's, I don't think its a real concern anymore if things are set up correctly. They suggested we add static routes to our equipment to address this… This seems like a bad idea and I am not comforta ble adjusting my routing table to address one site's issues on the internet due to their (not ours) routing/security policies.
am I correct here? any comments on this would be greatly appreciated as I'll be called into a meeting to discuss this further (they are digging in their heals in on this, and higher ups are getting involved now). I'd like to arm myself with a few perspectives.
thanks very much for your time again,
greg
--
This message and any attachments may contain confidential and/or privileged information for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this message in error, please contact the sender and delete all copies. Opinions, conclusions or other information contained in this message may not be that of the organization.
participants (7)
-
Anthony Pardini -
Greg Whynott -
John Kristoff -
Justin M. Streiner -
Ken Chase -
Robert Bonomi -
Tarig Ahmed