Wow, very fast responses, Thanks Larry Sheldon and Ricardo Tavares! On 27 May 2010 18:07, Ricardo Tavares <curupas@gmail.com> wrote:

Not sure if I correctly undestand you but default route its the route that the packet must follow if it do not have a specific route for the destination, so, if the next-hop for the source IP (3.3.3.3) is not in the route table then the packet will follow the default route (ISP1).

Yes I believe that would be the default if the session was initiated on the inside, but if it comes from outside on a particular interface which is not the default route, why would the router then send the packet out another interface? Should the device not route session-based traffic according to where it originated?

So, this behavior will be correct if next-hop for 3.3.3.0/24 is not installed. Just for troubleshooting purpose install a static route like:

set routing-options static route 3.3.3.0/24 next-hop <the-correct-gateway-address> (ISP2)

Yes sir, this works, but when you change the static route to point 0.0.0.0/0to the next hop on the virtual router for the particular interface (ISP2) it starts going over the interface for ISP1 again. I also set qualified-next-hop for ISP2 in the main routing table to no avail.

If this works fine then verify the route table, are you using BGP to receive such routing info? If you are not filtering the update maybe the sender is. Verify the received routes using the "show route protocol bgp receive-protocol bgp x.x.x.x" (x.x.x.x is the bgp neighbor)

Yes sir, I have also gone to the extent of deactivating BGP and using only static routes. Thanks for your help! Regards, Ken

f the route announce is coming from the BGP neighbor you need to verify if the next-hop indicated for this route is itself reached by the router, if by recursion the router do not resolve how to go to the next-hop then the announced route will be not available. THe bgp sender must set the next-hop with a reachable address, sometimes this is achieved by the sender using the next-hop-self in the export policy, but it is possible other situations where the next-hop is unreachable. If the sender is using a specific address for all the next-hops for all the announced routes you will need just a static route pointing to the gateway for his next-hop. If the BGP session for some reasons goes down then the default route will apply and the redundancy through ISP1 will work fine. Best Regards, Ricardo On Thu, May 27, 2010 at 9:38 PM, Ken Gilmour <ken.gilmour@gmail.com> wrote:

Wow, very fast responses, Thanks Larry Sheldon and Ricardo Tavares!

On 27 May 2010 18:07, Ricardo Tavares <curupas@gmail.com> wrote:

...

Not sure if I correctly undestand you but default route its the route that the packet must follow if it do not have a specific route for the destination, so, if the next-hop for the source IP (3.3.3.3) is not in the route table then the packet will follow the default route (ISP1).

Yes I believe that would be the default if the session was initiated on the inside, but if it comes from outside on a particular interface which is not the default route, why would the router then send the packet out another interface? Should the device not route session-based traffic according to where it originated?

...

So, this behavior will be correct if next-hop for 3.3.3.0/24 is not installed. Just for troubleshooting purpose install a static route like:

set routing-options static route 3.3.3.0/24 next-hop <the-correct-gateway-address> (ISP2)

Yes sir, this works, but when you change the static route to point 0.0.0.0/0 to the next hop on the virtual router for the particular interface (ISP2) it starts going over the interface for ISP1 again. I also set qualified-next-hop for ISP2 in the main routing table to no avail.

...

If this works fine then verify the route table, are you using BGP to receive such routing info? If you are not filtering the update maybe the sender is. Verify the received routes using the "show route protocol bgp receive-protocol bgp x.x.x.x" (x.x.x.x is the bgp neighbor)

Yes sir, I have also gone to the extent of deactivating BGP and using only static routes.

Thanks for your help!

Regards,

Ken

On Thu, May 27, 2010 at 8:38 PM, Ken Gilmour <ken.gilmour@gmail.com> wrote:

Yes I believe that would be the default if the session was initiated on the inside, but if it comes from outside on a particular interface which is not the default route, why would the router then send the packet out another interface? Should the device not route session-based traffic according to where it originated?

Ken, As others have pointed out typically interfaces are not kept track of in state tables. Having said that, I've worked in the past with the ScreenOS based SSG platforms that do this. So if you're coming from an SSG background this makes sense. These devices seem to keep track of source interface in their state tables. For example I've worked on a one-arm'ed Load Balancer with no Source NAT such that one would typically require some policy based routing to get the traffic back to the LB, to be have the Destination NAT handled. However, with a Juniper SSG, as the router, it's state tables kept track of the interfaces and routed traffic correctly without any policy based routing required. When I took over administration of that environment I spent some time trying to figure out how the routing worked since there was no configuration such as policy based routes that would make sense. Having said that, If the JunOS based SRX platform does not do session tracking in the same was as the SSG platform it would seem that the most reasonable solution would be to NAT the traffic as has already been pointed out. Mark -- Cheers! Mark Hermsdorfer

On Fri, May 28, 2010, Ken Gilmour wrote:

Yes sir I have used SSG for several years but mainly used BSD for the last decade and most recently OpenBSD. There is an easy fix for this on PF for OpenBSD and that is to tag the packets from each provider (as in not using 802.1q but a specific function in PF). This works extremely well

That keeps per-connection state. Be aware of the repercussions! Adrian

Ken Gilmour wrote:

Yes I believe that would be the default if the session was initiated on the inside, but if it comes from outside on a particular interface which is not the default route, why would the router then send the packet out another interface? Should the device not route session-based traffic according to where it originated?

To my knowledge, routers don't generally route based on session. They maintain flow information is cases, but you learn quickly that it's a one way record, and the corresponding flows may have a different path. There are exceptions, and perhaps some Junipers even support more oddball session based routing, though my m120 and cisco don't seem to. Jack

That would seem to be a good resolution (Firewall/NAT) . Aside from that, perhaps a load balancer for each segment might help? One question that comes to mind is why (if ISP2 is a backup) would valid traffic be using that route? Unless maybe your loadbalancing using a DNS round robin perhaps to hit the second IP space or loadbalancing the 2 ISPs? Another "maybe" resolve would be to multi-home the application to that segment, i.e. 2 nics on the server, one on the primary network, the other on the secondary with appropriate Def.GWs, of course since there is little information on the infrastructure here this may not be possible. I suppose if one were to get really detailed about this, you could look into reverse routing using MAC, but theoretically that would/could open a whole other set of issues. Regards, Joe Blanchard Jian Gu wrote:

Wouldn't simply configure source NAT on firewall 2.2.2.1 resolve the problem gracefully? when connection requests coming in through ISP2, source NAT the incoming traffic's source IP with IPs on firewall inside interface, that way when server replies, firewall 2.2.2.1 will guarantee to receive the ACK because ACK traffic won't follow default routing to ISP1.

On Thu, May 27, 2010 at 4:27 PM, Ken Gilmour <ken.gilmour@gmail.com> wrote:

...

Hi all,

I have a very peculiar situation here that i seem to have difficulty explaining in such a way for people to understand. I just got off the phone to the Juniper Devs after about 4 hours with no result. They understand the problem but can't seem to think of a working solution (last solution led to the primary firewall hard crashing and then failing over after a commit (which also makes me wonder what made the primary crash and not the secondary)). I am wondering if there is anyone "creative" on the list who has encountered and worked around this problem before...

Here goes *sigh*

ISP1 - 1.1.1.0/24 ISP2 - 2.2.2.0/24

ISP1 is the default gateway, ISP2 is a backup provider but which is always active. Client comes in on ISP1's link, traffic goes back out on ISP1s link. Client comes in on ISP2's link (non default gateway) but for some reason, the packets seem to be going back out through the link for ISP1.

So look at it this way:

SYN comes from client at 3.3.3.3 aimed at 2.2.2.2, packet is received by the firewall. Firewall sends a SYN/ACK but the firewall at 1.1.1.1 sees it in TCPDump, the firewall at 2.2.2.1 never sees it.

Here's a log snippet (I can send you more if you need:

May 27 21:38:49 21:38:48.1509569:CID-1:RT: route lookup: dest-ip 3.3.3.3 *orig ifp reth3.0* *output_ifp reth2.0* orig-zone 19 out-zone 19 vsd 3

You will see that the orig and out zones are the same zone, however this was a last ditch effort (putting both interfaces into one zone, effectively creating a swamp).

Our current (non-preferred) solution is to put match-all rules on our Catalyst 6513s and put both providers into a swamp and the switch will then intercept the packets if they are destined for the wrong interface and send them out the right one based on a bunch of boolean.

We've tried setting up a virtual instance on the offending interface and a firewall filter, but this had little to no effect (at one point it stopped passing the packets to the end machine altogether). We're using small SRX 650ies. Why do we want to do it this way you ask? In the event of a BGP session failure we need to be able to use our statically routed IPs and rely on someone else.

Thanks!

Ken

Hi Jian, Yes sir that's what I thought too. The packets are being NATted (and I also used a bit of DNAT for port forwarding to test the theory) but the result is the same. Regards, Ken On 27 May 2010 23:46, Jian Gu <guxiaojian@gmail.com> wrote:

Wouldn't simply configure source NAT on firewall 2.2.2.1 resolve the problem gracefully? when connection requests coming in through ISP2, source NAT the incoming traffic's source IP with IPs on firewall inside interface, that way when server replies, firewall 2.2.2.1 will guarantee to receive the ACK because ACK traffic won't follow default routing to ISP1.

...

Hi all,

I have a very peculiar situation here that i seem to have difficulty explaining in such a way for people to understand. I just got off the

...

to the Juniper Devs after about 4 hours with no result. They understand

...

problem but can't seem to think of a working solution (last solution led to the primary firewall hard crashing and then failing over after a commit (which also makes me wonder what made the primary crash and not the secondary)). I am wondering if there is anyone "creative" on the list who has encountered and worked around this problem before...

Here goes *sigh*

ISP1 - 1.1.1.0/24 ISP2 - 2.2.2.0/24

ISP1 is the default gateway, ISP2 is a backup provider but which is always active. Client comes in on ISP1's link, traffic goes back out on ISP1s

...

Client comes in on ISP2's link (non default gateway) but for some reason, the packets seem to be going back out through the link for ISP1.

So look at it this way:

SYN comes from client at 3.3.3.3 aimed at 2.2.2.2, packet is received by

...

firewall. Firewall sends a SYN/ACK but the firewall at 1.1.1.1 sees it in TCPDump, the firewall at 2.2.2.1 never sees it.

Here's a log snippet (I can send you more if you need:

May 27 21:38:49 21:38:48.1509569:CID-1:RT: route lookup: dest-ip 3.3.3.3 *orig ifp reth3.0* *output_ifp reth2.0* orig-zone 19 out-zone 19 vsd 3

You will see that the orig and out zones are the same zone, however this was a last ditch effort (putting both interfaces into one zone, effectively creating a swamp).

Our current (non-preferred) solution is to put match-all rules on our Catalyst 6513s and put both providers into a swamp and the switch will

On Thu, May 27, 2010 at 4:27 PM, Ken Gilmour <ken.gilmour@gmail.com> wrote: phone the link. the then

...

intercept the packets if they are destined for the wrong interface and send them out the right one based on a bunch of boolean.

We've tried setting up a virtual instance on the offending interface and a firewall filter, but this had little to no effect (at one point it stopped passing the packets to the end machine altogether). We're using small SRX 650ies. Why do we want to do it this way you ask? In the event of a BGP session failure we need to be able to use our statically routed IPs and rely on someone else.

Thanks!

Ken

Ken Gilmour wrote:

Strangely, BGP actually works without issues. The only issue is with statically routed ranges.

Same rules apply, just without control on your end. If a packet hits ISP2, ISP2 will send it to you by most ISP standards (prefer direct customers over peers). Outbound, you will send your normal route (you prefer ISP1). There are methods of handling session based routing in some products from what I gather, but in traditional routing, each direction of a session is independent (session = 2 flows) and the router is unaware of actual sessions. Some real world examples I've dealt with which reduces asymmetric routing, though not always eliminates it. 1) full backup ISP (we don't use it unless there's no other options) a. AS prepend (tell the outside world we prefer them not to come this way) b. community to ISP setting local pref (if the packet does hit provider, tell the provider we prefer them to use their external peer over sending direct to us). c. set local pref on received routes so they are least preferred. 2) backup with partial traffic (generally we want to receive and send packets via this ISP if the customer is directly connected to them). a. AS prepend (least preferred way to reach me) b. set local pref on received routes based on providers communities (perhaps we'll only send this way if it's a non-bgp customer, or to any network which didn't come through exchange points; very ISP dependent). The goal of the second is to reduce asymmetric traffic, while allowing us to use the backup link to reach the ISP's networks and their directly connected customers. Some multihomed customers may still go asymmetric. Primarily used in cases where ISP has piss poor exchange connectivity at times, so you want to reach their customers without going the long way around through the exchanges. The first I've used before with split network scenarios, where one provider handles some networks, and the other provider handles other networks. Setting the local pref forces traffic even on ISP2 (backup ISP for specific network) to make it's way to ISP1 (primary ISP for the specific network) instead of coming direct (suboptimal, but symmetric). Source address based policy rules pushed traffic back out the correct path for that network so long as it was available. Jack

When you set-up your virtual-instance, was your ISP2 interface a member of that instance? I have a working setup that ran on a J-series running 9.6 something. This is a Juniper guide I used but it was a little bit different and didn't work for me. http://www.juniper.net/us/en/local/pdf/day-one-guides/7100110-en.pdf I used below: --Routing instance: routing-instances { DSL { instance-type virtual-router; interface ge-0/0/1.0; // <-- Most important part - ISP2 Interface must be a member to get correct incoming context. routing-options { static { route 0.0.0.0/0 next-hop 172.24.1.1; // ISP2 next hop } } } } --Firewall filter: firewall { filter DSL { term DSL { from { address { 192.0.1.210/32; // This is the address that will go into the virtual router (ISP2 addresses should go here) } } then { count source-route; log; routing-instance DSL; } } term default { // Everything else uses default routing table. then { count defualt-counter; log; accept; } } } } Jensen Tyler Network Engineer Fiberutilities Group, LLC -----Original Message----- From: Joe Maimon [mailto:jmaimon@ttec.com] Sent: Friday, May 28, 2010 9:14 AM To: Ken Gilmour Cc: nanog@nanog.org Subject: Re: Junos Asymmetric Routing Firewalls that can route and service properly multiple "untrusts"? Good luck. Hit or miss. Constant flux. Place a router in front of it and that will get you a ways there. Ken Gilmour wrote:

Hi all,

I have a very peculiar situation here that i seem to have difficulty explaining in such a way for people to understand. I just got off the phone to the Juniper Devs after about 4 hours with no result. They understand the problem but can't seem to think of a working solution (last solution led to the primary firewall hard crashing and then failing over after a commit (which also makes me wonder what made the primary crash and not the secondary)). I am wondering if there is anyone "creative" on the list who has encountered and worked around this problem before...

Here goes *sigh*

ISP1 - 1.1.1.0/24 ISP2 - 2.2.2.0/24

ISP1 is the default gateway, ISP2 is a backup provider but which is always active. Client comes in on ISP1's link, traffic goes back out on ISP1s link. Client comes in on ISP2's link (non default gateway) but for some reason, the packets seem to be going back out through the link for ISP1.

So look at it this way:

SYN comes from client at 3.3.3.3 aimed at 2.2.2.2, packet is received by the firewall. Firewall sends a SYN/ACK but the firewall at 1.1.1.1 sees it in TCPDump, the firewall at 2.2.2.1 never sees it.

Here's a log snippet (I can send you more if you need:

May 27 21:38:49 21:38:48.1509569:CID-1:RT: route lookup: dest-ip 3.3.3.3 *orig ifp reth3.0* *output_ifp reth2.0* orig-zone 19 out-zone 19 vsd 3

You will see that the orig and out zones are the same zone, however this was a last ditch effort (putting both interfaces into one zone, effectively creating a swamp).

Our current (non-preferred) solution is to put match-all rules on our Catalyst 6513s and put both providers into a swamp and the switch will then intercept the packets if they are destined for the wrong interface and send them out the right one based on a bunch of boolean.

We've tried setting up a virtual instance on the offending interface and a firewall filter, but this had little to no effect (at one point it stopped passing the packets to the end machine altogether). We're using small SRX 650ies. Why do we want to do it this way you ask? In the event of a BGP session failure we need to be able to use our statically routed IPs and rely on someone else.

Thanks!

Ken

Hi Joe, On 28 May 2010 08:13, Joe Maimon <jmaimon@ttec.com> wrote:

Firewalls that can route and service properly multiple "untrusts"?

Good luck. Hit or miss. Constant flux.

Place a router in front of it and that will get you a ways there.

We replaced our OpenBSD routers with these SRXes since they were supposed to be multifunction devices (gateways and routers at the same time) which was the selling point. So we expected them to do asymmetric routing and were told they could, easily, but apparently they are not acting normally and also the configuration is perfect according to JTAC.

You need to put a filter on your interfaces that references a filter later on to not session track a flow.  I think you need to be running Junos-jsr[0] 10.0 or 10.1 to use this :

The same goes for 9.x, just be sure to except traffic to the router (like BGP session) from the packet-mode, they need to be in flow mode. Traffic routed from one interface to another can be in packet mode. Rubens

your perfectly fine multihop BGP session could break when rerouting occurs.

one of the many reasons that there are no perfectly fine multi-hop bgp sessions. randy

...

...

your perfectly fine multihop BGP session could break when rerouting occurs. one of the many reasons that there are no perfectly fine multi-hop bgp sessions. Uhm, is there a way around them when building the iBGP mesh?

nope

* Rubens Kuhl:

On Sun, May 30, 2010 at 1:46 PM, Florian Weimer <fw@deneb.enyo.de> wrote:

...

* Randy Bush:

...

...

your perfectly fine multihop BGP session could break when rerouting occurs.

one of the many reasons that there are no perfectly fine multi-hop bgp sessions.

Uhm, is there a way around them when building the iBGP mesh?

Not using flow-based devices as datagram routers when they aren't ?

Then the sessions are still multi-hop. 8-) Flow-based forwarding is only bad if you notice it. Usually, that happens with random packets you don't want to forward anyway, but Juniper managed to build something that's unusable in a fairly wide range of scenarios.

On Sun, May 30, 2010 at 10:16:14AM -0700, Kevin Oberman wrote:

I remember a posting to this list back in the late 90s from Tony Li, who knows a bit about BGP. He urged that multi-hop BGP never be used and pointed out that it had not been intended for use except as a test tool, not a production one and should have been stripped from IOS before it was shipped.

While there are a few good cases for using it, it is generally a bad, bad idea. And this thread demonstrates that he had reason for the warning

I think you guys may be getting a tad carried away with the crusade against multihop BGP. The only thing you're really giving up when you use it is liveness detection, which as we all know BGP is actually pretty terrible at implementing anyways (hows that 180 sec IOS default working out for you?). There are much better mechanisms out there, like BFD, which could be used to provide better liveness detection to BGP through nexthop invalidation. I'm not saying everyone should run out and do all their peering over multihop EBGP without carefully considering a replacement for the liveness detection component, I just hate it when people get religious about such a simple concept for no good reason (well, other than Randy Bush getting to do his best Andy Rooney impersonation :P). Multihop BGP is no more evil than anything else we do with the Internet, and the fact that we've all managed to use it successfully for IBGP proves that it can work out just fine. There are some pretty interesting things you can accomplish as far as large scale traffic engineering if you can free yourself from the requirement of speaking EBGP with a directly connected neighbor, processed by whatever slow overpriced router CPU could be stuffed into that box. Again, I just hate to see the concepts dismissed out of hand because of some old BGP ideology about a problem that can be addressed any number of other ways. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)