I'm reasonable certain a customer of ours who is using one of our netblocks is using a different reverse path to reach us. How might I figure out who is allowing them to source traffic from IPs that belong to us?
I'm reasonable certain a customer of ours who is using one of our netblocks is using a different reverse path to reach us. How might I figure out who is allowing them to source traffic from IPs that belong to us?
you are implying that they are not allowed to multi-home using the ip space you have assigned to them. good way to lose a customer. randy
Randy Bush wrote:
I'm reasonable certain a customer of ours who is using one of our netblocks is using a different reverse path to reach us. How might I figure out who is allowing them to source traffic from IPs that belong to us?
you are implying that they are not allowed to multi-home using the ip space you have assigned to them. good way to lose a customer.
randy
Does it count as multihoming when we are the only ones announcing the space?
I'm reasonable certain a customer of ours who is using one of our netblocks is using a different reverse path to reach us. How might I figure out who is allowing them to source traffic from IPs that belong to us? you are implying that they are not allowed to multi-home using the ip space you have assigned to them. good way to lose a customer. Does it count as multihoming when we are the only ones announcing the space?
almost an interesting question. but i think it is playing with words. if i understand your original statement, they are clearly attached to at least two providers. perhaps it is fear of what they, possibly mistakenly, perceive to be your policy regarding announcement of space that keeps them from announcing normally to both, or more, links? randy
On Fri, Nov 27, 2009 at 11:23 PM, Randy Bush <randy@psg.com> wrote:
perhaps it is fear of what they, possibly mistakenly, perceive to be your policy regarding announcement of space that keeps them from announcing normally to both, or more, links?
or maybe just better pricing on the other provider, and that provider offers something akin to 'no-export' (or maybe they just used no-export). -chris
I'm reasonable certain a customer of ours who is using one of our netblocks is using a different reverse path to reach us. How might I figure out who is allowing them to source traffic from IPs that belong to us? you are implying that they are not allowed to multi-home using the ip space you have assigned to them. good way to lose a customer. Does it count as multihoming when we are the only ones announcing the space?
almost an interesting question. but i think it is playing with words. if i understand your original statement, they are clearly attached to at least two providers.
perhaps it is fear of what they, possibly mistakenly, perceive to be your policy regarding announcement of space that keeps them from announcing normally to both, or more, links?
It could also be something simple like pricing. For example, in a large colo facility, you might easily find that a number of providers offer low cost transit, but not IP space. For a customer who is heavy on the outbound traffic, they might find it more affordable to buy their inbound plus IP space from you, and then dump onto Cogent or something like that for outbound. Unless your contract specifically prohibits this, you're probably not going to be able to prevent it. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
On Sat, Nov 28, 2009 at 09:41:09AM -0600, Joe Greco wrote: [attributions lost]
I'm reasonable certain a customer of ours who is using one of our netblocks is using a different reverse path to reach us. How might I figure out who is allowing them to source traffic from IPs that belong to us? you are implying that they are not allowed to multi-home using the ip space you have assigned to them. good way to lose a customer. Does it count as multihoming when we are the only ones announcing the space?
almost an interesting question. but i think it is playing with words. if i understand your original statement, they are clearly attached to at least two providers.
perhaps it is fear of what they, possibly mistakenly, perceive to be your policy regarding announcement of space that keeps them from announcing normally to both, or more, links?
It wasn't clear that the customer was a BGP downstream though by saying 'We are the only ones announcing the space', I think not. Non-BGP multihoming is broken* and when not done out of ignorance generally is the smoke pointing to the fire of someone trying to hide something. Was very common for spammers to abuse no-uRPF networks in the early days of broadband.
It could also be something simple like pricing. For example, in a large colo facility, you might easily find that a number of providers offer low cost transit, but not IP space. For a customer who is heavy on the outbound traffic, they might find it more affordable to buy their inbound plus IP space from you, and then dump onto Cogent or something like that for outbound. Unless your contract specifically prohibits this, you're probably not going to be able to prevent it.
I wonder if there is a drift of baseline assumptions between the current wave of operators and previous ones? To me (and BCP38) it is beyond bad practice to allow -and if allowed, to make use of- such sloppy edges. If the other network truly is practicing bad forwarding hygiene then they are a security problem for everyone else and IMO would be good for naming and shaming. Cheers, Joe * for the majority of the cases. I know there are purposeful Non-BGP MOAS/anycast purposefully run by those who understand the implications. It is unfortunate that their use of lack of inherent BGP path security contribute to fuzzing what would otherwise have been a clear indicator of 'bad' behavior. But same could be said for the deaggregators using longest-match to have everyone else do their TE; water under the bridge pushing work onto everyone else. -- RSUC / GweepNet / Spunk / FnB / Usenix / SAGE
On Sat, Nov 28, 2009 at 09:41:09AM -0600, Joe Greco wrote: [attributions lost]
I'm reasonable certain a customer of ours who is using one of our netblocks is using a different reverse path to reach us. How might I figure out who is allowing them to source traffic from IPs that belong to us? you are implying that they are not allowed to multi-home using the ip space you have assigned to them. good way to lose a customer. Does it count as multihoming when we are the only ones announcing the space?
almost an interesting question. but i think it is playing with words. if i understand your original statement, they are clearly attached to at least two providers.
perhaps it is fear of what they, possibly mistakenly, perceive to be your policy regarding announcement of space that keeps them from announcing normally to both, or more, links?
It wasn't clear that the customer was a BGP downstream though by saying 'We are the only ones announcing the space', I think not. Non-BGP multihoming is broken* and when not done out of ignorance generally is the smoke pointing to the fire of someone trying to hide something. Was very common for spammers to abuse no-uRPF networks in the early days of broadband.
This is still rather common for people to do, at least where there's a significant cost differential. There are enough networks that can accept arbitrary traffic that BGP doesn't really play into it at all.
It could also be something simple like pricing. For example, in a large colo facility, you might easily find that a number of providers offer low cost transit, but not IP space. For a customer who is heavy on the outbound traffic, they might find it more affordable to buy their inbound plus IP space from you, and then dump onto Cogent or something like that for outbound. Unless your contract specifically prohibits this, you're probably not going to be able to prevent it.
I wonder if there is a drift of baseline assumptions between the current wave of operators and previous ones? To me (and BCP38) it is beyond bad practice to allow -and if allowed, to make use of- such sloppy edges.
Of course it is.
If the other network truly is practicing bad forwarding hygiene then they are a security problem for everyone else and IMO would be good for naming and shaming.
Sure. But it's easy enough to go to, for example, $BACKBONE_OF_CHOICE and say "I'm delegated 1.2.3.4/24 from $SMALLISP, I would like you to accept traffic from that prefix, here's my SWIP'd WHOIS to prove that" and there are lots of providers for whom that won't be a problem; it is not the same sort of security problem that a complete lack of filtering is. Generally speaking, what's needed is a control over what's being cast into the network.
* for the majority of the cases. I know there are purposeful Non-BGP MOAS/anycast purposefully run by those who understand the implications. It is unfortunate that their use of lack of inherent BGP path security contribute to fuzzing what would otherwise have been a clear indicator of 'bad' behavior. But same could be said for the deaggregators using longest-match to have everyone else do their TE; water under the bridge pushing work onto everyone else.
:-) ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
-----Original Message----- From: ML [mailto:ml@kenweb.org] Sent: Friday, November 27, 2009 10:44 PM
I'm reasonable certain a customer of ours who is using one of our netblocks is using a different reverse path to reach us. How might I figure out who is allowing them to source traffic from IPs that belong to us?
It's been my experience that many providers do not care what address is used for sourcing traffic. This is why it is not uncommon to see traffic sourced from RFC1918 space coming across various providers networks. If more providers adopted BCP 38 this wouldn't be a problem, but that doesn't seem to be happening anytime soon... I'd try to identify which providers the customer is connected to and take it from there... Stefan Fouant www.shortestpathfirst.net GPG Key ID: 0xB5E3803D
On Fri, Nov 27, 2009 at 10:43 PM, ML <ml@kenweb.org> wrote:
I'm reasonable certain a customer of ours who is using one of our netblocks is using a different reverse path to reach us. How might I figure out who is allowing them to source traffic from IPs that belong to us?
Hi, Are they complaining about somehting? If so, ask for a traceroute. You might also try calling and asking. "We're seeing some strange traffic purporting to be from your addresses but not coming from your circuit. We're concerned that someone might be attacking your network. Before taking action to protect you, we want to eliminate the possibility that you have a second ISP through which you're accidentally sourcing the packets." Beyond that, what's your game plan once you know the answer? Threaten to cut them off? That's a great way to lose a customer who you know *already* has a second ISP. Maybe you'll call their second ISP and complain about their filtering practices? I'd love to get that call from you. Tells me exactly which name to pass to my upsell specialist. Yes Mr. Customer, your other ISP is trying to cut you off. They even asked us to block your packets. But for just a little more we'll give you IP addresses that you can use with any ISP. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
On 11/27/09 8:43 PM, ML wrote:
I'm reasonable certain a customer of ours who is using one of our netblocks is using a different reverse path to reach us. How might I figure out who is allowing them to source traffic from IPs that belong to us?
I've had two customers pull this stunt in the past with me - one, a spammer, tried to do this with an ADSL modem from me, the other (a non-spammer with a clueless 'consultant') had a T1 from me and a T1 from UUNet. It started with the T1 customer. I believe they had a smaller block of IPs (less then /24, more like a /25 or /26), and their 'computer consultant' with his infinite wisdom decided to send all outbound traffic through the UUNet T1 rather then source routing which we highly recommended to them. Of course, we had ingress filters in place to block IP ranges we have from coming into us from the WAN links, so when they tried to contact servers on the other half of the netblock on our end, the connections mysteriously failed. After lying up and down that it was our fault, that their computer 'consultant' was regarded as best in the country, blah blah blah, we flipped on logging on the ingress filters out of sheer curiosity and discovered exactly what was going on. The ADSL customer was a bit more tricky - we were getting spam reports about his single IP address sending spam, but we had his outbound port 25 blocked. Ended up sniffing the port off the router he sat off of, and discovering that it was all one sided, wasn't even tickling the ingress filters. Hey, at least your customer didn't convince AT&T to allow them to announce out one of your /24s when all they had was a /29. Your in a tricky bind, I'd approach them under the guise of ingress filtering issues. -- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org / http://www.ahbl.org
Brielle Bruns wrote:
On 11/27/09 8:43 PM, ML wrote:
I'm reasonable certain a customer of ours who is using one of our netblocks is using a different reverse path to reach us. How might I figure out who is allowing them to source traffic from IPs that belong to us?
I've had two customers pull this stunt in the past with me - one, a spammer, tried to do this with an ADSL modem from me, the other (a non-spammer with a clueless 'consultant') had a T1 from me and a T1 from UUNet.
It started with the T1 customer. I believe they had a smaller block of IPs (less then /24, more like a /25 or /26), and their 'computer consultant' with his infinite wisdom decided to send all outbound traffic through the UUNet T1 rather then source routing which we highly recommended to them. Of course, we had ingress filters in place to block IP ranges we have from coming into us from the WAN links, so when they tried to contact servers on the other half of the netblock on our end, the connections mysteriously failed. After lying up and down that it was our fault, that their computer 'consultant' was regarded as best in the country, blah blah blah, we flipped on logging on the ingress filters out of sheer curiosity and discovered exactly what was going on.
The ADSL customer was a bit more tricky - we were getting spam reports about his single IP address sending spam, but we had his outbound port 25 blocked. Ended up sniffing the port off the router he sat off of, and discovering that it was all one sided, wasn't even tickling the ingress filters.
Hey, at least your customer didn't convince AT&T to allow them to announce out one of your /24s when all they had was a /29.
Your in a tricky bind, I'd approach them under the guise of ingress filtering issues.
Brielle is correct. The customer in question is spamming networks and we are having trouble filtering them because another provider allows them to source traffic however they please. Of course the other provider seems to be a spam friendly upstream. Hopefully you can understand why I'm not hopeful of getting anywhere with them either.
(Forgive the top posting, stupid blackberry can't do inline) A creative idea that I did in a test lab one time - stateful connection tracking, its not just for NAT you know. Would require a bit of moving stuff around and reengineering of your connection to them, but it would cripple their connection unless it originated through you. IE: You <-> UNIX/Linux firewall <-> T1/eth/dsl/etc If stuff went out the other way, it would come in, firewall says WTF, and drops it because it didn't see the initial SYN exchange. My partner Tammy says a PIX could probably accomplish the same task (we have some here for the corp lan stuff, including spares). Brielle -- Brielle Bruns http://www.sosdg.org / http://www.ahbl.org -----Original Message----- From: ML <ml@kenweb.org> Date: Sat, 28 Nov 2009 14:14:07 To: nanog@nanog.org<nanog@nanog.org> Subject: Re: Finding asymmetric path Brielle Bruns wrote:
On 11/27/09 8:43 PM, ML wrote:
I'm reasonable certain a customer of ours who is using one of our netblocks is using a different reverse path to reach us. How might I figure out who is allowing them to source traffic from IPs that belong to us?
I've had two customers pull this stunt in the past with me - one, a spammer, tried to do this with an ADSL modem from me, the other (a non-spammer with a clueless 'consultant') had a T1 from me and a T1 from UUNet.
It started with the T1 customer. I believe they had a smaller block of IPs (less then /24, more like a /25 or /26), and their 'computer consultant' with his infinite wisdom decided to send all outbound traffic through the UUNet T1 rather then source routing which we highly recommended to them. Of course, we had ingress filters in place to block IP ranges we have from coming into us from the WAN links, so when they tried to contact servers on the other half of the netblock on our end, the connections mysteriously failed. After lying up and down that it was our fault, that their computer 'consultant' was regarded as best in the country, blah blah blah, we flipped on logging on the ingress filters out of sheer curiosity and discovered exactly what was going on.
The ADSL customer was a bit more tricky - we were getting spam reports about his single IP address sending spam, but we had his outbound port 25 blocked. Ended up sniffing the port off the router he sat off of, and discovering that it was all one sided, wasn't even tickling the ingress filters.
Hey, at least your customer didn't convince AT&T to allow them to announce out one of your /24s when all they had was a /29.
Your in a tricky bind, I'd approach them under the guise of ingress filtering issues.
Brielle is correct. The customer in question is spamming networks and we are having trouble filtering them because another provider allows them to source traffic however they please. Of course the other provider seems to be a spam friendly upstream. Hopefully you can understand why I'm not hopeful of getting anywhere with them either.
On Sat, Nov 28, 2009 at 1:41 PM, Brielle Bruns <bruns@2mbit.com> wrote:
My partner Tammy says a PIX could probably accomplish the same task (we have some here for the corp lan stuff, including spares).
Yes, a PIX/ASA would stop this cold. The TCP state tracking would not allow traffic to pass unless the whole 3-way handshake was observed by the box. Only recently did Cisco add features to make tracking the TCP connection state optional. (http://www.cisco.com/en/US/docs/security/asa/asa82/configuration/guide/conns...) The larger ASA-5580 machines can be virtualized into dozens (or more) security contexts as needed. I imagine it would take some effort to figure out how to cleanly integrate such a configuration into a POP. --D
Actually, this can be achieved easily using reflexive ACLs on any Cisco router, so no real need to change the topology or add new devices in the path: http://www.cisco.com/en/US/products/sw/secursw/ps1018/products_tech_note0918... Arie On Sat, Nov 28, 2009 at 10:26 PM, Duane Waddle <duane.waddle@gmail.com>wrote:
On Sat, Nov 28, 2009 at 1:41 PM, Brielle Bruns <bruns@2mbit.com> wrote:
My partner Tammy says a PIX could probably accomplish the same task (we have some here for the corp lan stuff, including spares).
Yes, a PIX/ASA would stop this cold. The TCP state tracking would not allow traffic to pass unless the whole 3-way handshake was observed by the box. Only recently did Cisco add features to make tracking the TCP connection state optional. ( http://www.cisco.com/en/US/docs/security/asa/asa82/configuration/guide/conns... ) The larger ASA-5580 machines can be virtualized into dozens (or more) security contexts as needed. I imagine it would take some effort to figure out how to cleanly integrate such a configuration into a POP.
--D
On Sat, Nov 28, 2009 at 2:14 PM, ML <ml@kenweb.org> wrote:
Brielle is correct. The customer in question is spamming networks and we are having trouble filtering them because another provider allows them to source traffic however they please.
What trouble? SMTP requires two-way traffic with a static port number that nothing else uses. If for some reason you don't want to simply terminate their account altogether, block packets outbound to your customer sourced from TCP port 25 but not from your SMTP smarthosts. Seriously though, if you can prove they're spamming (regardless of whether the packets pass through your network) save yourself some grief and just terminate the account. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
Yes - term the account would be my recommendation And if you filter port 25 traffic do it both ways Read these old nanog threads .. http://www.irbs.net/internet/nanog/0408/0465.html and http://www.mail-archive.com/nanog@merit.edu/msg28863.html On Sun, Nov 29, 2009 at 3:58 AM, William Herrin <herrin-nanog@dirtside.com> wrote:
On Sat, Nov 28, 2009 at 2:14 PM, ML <ml@kenweb.org> wrote:
Brielle is correct. The customer in question is spamming networks and we are having trouble filtering them because another provider allows them to source traffic however they please.
What trouble? SMTP requires two-way traffic with a static port number that nothing else uses. If for some reason you don't want to simply terminate their account altogether, block packets outbound to your customer sourced from TCP port 25 but not from your SMTP smarthosts.
Seriously though, if you can prove they're spamming (regardless of whether the packets pass through your network) save yourself some grief and just terminate the account.
Regards, Bill Herrin
-- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
-- Suresh Ramasubramanian (ops.lists@gmail.com)
Brielle is correct. The customer in question is spamming networks and we are having trouble filtering them because another provider allows them to source traffic however they please.
then perhaps the issue is a bit larger than their traffic incoming to you. disconnect the schmucks. randy
Brielle is correct. The customer in question is spamming networks and we are having trouble filtering them because another provider allows them to source traffic however they please.
If they are spamming just pull the plug, whatever revenue you get from them is not worth your reputation and caring for other good customers, and the rest of us will speak highly of you for taking another spamcrock out of the net. Cheers Jorge
participants (12)
-
Arie Vayner -
Brielle Bruns -
Christopher Morrow -
Duane Waddle -
Joe Greco -
Joe Provo -
Jorge Amodio -
ML -
Randy Bush -
Stefan Fouant -
Suresh Ramasubramanian -
William Herrin