RE: Backbone IP network Economics - peering and transit
Patrick W.Gilmore wrote: Unless they have cheap access to a free NAP (TorIX, SIX, etc.), transit, even at higher prices, is probably be the best / cheapest way to reach the Internet.
This is true, but there are plenty of other opportunities for peering, such as: both parties buy DS-3 class transit from the same tier-2 or even maybe tier-3 provider in a colo (which will likely be a BFM, other problem) not a formal IX. In other words, peering in an IX does cost money, but peering at a colo might not, as these messy colos are mostly unmanaged and nobody cares about that 25ft cross-over cable :-) Michel.
On 4/20/04 1:34 AM, "Michel Py" <michel@arneill-py.sacramento.ca.us> wrote:
Patrick W.Gilmore wrote: Unless they have cheap access to a free NAP (TorIX, SIX, etc.), transit, even at higher prices, is probably be the best / cheapest way to reach the Internet.
This is true, but there are plenty of other opportunities for peering, such as: both parties buy DS-3 class transit from the same tier-2 or even maybe tier-3 provider in a colo (which will likely be a BFM, other problem) not a formal IX. In other words, peering in an IX does cost money, but peering at a colo might not, as these messy colos are mostly unmanaged and nobody cares about that 25ft cross-over cable :-)
Michel.
This is a classical mistake. Peering always costs money and its never free. The question is, how much, and is it cheaper than transit? Costs incurred in peering: - Port Costs (capex) - A share of a router's backplane capacity corresponding to the port - Cross connect costs (one time or recurring) - Operational costs such as legal review for BLPAs, NOC monitoring, troubleshooting when it flaps, putting MD5 on, etc - Administration - Public Switch costs It is difficult to defend peering strategies today unless your network is of a fairly significant size (gigabits of traffic) and you are collocated in an advantageous location(s). Otherwise, low cost transit is hard to beat. Some of the low cost transit providers will not last, and there WILL be a second round of bankruptcies and consolidations - Googin's words should be heeded. However, if you are multihomed and have sufficient transit diversity (and you don't assume any local loop liabilities), the low cost transit should be exploited while it lasts. -- Daniel Golding Network and Telecommunications Strategies Burton Group
On Apr 20, 2004, at 10:32 AM, Daniel Golding wrote:
On 4/20/04 1:34 AM, "Michel Py" <michel@arneill-py.sacramento.ca.us> wrote:
Patrick W.Gilmore wrote: Unless they have cheap access to a free NAP (TorIX, SIX, etc.), transit, even at higher prices, is probably be the best / cheapest way to reach the Internet.
This is true, but there are plenty of other opportunities for peering, such as: both parties buy DS-3 class transit from the same tier-2 or even maybe tier-3 provider in a colo (which will likely be a BFM, other problem) not a formal IX. In other words, peering in an IX does cost money, but peering at a colo might not, as these messy colos are mostly unmanaged and nobody cares about that 25ft cross-over cable :-)
Michel.
This is a classical mistake. Peering always costs money and its never free.
Maybe "Free" is the wrong word. Perhaps "No additional cost over <transit/whatever>". Or, for those of us who think that the time it takes to plug a patch cable into an unused switch port and do some configuration changes are irrelevant, maybe "free" is the right word. Either way, it is not NEARLY as bad as you or many other people make it out to be. Allow me to explain....
The question is, how much, and is it cheaper than transit?
Costs incurred in peering:
- Port Costs (capex)
Pthhhhhhhhh. In many, many cases, especially for smaller providers, this is a spare FE on a switch which already exists. For mid-sized providers, it is frequently a spare GE port on an existing switch, which means perhaps $500 for GBIC or something. For large providers, there is a cost here. But large providers are a different beast, and there is no way a simple e-mail could possibly capture the complexities implicit in peering between Very Large Providers. So we'll let them figure out their own costs.
- A share of a router's backplane capacity corresponding to the port
Irrelevant. The traffic has to go somewhere, if it does not go out the peering port, it will go out transit, but it is definitely going across the router's backplane. A better thing to put here would be possible use of a router which would not be used. Specifically, if I get a bit in a POP which has transit, I do not have to use the router out at the Peering Point. But how many people have router backpanes which are saturated? At worst you are running out of slots for ports in most cases. (Remember, we left the really big providers to their own devices.)
- Cross connect costs (one time or recurring)
Largely irrelevant - if you are really going to go out-of-biz for a $150/Month x-conn, you have bigger problems.
- Operational costs such as legal review for BLPAs, NOC monitoring, troubleshooting when it flaps, putting MD5 on, etc
These costs are frequently quoted as reasons not to peer by the larger providers. Strangely enough, if you are not a Tier 1 (or hoping to be a Tier 1), peering sessions are usually "set up and forget". Networks who have 10s of gigabits of traffic but are not looking for reasons to deny peering requests see nearly no cost in these (especially compared to the overall cost of running the network). BLPAs are only required by people who think they mean something. Putting on MD5 is a bug/unique situation, which affect peering perhaps once every half-decade or so. Most small and mid-sized providers can handle the "NOC monitoring, trouble shooting, etc." with single-digit-hours a month, max. And sometimes that time is handled by people who are sitting on their ass waiting for something to break anyway. (Read "sunk cost".) So, unless you are looking for reasons to *not* peer, these are mostly BS.
- Administration
Think we covered this one.
- Public Switch costs
This is a cost and should be considered. Unless, of course, you are at TorIX, SIX, or any of the other very fine free NAPs available. Or if you can x-conn between your rack and someone else's rack in the same colo facility without going to a public switch. Or if you are in a FR or ATM cloud with other providers and can get uber-cheap PVCs between your routers with no additional hardware and a simple configuration change. Or.... I think you get the point. :)
It is difficult to defend peering strategies today unless your network is of a fairly significant size (gigabits of traffic) and you are collocated in an advantageous location(s). Otherwise, low cost transit is hard to beat.
I think you mean "or you are colocated in an advantageous location", not "and". If I am in 151 Front street, for a small one-time fee, I can connect to TorIX. The amount of the fee and the time it takes to set up peering is probably in the noise, even for a relatively small provider. Obviously if my entire traffic fits on a T1, things might be different, but I do not need anywhere near a gigabit of traffic to justify peering. You are probably at least an order of magnitude off. In general, Peering is a Good Thing [tm]. It increases performance, can lower costs, and might even increase your network reliability. But all the "other" things (e.g. performance benefits) are probably nice-to-have, not requirements. I'd look at the money. If you can break even or better, peering is probably a good idea. Most of the things analysts and Tier 1 providers talk about with peering costs (legal costs with contracts, managing sessions, NOC time, etc.) are mostly irrelevant, especially to anyone without a stock symbol and the related overhead of large corporations. (In many cases, they are irrelevant even to companies with those things.) So look at your real costs, and real savings: Hard costs - How much is the NAP connection? How much is the line to the NAP? How much for a router at the NAP? (Etc.) Then look at your benefits - Who will peer with me? How much traffic can I dump to them? (Etc.) Add up all the costs in the first set of questions (one time and recurring), subtract your transit cost times the amount of traffic you will save, and see if it is positive or negative. Don't forget to factor in things like any additional cost you incur by having less traffic to commit to a transit provider. (For instance, if you are using tiered pricing, will dumping traffic to peers bring you down to a lower tier, and therefore a higher $/Mbps?) If your monthly costs are lower with peering than transit alone, it is probably a good idea to peer and ignore the NOC costs. Everyone's situation is different, but don't put too much stock in things like the cost of a good BLPA. :) -- TTFN, patrick
NISCC Vulnerability Advisory 236929 Vulnerability Issues in TCP Version Information Advisory Reference 236929 Release Date 20 April 2004 Last Revision 20 April 2004 Version Number 1.0 What is Affected? The vulnerability described in this advisory affects implementations of the Transmission Control Protocol (TCP) that comply with the Internet Engineering Task Force�s (IETF�s) Requests For Comments (RFCs) for TCP, including RFC 793, the original specification, and RFC 1323, TCP Extensions for High Performance. TCP is a core network protocol used in the majority of networked computer systems today. Many vendors include support for this protocol in their products and may be impacted to varying degrees. Furthermore any network service or application that relies on a TCP connection will also be impacted, the severity depending primarily on the duration of the TCP session. Severity The impact of this vulnerability varies by vendor and application, but in some deployment scenarios it is rated critical. Please see the vendor section below for further information. Alternatively contact your vendor for product specific information. If exploited, the vulnerability could allow an attacker to create a Denial of Service condition against existing TCP connections, resulting in premature session termination. The resulting session termination will affect the application layer, the nature and severity of the effects being dependent on the application layer protocol. The primary dependency is on the duration of the TCP connection, with a further dependency on knowledge of the network (IP) addresses of the end points of the TCP connection. The Border Gateway Protocol (BGP) is judged to be potentially most affected by this vulnerability. BGP relies on a persistent TCP session between BGP peers. Resetting the connection can result in medium term unavailability due to the need to rebuild routing tables and route flapping. Route flapping may result in route dampening (suppression) if the route flaps occur frequently within a short time interval. The overall impact on BGP is likely to be moderate based on the likelihood of successful attack. If the TCP MD5 Signature Option and anti-spoofing measures are used then the impact will be low as these measures will successfully mitigate the vulnerability. There is a potential impact on other application protocols such as DNS (Domain Name System) and SSL (Secure Sockets Layer) in the case of zone transfers and ecommerce transactions respectively, but the duration of the sessions is relatively short and the sessions can be restarted without medium term unavailability problems. In the case of SSL it may be difficult to guess the source IP address. Data injection may be possible. However, this has not been demonstrated and appears to be problematic. Summary The issue described in this advisory is the practicability of resetting an established TCP connection by sending suitable TCP packets with the RST (Reset) or SYN (Synchronise) flags set. The packets need to have source and destination IP addresses that match the established connection as well as the same source and destination TCP ports. The fact that TCP sessions can be reset by sending suitable RST and SYN packets is a design feature of TCP according to RFC 793, but a reset attack is only possible at all because the source IP address and TCP port can be forged or �spoofed�. Although denial of service using crafted TCP packets is a well known weakness of TCP, until recently it was believed that a successful denial of service attack was not achievable in practice. The reason for this is that the receiving TCP implementation checks the sequence number of the RST or SYN packet, which is a 32 bit number, giving a probability of 1/232 of guessing the sequence number correctly (assuming a random distribution). The discoverer of the practicability of the RST attack was Paul A. Watson, who describes his research in his paper �Slipping In The Window: TCP Reset Attacks�, presented at the CanSecWest 2004 conference. He noticed that the probability of guessing an acceptable sequence number is much higher than 1/232 because the receiving TCP implementation will accept any sequence number in a certain range (or �window�) of the expected sequence number. The window makes TCP reset attacks practicable. Any application protocol which relies on long term TCP connections and for which the source and destination IP addresses and TCP ports are known or can be easily guessed will be vulnerable to at least denial of service attacks. Details TCP is the transport layer protocol designed to provide connection-oriented reliable delivery of IP packets. To do this TCP uses a mixture of flags, to indicate state, and sequence numbers, to identify the order in which the packets are to be reassembled. TCP also provides a number, called an acknowledgement number, that is used to indicate the sequence number of the next packet expected. The packets are reassembled by the receiving TCP implementation only if their sequence numbers fall within a range of the acknowledgement number (called a "window"). The acknowledgement number is not used in a RST packet because a reset does not expect a packet in return. (To be completely accurate, although the last statement is true for a RST packet without the ACK flag set, used to indicate that a TCP port is closed, a RST/ACK is used to terminate an active connection in the event of error. In a RST/ACK packet an acknowledgement number is included in the packet, although it is not checked by the receiving TCP implementation.) RFC 793, p36, states the following: "In all states except SYN-SENT, all reset (RST) segments are validated by checking their SEQ-fields [sequence numbers]. A reset is valid if its sequence number is in the window. In the SYN-SENT state (a RST received in response to an initial SYN), the RST is acceptable if the ACK field acknowledges the SYN." Resets must be processed immediately. RFC 793, p25, says "[�] [E]ven when the receive window is zero, a TCP must process the RST and URG fields of all incoming segments." It is also possible to perform the same attack with SYN (synchronise) packets. An established connection will abort by sending a RST if it receives a duplicate SYN packet with initial sequence number within the TCP window. RFC 793, p31 states: �The principle reason for the three-way handshake is to prevent old duplicate connection initiations from causing confusion. To deal with this, a special control message, reset, has been devised. [�] If the TCP is in one of the synchronized states (ESTABLISHED, FIN-WAIT-1, FIN-WAIT-2, CLOSE-WAIT, CLOSING, LAST-ACK, TIME-WAIT), it aborts the connection and informs its user.� TCP window sizes are negotiated in the initial 3-way handshake used to set up a TCP connection, with higher values serving to improve throughput in some circumstances. Vendor-chosen defaults also influence the selection. In any case, the larger the window size, the greater is the probability that a randomly chosen TCP sequence number will lie within the window range. This is the basis for the attack. A TCP connection is defined by a 4-tuple comprising source and destination IP addresses, and source and destination ports. An attacker seeking to disrupt an existing TCP connection must supply the 4-tuple correctly. As the source port varies, additional work is generally called for on the part of the attacker. However, research (referenced below) has shown that the process of source port selection on many platforms includes predictable elements, so that the attack remains practicable. By weighting 'likely' source port values carefully, an attacker can disrupt TCP implementations that employ a range of window sizes. Application layer protocols that are critically affected are those that: � Depend on long lived TCP connections � Have known or easy-to-guess IP address end points � Have easy to an easy-to-guess source TCP port As noted above BGP does use long lived TCP connections, and the IP addresses and source port (and destination port) are sometimes available through the use of BGP looking glasses (multi-source, multi-destination trace route tools) or DNS resource records. Using �trace route� commands can provide information on peering point IP addresses. Thus BGP is likely to be critically affected by the TCP vulnerability. These denial of service attacks can be carried out by single machine, or by multiple co-operating systems (to form a distributed denial of service attack). It is also possible to inject packets, which will be processed if they are in the window. The difficulty with data injection attacks is that the receiving TCP implementation will reassemble the packets received according to sequence number, dropping any duplicate packets. Vendor specific information will be released as it becomes available and if vendor permission has been received. Subscribers are advised to check the following URL regularly for updates: http://www.uniras.gov.uk/vuls/2004/236929/index.htm [Please note that updates to this advisory will not be notified by email.] This vulnerability has been assigned the CVE name CAN-2004-0230. The Open Source Vulnerability Database ID number for this vulnerability is 4030. Mitigation The following mitigation steps are still being evaluated and may be incomplete. Customers should work with vendors for the workaround most appropriate for the product in question. In the absence of vendor patching of the TCP implementation, the following are general mitigating steps: � Implement IP Security (IPSEC) which will encrypt traffic at the network layer, so TCP information will not be visible � Reduce the TCP window size (although this could increase traffic loss and subsequent retransmission) � Do not publish TCP source port information It should be noted that IPSEC provides confidentiality and authentication services at the network layer, and can provide a measure of trust in the authenticity of the end points as well as encryption of traffic between the end points. However, in the context of the current attack IPSEC will reject RST and SYN packets that are not part of a secure IP packet stream. To change the TCP window size, in some Unix variants you can set a value of the default TCP windows size by using the �sysctl� program (�ndd -set� in the case of Sun Solaris). In the case of Microsoft Windows NT/2000/XP/2003, the default window size can be changed by modifying the value of the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters key. As noted above, great care should be exercised when altering the default TCP window size as network performance could be adversely affected. In the case of BGP, the following may counter the problem: � Implement ingress and egress filtering to check that the traffic entering or leaving the network has a source IP address that is expected on the router/firewall interface that receives the traffic � Implement the TCP MD5 Signature Option to checksum the TCP packet carrying the BGP application data (see RFC 2385), being careful to set and maintain strong (i.e. difficult to guess) passwords to which the MD5 checksum is applied. Also see RFC 3562 which discusses the security requirements of this keying material. � Limit the amount of information available through looking glasses and DNS resource records, being careful not to expose TCP port information unnecessarily The IETF ingress filtering standard is defined in RFC 2827. A discussion of egress filtering can be found at http://www.sans.org/y2k/egress.htm. The use of the TCP MD5 Signature Option will prevent the exploitation of this vulnerability. Router customers should implement this on all BGP peering points if it is supported by the router, upgrading the router firmware if necessary. Solution Please refer to the Vendor Information section of this advisory for implementation specific remediation. Some vendors will have reduced the likelihood of successful denial of service by amending the TCP implementation to issue a further acknowledgment packet challenge for RST and SYN packets that do not have exactly the expected sequence number. The Internet Engineering Task Force (IETF) has published an Internet Draft to co-incide with the release of this advisory. The text of this draft is available from the IETF web site: http://www.ietf.org/internet-drafts/draft-ietf-tcpm-tcpsecure-00.txt NISCC has produced best practice guidelines for BGP available at http://www.niscc.gov.uk/BGP Filtering Guide.pdf Secure configuration templates for BGP implementations on Cisco IOS and Juniper JunOS can be found at: � Cisco http://www.cymru.com/Documents/secure-bgp-template.html � Juniper http://www.qorbit.net/documents/junos-bgp-template.pdf Guidance on tuning of the IP stack for a number of different UNIX operating systems is available at http://www.cymru.com/Documents/ip-stack-tuning.html Vendor Information The following vendors have provided information about how their products are affected by these vulnerabilities. Please note that JPCERT/CC have released a Japanese language advisory for this vulnerability which contains additional information regarding Japanese vendors. This advisory is available at http://www.jpcert.or.jp/at/2004/at040003.txt. Certicom Internet Initiative Japan, Inc Check Point InterNiche Cisco Juniper Networks Cray Inc NEC Hitachi Polycom Innovaphone Yamaha Certicom Certicom's SSL software developer toolkits (SDK), requires a transport mechanism, however it is not restricted to TCP. The default implementation that is shipped with the product utilizes the supported operating system's TCP/IP stack. Certicom recognizes that the indicated vulnerability is against the protocol stack itself and not directly the application on top. As our products (SSL Plus, SSL Plus for Java, Security Builder SSL-C, and Security Builder SSL-J), are primarily used in a web server environment, a denial of service attack is important to us and our customers. As there is no patch or workaround that Certicom can implement within our products, we feel that we are not directly vulnerable to this advisory. Certicom's website is www.certicom.com. Check Point The latest release for VPN-1/FireWall-1 (R55 HFA-03) contains a protection against this vulnerability. The protection applies to both the firewall device and to hosts behind the firewall. Please refer to the Check Point web site for further information at: http://www.checkpoint.com/techsupport/alerts/tcp_dos.html. Cisco Place holder. Cray Inc Cray Inc. is vulnerable on their UNICOS, UNICOS/mk and UNICOS/mp systems. Spr's have been opened to track this issue. Please contact your local Cray Service Representative for more information. Hitachi Hitachi is investigating the potential impact to Hitachi's products. Innovaphone Not vulnerable. Internet Initiative Japan, Inc (IIJ) IIJ will release a new firmware to fix this vulnerability. Details are available on their web site at http://www.seil.jp/en/ann/announce_en_20040421_01.txt. InterNiche === NicheStack v2.0 TCP/IP === InterNiche Technologies has updated its NicheStack v2.0 TCP/IP product to handle the scenarios described in NISCC Vulnerability Notice #236929. The patch is available to all InterNiche customers in accordance with the terms of their current support agreements. More information can be found on www.iNiche.com or through support@iNiche.com === NicheLite v2.0 TCP/IP === InterNiche Technologies has updated its NicheLite v2.0 TCP/IP product to handle the scenarios described in NISCC Vulnerability Notice #236929. The patch is available to all InterNiche customers in accordance with the terms of their current support agreements. More information can be found on www.iNiche.com or through support@iNiche.com Juniper Networks Juniper Networks products are susceptible to this vulnerability. Software is available that implements several mechanisms to mitigate the associated risks. Customers should contact Juniper Networks Technical Assistance Center for availability and download instructions. Additional information is posted on our web site at https://www.juniper.net/support. NEC NEC is aware of this vulnerability and is trying to determine potential impacts on our products. Polycom Polycom has investigated the potential impact to our products for NISCC Advisory 236929. Specific product information will be provided at http://www.polycom.com/securitycenter. Yamaha Pending. Acknowledgements NISCC wishes to thank the following: � Steve Bellovin, Rob Thomas and Paul Watson for their contributions to this advisory. � Cisco Systems Inc. and Juniper Networks Inc. for their help with the content of this advisory and for their support during the disclosure process. � JPCERT/CC for their assistance in co-ordinating this disclosure in Japan. Contact Information The NISCC Vulnerability Management Team can be contacted as follows: Email vulteam@niscc.gov.uk (Please quote the advisory reference in the subject line.) Telephone +44 (0)20 7821 1330 Extension 4511 (Monday to Friday 08:30 - 17:00) Fax +44 (0)20 7821 1686 Post Vulnerability Management Team NISCC PO Box 832 London SW1P 1BG We encourage those who wish to communicate via email to make use of our PGP key. This is available from http://www.uniras.gov.uk/UNIRAS.asc. Please note that UK government protectively marked material should not be sent to the email address above. If you wish to be added to our email distribution list, please email your request to uniras@niscc.gov.uk. What is NISCC? For further information regarding the UK National Infrastructure Security Co-Ordination Centre, please visit the NISCC web site at: http://www.niscc.gov.uk/aboutniscc/index.htm Reference to any specific commercial product, process or service by trade name, trademark manufacturer or otherwise, does not constitute or imply its endorsement, recommendation, or favouring by NISCC. The views and opinions of authors expressed within this notice shall not be used for advertising or product endorsement purposes. Neither shall NISCC accept responsibility for any errors or omissions contained within this advisory. In particular, they shall not be liable for any loss or damage whatsoever, arising from or in connection with the usage of information contained within this notice. � 2004 Crown Copyright Revision History April 20, 2004: Initial release (1.0) <End of NISCC Vulnerability Advisory> --------------------------------- Do you Yahoo!? Yahoo! Photos: High-quality 4x6 digital prints for 25�
On Tue, 20 Apr 2004, tad pedley wrote:
Although denial of service using crafted TCP packets is a well known weakness of TCP, until recently it was believed that a successful denial of service attack was not achievable in practice. The reason for this is that the receiving TCP implementation checks the sequence number of the RST or SYN packet, which is a 32 bit number, giving a probability of 1/232 of guessing the sequence number correctly (assuming a random distribution).
The discoverer of the practicability of the RST attack was Paul A. Watson, who describes his research in his paper Slipping In The Window: TCP Reset Attacks, presented at the CanSecWest 2004 conference. He noticed that the probability of guessing an acceptable sequence number is much higher than 1/232 because the receiving TCP implementation will accept any sequence number in a certain range (or window) of the expected sequence number. The window makes TCP reset attacks practicable.
Believed by whom, is the question. It has been clearly documented for a long time now that such larger windows exist. They have even been documented specifically about BGP (draft-ietf-idr-bgp-vuln-00.txt). -- Pekka Savola "You each name yourselves king, yet the Netcore Oy kingdom bleeds." Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings
On Tue, 20 Apr 2004, Patrick W.Gilmore wrote:
In many, many cases, especially for smaller providers, this is a spare FE on a switch which already exists.
I assume Vijay meant the cost of a port for private peering, in which case if you private with all your peers and you have a lot of small peers thats going to be a lot of cost for a few kbps of traffic
- Operational costs such as legal review for BLPAs, NOC monitoring, troubleshooting when it flaps, putting MD5 on, etc
These costs are frequently quoted as reasons not to peer by the larger providers.
BLPAs are only required by people who think they mean something.
Well theyre a good excuse thats for certain :) But I would say they do mean something.. if you're BigISP-A and you are peering with BigISP-B you want to make sure that continues reliably and that means a formal arrangement. Even if your a small ISP its worthwhile considering a formal arrangement particularly with the larger peers to make sure they dont ditch you without some good notice or that they will upgrade without cost if your traffic increases....
In general, Peering is a Good Thing [tm]. It increases performance, can lower costs, and might even increase your network reliability.
Hmm, we're fairly open on peering and have a bunch of small peers, in fact most of our new peerings are with small peers (small is something like announcing a single /24 and doing almost no traffic). We occasionally see performance problems with these small peers, where they maybe drop the session without warning raising an alarm here or do something screwy with their config and leak or whatever. They also tend to only have one connection, this forces how we route traffic to them, as we're in the process of expanding I really want to have multiple equal paths so that we can be sure the traffic is taking the best way to them. My summary of these points is that I'm seriously considering what our policy will be in the future and for good reason (altho it will undoubtedly continue to be fairly relaxed).
If your monthly costs are lower with peering than transit alone, it is probably a good idea to peer and ignore the NOC costs.
In some instances I'm willing to pay more for a connection (eg paid peering or costs of backbone circuits) to ensure I'm receiving quality. There are a couple other issues not raised... One is the cost on the router in terms of memory and cpu of maintaining such a large number of sessions (usually less of an issue with your big multiprocessor routers) The other is our new hot topic of security, not sure if anyone has thought of this yet (or how interesting it is) but the nature of the bgp attack means that if you can view a BGP session you can figure things about a peer that would otherwise be hidden from you in particular the port numbers in use.. and I'm not entirely clear on the details but it sounds like when you hit the first session, you can take the rest out very easily. We cant take BGP out of band (yet!), perhaps we can keep it better hidden from view tho.. Steve
On Apr 20, 2004, at 2:15 PM, Stephen J. Wilcox wrote:
On Tue, 20 Apr 2004, Patrick W.Gilmore wrote:
In many, many cases, especially for smaller providers, this is a spare FE on a switch which already exists.
I assume Vijay meant the cost of a port for private peering, in which case if you private with all your peers and you have a lot of small peers thats going to be a lot of cost for a few kbps of traffic
It was Dan, not Vijay. And clearly we are not talking about running a pair of fiber to everyone who has a modem's worth of traffic. He mentioned the cost of the port. I said many people have spare FEs / GEs on existing switches. And if they do not, a few hundred dollars will get them one.
- Operational costs such as legal review for BLPAs, NOC monitoring, troubleshooting when it flaps, putting MD5 on, etc
These costs are frequently quoted as reasons not to peer by the larger providers.
BLPAs are only required by people who think they mean something.
Well theyre a good excuse thats for certain :) But I would say they do mean something.. if you're BigISP-A and you are peering with BigISP-B you want to make sure that continues reliably and that means a formal arrangement. Even if your a small ISP its worthwhile considering a formal arrangement particularly with the larger peers to make sure they dont ditch you without some good notice or that they will upgrade without cost if your traffic increases....
I specifically left out BigISP-*. The complexities of peering on a Tier 1 network are not really describable in a single e-mail. As for the smaller ISPs, read every peering agreement you've signed. They all say they can cancel with at most 30 days notice, for no reason, with no recourse, and nothing you can do about it. Furthermore, many include the ability to shut down peering if they even *think* you are doing something funny, and again you have no recourse. Peering agreements are not worth anything to keep peering up. They are only worth something if you are worried about the peer doing something like pointing default.
In general, Peering is a Good Thing [tm]. It increases performance, can lower costs, and might even increase your network reliability.
Hmm, we're fairly open on peering and have a bunch of small peers, in fact most of our new peerings are with small peers (small is something like announcing a single /24 and doing almost no traffic).
The second number there is important, the first is not. There are peers which announce a /24 or few and have gigabits of traffic.
We occasionally see performance problems with these small peers, where they maybe drop the session without warning raising an alarm here or do something screwy with their config and leak or whatever.
Nowhere was I saying it is a good idea to peer with someone who hurts your network. But most of the peers, even the small ones, can keep their network stable.
They also tend to only have one connection, this forces how we route traffic to them, as we're in the process of expanding I really want to have multiple equal paths so that we can be sure the traffic is taking the best way to them.
Perfectly valid concern. Which is why I specifically told people to find out who would peer with them before paying to go to a peering point. Don't count your chickens until they're hatched and all that. :)
My summary of these points is that I'm seriously considering what our policy will be in the future and for good reason (altho it will undoubtedly continue to be fairly relaxed).
And I see nothing you mentioned which in any way goes against what I was saying. Your particular situation is very different than the next networks, as the next networks is unique to that network, etc. But that doesn't make peering bad.
If your monthly costs are lower with peering than transit alone, it is probably a good idea to peer and ignore the NOC costs.
In some instances I'm willing to pay more for a connection (eg paid peering or costs of backbone circuits) to ensure I'm receiving quality.
It is nice to ensure quality. But if quality is your primary goal, then directly peering with a network will give you better "quality" from an end user (read "paying customer") PoV than transit in most cases. Extra latency is usually not viewed as better quality. If you are worried about the connection being flaky, well, like I said, don't peer with flaky networks. Besides, most small to medium guys have enough headroom on their transit connections to take down many of their peers and push it over transit without congestion.
There are a couple other issues not raised...
One is the cost on the router in terms of memory and cpu of maintaining such a large number of sessions (usually less of an issue with your big multiprocessor routers)
Agreed. But since we are not talking to the one-T1-ISP (which I also said would not fit the model), people probably have enough CPU to handle a few extra BGP sessions. If not, well, another cost to consider before peering.
The other is our new hot topic of security, not sure if anyone has thought of this yet (or how interesting it is) but the nature of the bgp attack means that if you can view a BGP session you can figure things about a peer that would otherwise be hidden from you in particular the port numbers in use.. and I'm not entirely clear on the details but it sounds like when you hit the first session, you can take the rest out very easily.
Riiiiiiiiiiiiiiiiiiiight.
We cant take BGP out of band (yet!), perhaps we can keep it better hidden from view tho..
Good idea. Get right on that, would you? :) -- TTFN, patrick
The other is our new hot topic of security, not sure if anyone has thought of this yet (or how interesting it is) but the nature of the bgp attack means that if you can view a BGP session you can figure things about a peer that would otherwise be hidden from you in particular the port numbers in use.. and I'm not entirely clear on the details but it sounds like when you hit the first session, you can take the rest out very easily.
We cant take BGP out of band (yet!), perhaps we can keep it better hidden from view tho..
There are more protection methods available than just MD5 (as you allude to Steve). One mitigator is to use "non-routed" space for BGP peer connections. If you have the ability to filter on TTL 255 you are in even better shape (arguably perfectly secure against all but configuration/hardware failures). You have some vulnerability with non-routed space if you do default routing or have folks who default towards the device doing the BGP peering though. Source routing is also a potential hazard for the non-routed solution (does anyone have this enabled anymore?). Apologies for the morph but this raised a great point. Regards, Blaine
On Tue, 20 Apr 2004, Blaine Christian wrote:
The other is our new hot topic of security, not sure if anyone has thought of this yet (or how interesting it is) but the nature of the bgp attack means that if you can view a BGP session you can figure things about a peer that would otherwise be hidden from you in particular the port numbers in use.. and I'm not entirely clear on the details but it sounds like when you hit the first session, you can take the rest out very easily.
We cant take BGP out of band (yet!), perhaps we can keep it better hidden from view tho..
There are more protection methods available than just MD5 (as you allude to Steve). One mitigator is to use "non-routed" space for BGP peer connections. If you have the ability to filter on TTL 255 you are in even better shape (arguably perfectly secure against all but configuration/hardware failures). You have some vulnerability with non-routed space if you do default routing or have folks who default towards the device doing the BGP peering though. Source routing is also a potential hazard for the non-routed solution (does anyone have this enabled anymore?).
Apologies for the morph but this raised a great point.
Hmm ok so assume for a moment that I dont want RFC1918 for my links, what are my options? : There isnt a "link-local" for IP altho this would be a great solution (surely this can be written for BGP??). Or I could use all eBGP addresses from a block which I dont route and filter internally.. I suspect this is a non-starter, I will have to include all my addresses given to me by peers and its gonna screw traces, monitoring etc. Can I use secondary IP addresses and then BGP with these addresses, this would be a form of "security by obscurity" but providing you can keep the info a secret thats surely going to do it? Steve
* steve@telecomplete.co.uk (Stephen J. Wilcox) [Thu 22 Apr 2004, 16:11 CEST]:
There isnt a "link-local" for IP altho this would be a great solution (surely this can be written for BGP??).
169.254/16
Or I could use all eBGP addresses from a block which I dont route and filter internally.. I suspect this is a non-starter, I will have to include all my addresses given to me by peers and its gonna screw traces, monitoring etc.
I've seen e.g. BT do this -- Niels.
On Thu, 22 Apr 2004, Niels Bakker wrote:
* steve@telecomplete.co.uk (Stephen J. Wilcox) [Thu 22 Apr 2004, 16:11 CEST]:
There isnt a "link-local" for IP altho this would be a great solution (surely this can be written for BGP??).
169.254/16
is not link-local and will be routed if you try. i was thinking link-local as in ISIS style
Or I could use all eBGP addresses from a block which I dont route and filter internally.. I suspect this is a non-starter, I will have to include all my addresses given to me by peers and its gonna screw traces, monitoring etc.
I've seen e.g. BT do this
and public peering? or when viewed from their peers networks..? Steve
-- Niels.
Can I use secondary IP addresses and then BGP with these addresses, this would be a form of "security by obscurity" but providing you can keep the info a secret thats surely going to do it?
It will depend on your architecture in large part. In some cases there is absolutely no need to route the prefixes that you use for your BGP sessions beyond the devices doing BGP. This can reduce your exposure to MD5 related cpu churn etc...
On Thu, 22 Apr 2004, Blaine Christian wrote:
Can I use secondary IP addresses and then BGP with these addresses, this would be a form of "security by obscurity" but providing you can keep the info a secret thats surely going to do it?
It will depend on your architecture in large part. In some cases there is absolutely no need to route the prefixes that you use for your BGP sessions beyond the devices doing BGP. This can reduce your exposure to MD5 related cpu churn etc...
Yes, but (1) its difficult and (2) as these are external sessions I need to ensure my peers are doing the same, as the chances are they wont and the chances are the attack comes in externally then I'm still at risk Steve
On 22-apr-04, at 16:11, Stephen J. Wilcox wrote:
There are more protection methods available than just MD5 (as you allude to Steve). One mitigator is to use "non-routed" space for BGP peer connections.
Hmm ok so assume for a moment that I dont want RFC1918 for my links, what are my options? :
There isnt a "link-local" for IP altho this would be a great solution (surely this can be written for BGP??).
Who says BGP sessions must run over IP(v4)? In theory it shouldn't be a problem to exchange IPv4 routing information over IPv6 BGP TCP sessions. (But it seems some of our favorite vendors didn't add this scenario to their regression tests.)
Or I could use all eBGP addresses from a block which I dont route and filter internally.. I suspect this is a non-starter, I will have to include all my addresses given to me by peers and its gonna screw traces, monitoring etc.
Can I use secondary IP addresses and then BGP with these addresses, this would be a form of "security by obscurity" but providing you can keep the info a secret thats surely going to do it?
If you combine the two approaches above and filter all traffic to the primary address, traceroutes et al still work but people from the outside don't get to hit the route processor.
IvB> Date: Thu, 22 Apr 2004 18:03:33 +0200 IvB> From: Iljitsch van Beijnum IvB> Who says BGP sessions must run over IP(v4)? NetBEUI, anyone? No bickering over RFC1918 on WAN links... ;-) Eddy -- EverQuick Internet - http://www.everquick.net/ A division of Brotsman & Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita _________________________________________________________________ DO NOT send mail to the following addresses : blacklist@brics.com -or- alfra@intc.net -or- curbjmp@intc.net Sending mail to spambait addresses is a great way to get blocked.
Hmm. Interesting. I am (here is SFO area) DSL customer and DialUp customer. But I never received a notification from my provider(s), possible with free CD, explaining me (if I am a homewife, not an engineer, of course) what to do and how to prevent a problems. We have a lot of room for improvement. In includes not only viiruses, but wireless (when I work in my friend's home, I use his neighbour WiFi, because it have a bettr quality, withouut /of course/ his even knowing it -:) - or, better to say, his notebook prefer his neighbour, for some reasons), and so on.
On Apr 20, 2004, at 10:32 AM, Daniel Golding wrote:
On 4/20/04 1:34 AM, "Michel Py" <michel@arneill-py.sacramento.ca.us> wrote:
Patrick W.Gilmore wrote: Unless they have cheap access to a free NAP (TorIX, SIX, etc.), transit, even at higher prices, is probably be the best / cheapest way to reach the Internet.
This is true, but there are plenty of other opportunities for peering, such as: both parties buy DS-3 class transit from the same tier-2 or even maybe tier-3 provider in a colo (which will likely be a BFM, other problem) not a formal IX. In other words, peering in an IX does cost money, but peering at a colo might not, as these messy colos are mostly unmanaged and nobody cares about that 25ft cross-over cable :-)
Michel.
This is a classical mistake. Peering always costs money and its never free.
Maybe "Free" is the wrong word. Perhaps "No additional cost over <transit/whatever>".
Or, for those of us who think that the time it takes to plug a patch cable into an unused switch port and do some configuration changes are irrelevant, maybe "free" is the right word.
Either way, it is not NEARLY as bad as you or many other people make it out to be. Allow me to explain....
The question is, how much, and is it cheaper than transit?
Costs incurred in peering:
- Port Costs (capex)
Pthhhhhhhhh. In many, many cases, especially for smaller providers, this is a spare FE on a switch which already exists.
For mid-sized providers, it is frequently a spare GE port on an existing switch, which means perhaps $500 for GBIC or something.
For large providers, there is a cost here. But large providers are a different beast, and there is no way a simple e-mail could possibly capture the complexities implicit in peering between Very Large Providers. So we'll let them figure out their own costs.
- A share of a router's backplane capacity corresponding to the port
Irrelevant. The traffic has to go somewhere, if it does not go out the peering port, it will go out transit, but it is definitely going across the router's backplane.
A better thing to put here would be possible use of a router which would not be used. Specifically, if I get a bit in a POP which has transit, I do not have to use the router out at the Peering Point. But how many people have router backpanes which are saturated? At worst you are running out of slots for ports in most cases. (Remember, we left the really big providers to their own devices.)
- Cross connect costs (one time or recurring)
Largely irrelevant - if you are really going to go out-of-biz for a $150/Month x-conn, you have bigger problems.
- Operational costs such as legal review for BLPAs, NOC monitoring, troubleshooting when it flaps, putting MD5 on, etc
These costs are frequently quoted as reasons not to peer by the larger providers. Strangely enough, if you are not a Tier 1 (or hoping to be a Tier 1), peering sessions are usually "set up and forget". Networks who have 10s of gigabits of traffic but are not looking for reasons to deny peering requests see nearly no cost in these (especially compared to the overall cost of running the network).
BLPAs are only required by people who think they mean something. Putting on MD5 is a bug/unique situation, which affect peering perhaps once every half-decade or so. Most small and mid-sized providers can handle the "NOC monitoring, trouble shooting, etc." with single-digit-hours a month, max. And sometimes that time is handled by people who are sitting on their ass waiting for something to break anyway. (Read "sunk cost".)
So, unless you are looking for reasons to *not* peer, these are mostly BS.
- Administration
Think we covered this one.
- Public Switch costs
This is a cost and should be considered. Unless, of course, you are at TorIX, SIX, or any of the other very fine free NAPs available. Or if you can x-conn between your rack and someone else's rack in the same colo facility without going to a public switch. Or if you are in a FR or ATM cloud with other providers and can get uber-cheap PVCs between your routers with no additional hardware and a simple configuration change. Or....
I think you get the point. :)
It is difficult to defend peering strategies today unless your network is of a fairly significant size (gigabits of traffic) and you are collocated in an advantageous location(s). Otherwise, low cost transit is hard to beat.
I think you mean "or you are colocated in an advantageous location", not "and". If I am in 151 Front street, for a small one-time fee, I can connect to TorIX. The amount of the fee and the time it takes to set up peering is probably in the noise, even for a relatively small provider.
Obviously if my entire traffic fits on a T1, things might be different, but I do not need anywhere near a gigabit of traffic to justify peering. You are probably at least an order of magnitude off.
In general, Peering is a Good Thing [tm]. It increases performance, can lower costs, and might even increase your network reliability. But all the "other" things (e.g. performance benefits) are probably nice-to-have, not requirements. I'd look at the money.
If you can break even or better, peering is probably a good idea. Most of the things analysts and Tier 1 providers talk about with peering costs (legal costs with contracts, managing sessions, NOC time, etc.) are mostly irrelevant, especially to anyone without a stock symbol and the related overhead of large corporations. (In many cases, they are irrelevant even to companies with those things.) So look at your real costs, and real savings:
Hard costs - How much is the NAP connection? How much is the line to the NAP? How much for a router at the NAP? (Etc.)
Then look at your benefits - Who will peer with me? How much traffic can I dump to them? (Etc.)
Add up all the costs in the first set of questions (one time and recurring), subtract your transit cost times the amount of traffic you will save, and see if it is positive or negative. Don't forget to factor in things like any additional cost you incur by having less traffic to commit to a transit provider. (For instance, if you are using tiered pricing, will dumping traffic to peers bring you down to a lower tier, and therefore a higher $/Mbps?)
If your monthly costs are lower with peering than transit alone, it is probably a good idea to peer and ignore the NOC costs. Everyone's situation is different, but don't put too much stock in things like the cost of a good BLPA. :)
-- TTFN, patrick
participants (11)
-
Alexei Roudnev -
Blaine Christian -
Daniel Golding -
E.B. Dreger -
Iljitsch van Beijnum -
Michel Py -
Niels Bakker -
Patrick W.Gilmore -
Pekka Savola -
Stephen J. Wilcox -
tad pedley