Problems with newish IP block assignment issues from ARIN
Folks, Have a gremlin we have been chasing around for several months now and it’s becoming a major issue as we are getting tighter on IPV4 and needing to give some provider assigned space back. In June we received a /22 from ARIN. As is my workflow I started announcing it but waited a month while I checked out the geolocation databases for correct info, did testing ,etc. All this time our test accounts could browse web-sites, etc. We put one of the pools into production and things ran good for awhile. Then we started getting the occasional web-site was not working. After several of these we started assigning the customer an IP out of one of our other ARIN blocks and the web-site would be fine and reachable. The issue seems to reside just on this /22. We have other blocks from ARIN and they are just fine. We can assign an IP out of this new block and can’t reach certain web-sites. We turn around and assign out of another block and web-site works just fine. We have two upstreams and an IX on this network. We have tried withdrawing the route on this particular /22 and isolating to one upstream alone and the problems still persist. Many of the web-sites in question are government (both state and local), online universities, and the occasional local news station. They are diverse enough to not be traced down to a common point, except the IP block. We announce the IP block via BGP the same exact way we announce the other blocks. Traceroutes show the path going the same way no matter what IP block the customer has. It acts like the IP block was blacklisted at some point and got on some bad lists but I don’t want ti limit myself to that theory. I have opened up a ticket with ARIN asking for any guidance. Has anyone ran into this with new space assigned? Any tools, sites, etc. I can use to do further troubleshooting. The IP block does not appear to have any blacklisted IPs according to MX toolbox, and some others. The block in question is 134.195.44.0/22. It has been RPKI certified and has IRR entries. Thanks in advance Justin Wilson j2sw@mtin.net — https://j2sw.com - All things jsw (AS209109) https://blog.j2sw.com - Podcast and Blog
One common cause of this issue is entities out there that have very old 'bogons' filters in place for the larger block, as an entire /8, /12 to /16 size of space that, many years ago, was unallocated space. Without getting the end point organizations running the httpd, firewalls or whatever to fix their broken configuration, it's a hard issue to fix from your end. On a longer term time scale like multiple years, the reachability of an IP block like yours will gradually increase as people with broken services are contacted by additional persons to say "hey, this really is valid ARIN IP space". On Mon, Feb 8, 2021 at 12:15 PM Justin Wilson (Lists) <lists@mtin.net> wrote:
Folks, Have a gremlin we have been chasing around for several months now and it’s becoming a major issue as we are getting tighter on IPV4 and needing to give some provider assigned space back.
In June we received a /22 from ARIN. As is my workflow I started announcing it but waited a month while I checked out the geolocation databases for correct info, did testing ,etc. All this time our test accounts could browse web-sites, etc.
We put one of the pools into production and things ran good for awhile. Then we started getting the occasional web-site was not working. After several of these we started assigning the customer an IP out of one of our other ARIN blocks and the web-site would be fine and reachable. The issue seems to reside just on this /22. We have other blocks from ARIN and they are just fine. We can assign an IP out of this new block and can’t reach certain web-sites. We turn around and assign out of another block and web-site works just fine.
We have two upstreams and an IX on this network. We have tried withdrawing the route on this particular /22 and isolating to one upstream alone and the problems still persist.
Many of the web-sites in question are government (both state and local), online universities, and the occasional local news station. They are diverse enough to not be traced down to a common point, except the IP block.
We announce the IP block via BGP the same exact way we announce the other blocks. Traceroutes show the path going the same way no matter what IP block the customer has.
It acts like the IP block was blacklisted at some point and got on some bad lists but I don’t want ti limit myself to that theory. I have opened up a ticket with ARIN asking for any guidance. Has anyone ran into this with new space assigned? Any tools, sites, etc. I can use to do further troubleshooting. The IP block does not appear to have any blacklisted IPs according to MX toolbox, and some others.
The block in question is 134.195.44.0/22. It has been RPKI certified and has IRR entries.
Thanks in advance
Justin Wilson j2sw@mtin.net
— https://j2sw.com - All things jsw (AS209109) https://blog.j2sw.com - Podcast and Blog
Dear Justin, On Mon, Feb 08, 2021 at 03:14:47PM -0500, Justin Wilson (Lists) wrote:
It acts like the IP block was blacklisted at some point and got on some bad lists but I don’t want ti limit myself to that theory. I have opened up a ticket with ARIN asking for any guidance. Has anyone ran into this with new space assigned? Any tools, sites, etc. I can use to do further troubleshooting.
Here are some useful tools: ping.pe example: http://ping.pe/www.openbsd.org https://ring.nlnog.net/ good introduction here: https://labs.ripe.net/Members/martin_pels_3/10-years-of-nlnog-ring https://atlas.ripe.net/
The block in question is 134.195.44.0/22.
Is there any specific IP address in the range that should always respond to ICMP Echo Requests? This will help others see if they can reach you or not.
It has been RPKI certified and has IRR entries.
Indeed, nice :-) http://irrexplorer.nlnog.net/search/134.195.44.0/22 Kind regards, Job
I enabled 134.195.47.1 on one of our routers. Justin Wilson j2sw@mtin.net — https://j2sw.com - All things jsw (AS209109) https://blog.j2sw.com - Podcast and Blog
On Feb 8, 2021, at 3:46 PM, Job Snijders via NANOG <nanog@nanog.org> wrote:
Dear Justin,
On Mon, Feb 08, 2021 at 03:14:47PM -0500, Justin Wilson (Lists) wrote:
It acts like the IP block was blacklisted at some point and got on some bad lists but I don’t want ti limit myself to that theory. I have opened up a ticket with ARIN asking for any guidance. Has anyone ran into this with new space assigned? Any tools, sites, etc. I can use to do further troubleshooting.
Here are some useful tools:
ping.pe example: http://ping.pe/www.openbsd.org
https://ring.nlnog.net/ good introduction here: https://labs.ripe.net/Members/martin_pels_3/10-years-of-nlnog-ring
The block in question is 134.195.44.0/22.
Is there any specific IP address in the range that should always respond to ICMP Echo Requests? This will help others see if they can reach you or not.
It has been RPKI certified and has IRR entries.
Indeed, nice :-) http://irrexplorer.nlnog.net/search/134.195.44.0/22
Kind regards,
Job
On Mon, Feb 08, 2021 at 04:02:14PM -0500, Justin Wilson (Lists) wrote:
I enabled 134.195.47.1 on one of our routers.
Cool! I noticed the following: from many NLNOG RING nodes I can reach that IP address, but not from 195.66.134.42: deepmedia01.ring.nlnog.net:~$ mtr -z -w -r 134.195.47.1 Start: 2021-02-08T21:19:32+0000 HOST: deepmedia01.ring.nlnog.net Loss% Snt Last Avg Best Wrst StDev 1. AS39022 vlan100.ccr-1.gs.as39022.net 0.0% 10 0.5 0.5 0.4 0.5 0.1 2. AS??? speed-ix.he.net 0.0% 10 0.8 1.0 0.7 2.5 0.5 3. AS6939 100ge16-1.core1.lon2.he.net 0.0% 10 6.8 7.0 6.7 8.1 0.5 4. AS6939 100ge4-1.core1.nyc4.he.net 0.0% 10 83.7 77.7 72.5 93.8 8.4 5. AS6939 ve951.core2.nyc4.he.net 0.0% 10 73.0 73.0 72.6 74.9 0.7 6. AS6939 100ge0-31.core2.cmh1.he.net 0.0% 10 85.7 86.4 85.6 88.7 1.1 7. AS6939 100ge9-2.core1.ind1.he.net 0.0% 10 93.4 93.4 93.2 94.6 0.4 8. AS6939 184.105.30.134 0.0% 10 93.0 93.1 92.9 93.3 0.1 9. AS??? ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 Do you have a BGP route for 195.66.134.0/23 on the router with 134.195.47.1 ? Do you have a traceroute towards 195.66.134.42? Kind regards, Job
Off topic, but curious as to how you were able to procure new ip space? -----Original Message----- From: NANOG <nanog-bounces+jamann=mt.gov@nanog.org> On Behalf Of Justin Wilson (Lists) Sent: Monday, February 8, 2021 2:02 PM To: nanog@nanog.org Subject: [EXTERNAL] Re: Problems with newish IP block assignment issues from ARIN I enabled https://urldefense.com/v3/__http://134.195.47.1__;!!GaaboA!-uRygnjQP3TTzUH4T... on one of our routers. Justin Wilson j2sw@mtin.net — https://urldefense.com/v3/__https://j2sw.com__;!!GaaboA!-uRygnjQP3TTzUH4TizK... - All things jsw (AS209109) https://urldefense.com/v3/__https://blog.j2sw.com__;!!GaaboA!-uRygnjQP3TTzUH... - Podcast and Blog
On Feb 8, 2021, at 3:46 PM, Job Snijders via NANOG <nanog@nanog.org> wrote:
Dear Justin,
On Mon, Feb 08, 2021 at 03:14:47PM -0500, Justin Wilson (Lists) wrote:
It acts like the IP block was blacklisted at some point and got on some bad lists but I don’t want ti limit myself to that theory. I have opened up a ticket with ARIN asking for any guidance. Has anyone ran into this with new space assigned? Any tools, sites, etc. I can use to do further troubleshooting.
Here are some useful tools:
https://urldefense.com/v3/__http://ping.pe__;!!GaaboA!-uRygnjQP3TTzUH4TizKHK... example: https://urldefense.com/v3/__http://ping.pe/www.openbsd.org__;!!GaaboA! -uRygnjQP3TTzUH4TizKHK6l272nDUArLbdXQ_Gd9FGKInm0ECKk7PEDcrEwnA$
https://urldefense.com/v3/__https://ring.nlnog.net/__;!!GaaboA!-uRygnjQP3TTz... good introduction here: https://urldefense.com/v3/__https://labs.ripe.net/Members/martin_pels_ 3/10-years-of-nlnog-ring__;!!GaaboA!-uRygnjQP3TTzUH4TizKHK6l272nDUArLb dXQ_Gd9FGKInm0ECKk7PEoTe2KMw$
https://urldefense.com/v3/__https://atlas.ripe.net/__;!!GaaboA!-uRygnj QP3TTzUH4TizKHK6l272nDUArLbdXQ_Gd9FGKInm0ECKk7PHQK9vMaQ$
The block in question is https://urldefense.com/v3/__http://134.195.44.0/22__;!!GaaboA!-uRygnjQP3TTzU... .
Is there any specific IP address in the range that should always respond to ICMP Echo Requests? This will help others see if they can reach you or not.
It has been RPKI certified and has IRR entries.
Indeed, nice :-) https://urldefense.com/v3/__http://irrexplorer.nlnog.net/search/134.19 5.44.0/22__;!!GaaboA!-uRygnjQP3TTzUH4TizKHK6l272nDUArLbdXQ_Gd9FGKInm0E CKk7PG1xz7OuQ$
Kind regards,
Job
On Mon, 8 Feb 2021, Justin Wilson (Lists) wrote:
Folks,
Have a gremlin we have been chasing around for several months now and it’s becoming a major issue as we are getting tighter on IPV4 and needing to give some provider assigned space back.
In June we received a /22 from ARIN. As is my workflow I started announcing it but waited a month while I checked out the geolocation databases for correct info, did testing ,etc. All this time our test accounts could browse web-sites, etc.
We put one of the pools into production and things ran good for awhile. Then we started getting the occasional web-site was not working. After several of these we started assigning the customer an IP out of one of our other ARIN blocks and the web-site would be fine and reachable. The issue seems to reside just on this /22. We have other blocks from ARIN and they are just fine. We can assign an IP out of this new block and can’t reach certain web-sites. We turn around and assign out of another block and web-site works just fine.
Been there, and done that back in 2003. https://web.archive.org/web/20030722022858/http://69box.atlantic.net/ https://web.archive.org/web/20060214055930/http://not69box.atlantic.net/ Unfortunately, I've moved on from that job and don't have any of the code that I developed for not69box/69box (and AFAIK, the box itself is long gone), but you can get an idea from the above page what I did. i.e. The two names resolved to an IP in 69.28.64/19 or an IP in 209.208/17. One of the cooler (at least at the time) features was a dual-frame traceroute that visitors could run and watch the box traceroute to a destination from a each of it's IP's, thus showing where in the path their traceroute broke, if it did, from the "69 space". ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route StackPath, Sr. Neteng | therefore you are _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
Hi, On 2/8/21 10:22 PM, Hank Nussbacher wrote:
On 08/02/2021 22:14, Justin Wilson (Lists) wrote:
It acts like the IP block was blacklisted at some point and got on some bad lists but I don’t want ti limit myself to that theory. I have opened up a ticket with ARIN asking for any guidance. Has anyone ran into this with new space assigned? Any tools, sites, etc. I can use to do further troubleshooting. The IP block does not appear to have any blacklisted IPs according to MX toolbox, and some others.
Try:
I check IP blocks in all known blacklists as part of the process of listing IPs for sale on V4Escrow's IP brokerage platform. Here's a free blacklist report for your /22, block is clean as far as we can see ;) http://iplist.v4escrow.net/report/602232ff1adbbd648e7ad604 cheers, elvis
-Hank
The block in question is 134.195.44.0/22. It has been RPKI certified and has IRR entries.
Thanks in advance
Justin Wilson j2sw@mtin.net
— https://j2sw.com - All things jsw (AS209109) https://blog.j2sw.com - Podcast and Blog
participants (8)
-
Elvis Daniel Velea
-
Eric Kuhnke
-
Hank Nussbacher
-
Jesse DuPont
-
Job Snijders
-
Jon Lewis
-
Justin Wilson (Lists)
-
Mann, Jason