- Test - lists.nanog.org

RE: Crawler Ettiquette
by O'Neil,Kevin 24 Jan '02

24 Jan '02

We have a research project at OCLC that does web crawling. We created an email account (netsecurity(a)oclc.org) that admins could contact and posted the email address at the web site from which the crawling was done. That way when admins see our IP address in their firewall logs (from attempted http connects), they could browse to that address and see an explanation of the project and send email to the contact address to get more info or request removal from the IP space being crawled. See http://132.174.5.40 for our set up. ...Kevin O'Neil -----Original Message----- From: Deepak Jain [mailto:deepak@ai.net] Sent: Wednesday, January 23, 2002 2:35 PM To: Nanog@Merit. Edu Subject: Crawler Ettiquette I figured this was the best forum to post this, if anyone has suggestions where this might be better placed, please let me know. A University in our customer base has received funding to start a reasonably large spider project. It will crawl websites [search engine fashion] and save certain parts of the information it receives. This information will be made available to research institutions and other concerns. We have been asked for recommendations on what functions/procedures they can put in to be good netizens and not cause undo stress to networks out there. On the list of functions: a) Obey robots.txt files b) Allow network admins to automatically have their netblocks exempted on request c) Allow ISP's caches to sync with it. There are others, but they all revolve around a & b. C was something that seemed like a good idea, but I don't know if there is any real demand for it. Essentially, this project will have at least 1Gb/s of inbound bandwidth. Average usage is expected to be around 500mb/s for the first several months. ISPs who cache would have an advantage if they used the cache developed by this project to load their tables, but I do not know if there is an internet-wide WCCP or equivalent out there or if the improvement is worth the management overhead. Because the funding is there, this project is essentially a certainty. If there are suggestions that should be added or concerns that this raises, please let me know [privately is fine]. All input is appreciated, DJ

1 0

FreeIP database program
by Rishi Singh 24 Jan '02

24 Jan '02

A few months ago, I saw a posting on NANOG for a database program that manages IP addresses, that runs off a linux box. Does anyone remember the name of that program? Searches on Google with keywords, "free IP database" doesn't return anything useful. Thanks

3 2

Opinions on Time Warner
by Christopher J. Wolff 24 Jan '02

24 Jan '02

Dear List; I've heard very little about Time Warner's peering and general quality of connectivity at the DS3 and above levels. Does anyone have any experiences about their overall quality and customer service? Thank you in advance. Regards, Christopher J. Wolff, VP CIO Broadband Laboratories http://www.bblabs.com

1 0

Re: Anyone here from Microsoft? Hacked?
by David Diaz 24 Jan '02

24 Jan '02

I stand corrected. This one should be in a brain teaser book. dd At 14:33 -0500 1/24/02, Reid Fishler wrote: It would help if this URL was on MS...Look closely at it. Reid At 02:28 PM 1/24/2002 -0500, David Diaz wrote: If anyone from Microsoft is on the list. You might want to check to see if this is a hack or an old employee being funny. I did enjoy the link that sent me to McDonalds to apply for a job. Just curious, does anyone know what servers they run their content on? Is it still NT? Hope this info helps the webmaster. David http://www.microsoft.com&item%3Dq209354@hardware.no/nyheter/feb01/Q209354%2… -- David Diaz dave(a)smoton.net Smartphotons (smotons) trump dumb photons

1 0

Re: Anyone here from Microsoft? Hacked?
by Scott Gifford 24 Jan '02

24 Jan '02

David Diaz <techlist(a)smoton.net> writes: > If anyone from Microsoft is on the list. You might want to check to > see if this is a hack or an old employee being funny. > > I did enjoy the link that sent me to McDonalds to apply for a job. > > Just curious, does anyone know what servers they run their content on? > Is it still NT? > > Hope this info helps the webmaster. > > David > > http://www.microsoft.com&item%3Dq209354@hardware.no/nyheter/feb01/Q209354%2… This address is actually on hardware.no. See the @ sign? It instructs the browser to connect to 'hardware.no' and give the username 'www.microsoft.com&item%3Dq209354'. It's an increasingly common trick. So no security breaches, no pissed off employees, no chances to bash NT. Ah, well. ----ScottG.

1 0

RE: Anyone here from Microsoft? Hacked?
by Adam Herscher 24 Jan '02

24 Jan '02

You are requesting a page from hardware.no, not microsoft.com. -----Original Message----- From: David Diaz [mailto:techlist@smoton.net] Sent: Thursday, January 24, 2002 11:29 AM To: nanog(a)merit.edu Subject: Anyone here from Microsoft? Hacked? If anyone from Microsoft is on the list. You might want to check to see if this is a hack or an old employee being funny. I did enjoy the link that sent me to McDonalds to apply for a job. Just curious, does anyone know what servers they run their content on? Is it still NT? Hope this info helps the webmaster. David http://www.microsoft.com&item%3Dq209354@hardware.no/nyheter/feb01/Q209354%20 -%20HOWTO.htm -- David Diaz dave(a)smoton.net Smartphotons (smotons) trump dumb photons

1 0

RE: Crawler Ettiquette
by Hunter, Jonathan 24 Jan '02

24 Jan '02

Hi, > a) Obey robots.txt files > b) Allow network admins to automatically have their > netblocks exempted on request > c) Allow ISP's caches to sync with it. I don't know if this is already on your list, but I'd also suggest "d) Rate-limiting of requests to a netblock/server". I haven't got any references immediately to hand, but I do seem to recall a crawler written in such a way that it remained "server-friendly" and would not fire off too many requests too quickly. > ISPs who cache would have an advantage if they used the cache > developed by this project to load their tables, but I do not > know if there is an internet-wide WCCP or equivalent out there > or if the improvement is worth the management overhead. It may be worth having a quick look at http://www.ircache.net/ - there is a database of known caches available through a WHOIS interface, amongst other things. HTH, Jonathan

1 0

Crawler Ettiquette
by Deepak Jain 23 Jan '02

23 Jan '02

I figured this was the best forum to post this, if anyone has suggestions where this might be better placed, please let me know. A University in our customer base has received funding to start a reasonably large spider project. It will crawl websites [search engine fashion] and save certain parts of the information it receives. This information will be made available to research institutions and other concerns. We have been asked for recommendations on what functions/procedures they can put in to be good netizens and not cause undo stress to networks out there. On the list of functions: a) Obey robots.txt files b) Allow network admins to automatically have their netblocks exempted on request c) Allow ISP's caches to sync with it. There are others, but they all revolve around a & b. C was something that seemed like a good idea, but I don't know if there is any real demand for it. Essentially, this project will have at least 1Gb/s of inbound bandwidth. Average usage is expected to be around 500mb/s for the first several months. ISPs who cache would have an advantage if they used the cache developed by this project to load their tables, but I do not know if there is an internet-wide WCCP or equivalent out there or if the improvement is worth the management overhead. Because the funding is there, this project is essentially a certainty. If there are suggestions that should be added or concerns that this raises, please let me know [privately is fine]. All input is appreciated, DJ

2 2

Looking glasses with programmatic interface?
by Stephane Bortzmeyer 23 Jan '02

23 Jan '02

I'm surprised to see that every looking glass I know is only reachable via an interface made for humans, not for programs. When you want to query the LG via a program (for instance to monitor your routes on a distant LG), you need to emulate an human being and parse HTML (or sometimes Cisco text) replies. Does anyone operate looking glasses with an interface made for programs such as XML-RPC <URL:http://www.xml-rpc.org/>, SOAP <URL:http://www.soapware.org/>, Java RMI, Corba (no, I'm joking) or Microsoft COM (joking again)? What do you think of a standard XML-RPC API for looking glasses?

5 5

RE: AS migration
by Miguel de Leon Dimayuga 23 Jan '02

23 Jan '02

hey justin, Collapsing AS'es has its technical benefits, ultimately resulting in financial benefits. For starters, you'll be able to manage your transit traffic better (and thus grow your pipes in a more sane manner). You'll also be able to control your inter-AS traffic much better (especially if you plan to eventually collapse and use common servers/services across AS'es). An example of this is consolidating your news server farms or mail server farms. One approach on the BGP front before merging AS'es is to make sure you have a common local pref and community infrastructure among the AS'es you want to merge. attempting to go confeds with mismatching local pref and community based policies might be a headache. On the IGP front, make sure that your private networks are distinct among AS'es. Once you merge OSPF entities, you could have duplicate 192.168.x.x spaces. That's a start. my 0.02 cents worth. Btw, if you make it to nanog, make sure to grab josh fleishman/jeb linton of mindspring/earthlink a line. They also went through a big AS consolidation a few months ago. Miguel de Leon Dimayuga God does not ask us to be successful, Networks & Technologies only to be faithful. Cbeyond Communications -Mother Teresa of Calcutta

3 2