Date: Fri, 08 Feb 2002 22:09:12 -0800 From: Stephen Stuart <stuart@tech.org> Subject: Re: Reducing Usenet Bandwidth To: David Schwartz <davids@webmaster.com> Cc: nanog@merit.edu

[snip]

Some of those problems in that vein that appeared insurmountable ten years ago might have solutions in current "peer-to-peer" networking technologies (thus the off-hand comment about Napster).

What you're really asking for is a NNTP-over-FastTrack (Kazaa/Morpheus) feed object. This is sort of/really an odd idea, but if you think about it, all the required parts are pretty easy to articulate in skeleton form: 1) Each file dumped into FastTrack/Kazaa/Morpheus would get as its "filename" the article's message-ID. A site that wanted to retrieve an article would issue (via FastTrack/Kazaa/Morpheus) a search request for that message-ID, retrieve the article from a FastTrack/Kazaa/Morpheus server that offered that file, and then add it to the local news news server spool (and/or, optionally, a FastTrack/Kazaa/Morpheus shared directory on the local server). 2) How would servers know what article IDs to ask for? Well, Usenet overview data for each group could also be published by news servers via FastTrack/Kazaa/Morpheus with filenames of the format pathtag!hierarchy.subhierarchy.group!yy:mm:dd:hh:mm:ss The pathtag would be required because obviously different news servers will know about different articles, and you will want to disambiguate which server's overview data you want (not because there's any expectation that any news server will in fact provide public access to any articles, but simply because one might expect that N servers might all simultaneously offer comparable but non-identical overview data for any given group, and you might want to choose the view from server foo, or bar, or baz over other alternatives). The hierarchy.subhierarchy.group component of the overview file name simply allows folks to identify the overview data file they'd want. The date/time stamp (GMT) provides a means of selecting the most recently available overview data from a list of matching responses. Overview data would obviously need to be cryptographically signed with a key tied to the pathtag to avoid problems with people providing forged overview data (wouldn't want people to be fed lists of bogus/inappropriate lists of message ID's as a DOS/spam attack, right?). Overview files could be generated at some server-determined periodicity, and to avoid inspiring any sort of synchronization effect, I'll omit mentioning a value here except to say that I've got something in the double-digits-of- minutes in mind here. (Besides, if you're going to publish overview data for some tens of thousands of groups, it will take a while to walk that tree). 3) Client servers interested in retrieving articles via FastTrack/Kazaa/Morpheus could routinely retrieve overview records for groups of interest, retrieving all/some of the articles as a "prefetch" for popular groups routinely read by their users, or the server could wait until articles have been requested, then retrieve an overview file, then retrieve articles. Alternatively, individual users with "News over FastTrack/Kazaa/Morpheus" clients could be retrieving articles directly from Kazaa, removing any need for an intervening news server whatsoever (although hopefully a local "regular" news server would always be faster than a distributed FastTrack/Morpheus/Kazaa-based news spool). Regards, Joe P.S. Needless to say, this would really change the FastTrack/Kazaa/Morpheus content base, and given the volumes Usenet is currently transferring, would be sort of an interesting experiment if we're talking about a full feed...