On 2020-05-13 11:00, Mark Delany wrote:
On 13May20, Denys Fedoryshchenko allegedly wrote:
What about introducing some cache offloading, like CDN doing? (Google, Facebook, Netflix, Akamai, etc)
Maybe some opensource communities can help as well
Surely someone has already thought thru the idea of a community CDN? Perhaps along the lines of pool.ntp.org? What became of that discussion?
Maybe a TOR network could be repurposed to cover the same ground.
Mark. I believe tor is not efficient at all for this purposes. Privacy have very high overhead.
Several schemes exist: 1)ISP announce in some way subnets he want to be served from his cache. 1.A)Apple cache way - just HTTP(S) request will turn specific IP to ISP cache. Not secure at all. 1.B)BGP + DNS, most common way. ISP does peering with CDN, CDN will return ISP cache nodes IP's to DNS requests. It means for example content.archive.org will have local node A/AAAA records (btw where is IPv6 for archive?) for customers of ISP with this node, or anybody who is peering with it. Huge drawback - archive.org will need to provide TLS certificates for web.archive.org each local node, this is bad and probably no-go. Yes, i know some schemes exist, that certificate is not present on local node, but some "precalculated" result used, but it is too complex. 1.C)BGP + HTTP redirect. If ISP has peering with archive.org, to all subnets announced users will get 302 or some HTTP redirect. Next is almost same and much better, but will require small modifications of content engine or frontend balancers. 1.D)BGP + HTTP rewrite. If ISP <*same as before*> URL is rewritten within content e.g. http://web.archive.org/web/20200511193226/https://git.kernel.org/torvalds/t/... will appear as http://emu.st.node.archive.org/web/20200511193226/https://git.kernel.org/tor... or http://archive-org.proxy.emu.st/web/20200511193226/https://git.kernel.org/to... In second option ISP can handle SSL certificate by himself. 2)BGP announce of archive.org subnets locally. Prone to leaks, require TLS certificates and etc, no-go. You can still modify some schemes, and make other options that no one has yet implemented. For example, to do everything through javascript (CDNs cannot afford it, because of way they work), and for example, website generate content links dynamically, for that client request some /config.json file (which is dynamically generated and cached for a while), so we give it to IPs that have a local node - URL of the local node, for the rest - default url.