On Tue, Jan 19, 2021 at 8:20 AM Josh Luthman <josh@imaginenetworksllc.com> wrote:

I'm kind of confused when your concern is the reputability and yet you're providing your own IP space.

Hi Josh, I'm not above discarding announcements with a "bulletproof" hoster in the AS path. Are you? Regards, Bill Herrin On Tue, Jan 19, 2021 at 8:20 AM Josh Luthman <josh@imaginenetworksllc.com> wrote:

I'm kind of confused when your concern is the reputability and yet you're providing your own IP space.

It sounds more to me like you want to put 2-3 boxes in a data center. For that pretty much any decent sized data center in any state would work for the US.

Josh Luthman 24/7 Help Desk: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373

On Tue, Jan 19, 2021 at 11:14 AM William Herrin <bill@herrin.us> wrote:

...

On Tue, Jan 19, 2021 at 7:45 AM Bryan Holloway <bryan@shout.net> wrote:

...

Looking for a reputable (i.e., no hosting of spammers or other ne'er-do-wells) hosting provider with possibly a global footprint. If not, US is #1 desire; EU #2.

...

* Desire to host 2-3 hypervisors, probably running something akin to Proxmox ... * ~5-10TB storage with the possibility of expansion ...

Howdy,

You should clarify what sort of hosting service you're looking for. A normal cloud service won't see you running your own hypervisors. A server farm will see you deploy your own hardware with whatever storage you choose to install. Only a "bare metal" cloud service would meet both of your listed requirements, where you lease both the equipment and hosting service from the provider. However, combined with your BGP requirement your options are very limited and expanding storage usually means "lease a different one of our servers."

Regards, Bill Herrin

-- Hire me! https://bill.herrin.us/resume/

-- Hire me! https://bill.herrin.us/resume/

Fair questions -- answers in-line ... On 1/19/21 5:19 PM, Josh Luthman wrote:

I'm kind of confused when your concern is the reputability and yet you're providing your own IP space.

I care about the hosting environment's upstreams' opinions of the downstream. That is, I don't want to be in a situation where some jamoke (or jamokes) does something which affects me from a collateral damage standpoint.

It sounds more to me like you want to put 2-3 boxes in a data center. For that pretty much any decent sized data center in any state would work for the US.

I would like to stop personally dealing with bare-metal. That's what I'm doing now.

Perhaps I'm missing something, but in your #1 example "Cloud", what prevents me from running a Proxmox ISO (which is more or less Debian) vs. a "standard" Debian install on the provider's virtual server? If I can, I've succeeded. That is the sort of hosting provider I'm looking for, if they exist. #2 would be suitable, but it seems to be that if leased bare-metal dies, it will be some time for ETR. Less desirable, but I'm open to ideas. #3 I do now. Trying to move away from that. On 1/19/21 5:44 PM, William Herrin wrote:

On Tue, Jan 19, 2021 at 8:31 AM Bryan Holloway <bryan@shout.net> wrote:

...

I would like to stop personally dealing with bare-metal. That's what I'm doing now.

Hi Bryan,

Cloud = you get virtual servers with virtual storage, generally adjustable to meet your needs. You manage the operating systems and storage within the virtual environment. You DO NOT manage the host operating systems or hypervisors.

Bare metal = you lease physical equipment. You manage all software on the equipment including any hypervisors needed to run virtual servers. You DO NOT deal with hardware break/fix, that problem belongs to the service provider.

Colocation = You lease space in a data center. You provide physical equipment in your custom configuration.

With this terminology, at least one of your requirements is unmeetable for contradicting the others. So I ask again for clarification: which of these do you seek?

Regards, Bill Herrin

For #1, are you trying to do "Cloud-ception" e.g. running your own proxmox virtualization on top of an already virtual machine, so that you're basically two layers deep? For #2, of course you need to be able to survive a hardware failure (using RAID1 or some flavour of DRBD for example) but having to think about such things is the "trade-off" of having access to the bare-metal layer.. it does have advantages, for example if you want to install your own virtualization layer without any involvement from the hosting provider. You'd usually have agreements with the hosting provider about how/when hardware replacements would be done. Best regards, Martijn ________________________________ From: NANOG <nanog-bounces+martijnschmidt=i3d.net@nanog.org> on behalf of Bryan Holloway <bryan@shout.net> Sent: 19 January 2021 18:18 To: William Herrin <bill@herrin.us> Cc: NANOG list <nanog@nanog.org> Subject: Re: Hosting recommendations ... ? Perhaps I'm missing something, but in your #1 example "Cloud", what prevents me from running a Proxmox ISO (which is more or less Debian) vs. a "standard" Debian install on the provider's virtual server? If I can, I've succeeded. That is the sort of hosting provider I'm looking for, if they exist. #2 would be suitable, but it seems to be that if leased bare-metal dies, it will be some time for ETR. Less desirable, but I'm open to ideas. #3 I do now. Trying to move away from that. On 1/19/21 5:44 PM, William Herrin wrote:

On Tue, Jan 19, 2021 at 8:31 AM Bryan Holloway <bryan@shout.net> wrote:

...

I would like to stop personally dealing with bare-metal. That's what I'm doing now.

Hi Bryan,

Cloud = you get virtual servers with virtual storage, generally adjustable to meet your needs. You manage the operating systems and storage within the virtual environment. You DO NOT manage the host operating systems or hypervisors.

Bare metal = you lease physical equipment. You manage all software on the equipment including any hypervisors needed to run virtual servers. You DO NOT deal with hardware break/fix, that problem belongs to the service provider.

Colocation = You lease space in a data center. You provide physical equipment in your custom configuration.

With this terminology, at least one of your requirements is unmeetable for contradicting the others. So I ask again for clarification: which of these do you seek?

Regards, Bill Herrin

AceHost.com or Hivelocity. Sent from my Verizon, Samsung Galaxy smartphone Get Outlook for Android<https://aka.ms/ghei36> ________________________________ From: NANOG <nanog-bounces+michael=spears.io@nanog.org> on behalf of Bryan Holloway <bryan@shout.net> Sent: Tuesday, January 19, 2021 12:37:14 PM To: Martijn Schmidt <martijnschmidt@i3d.net>; William Herrin <bill@herrin.us> Cc: NANOG list <nanog@nanog.org> Subject: Re: Hosting recommendations ... ? You make an excellent point, Martijn ... (and I suspect this is what Bill was pointing out ...) Virtualization on top of virtualization is inherently not the best idea. I guess I'm looking for flexibility in the sense of being able to spin up additional VMs at my leisure. In which case #2 could be suitable in the right environment. ... in which case SLAs would be tantamount to success ... So ... that said, any recommendations? :) On 1/19/21 6:32 PM, Martijn Schmidt wrote:

For #1, are you trying to do "Cloud-ception" e.g. running your own proxmox virtualization on top of an already virtual machine, so that you're basically two layers deep?

For #2, of course you need to be able to survive a hardware failure (using RAID1 or some flavour of DRBD for example) but having to think about such things is the "trade-off" of having access to the bare-metal layer.. it does have advantages, for example if you want to install your own virtualization layer without any involvement from the hosting provider. You'd usually have agreements with the hosting provider about how/when hardware replacements would be done.

Best regards, Martijn ------------------------------------------------------------------------ *From:* NANOG <nanog-bounces+martijnschmidt=i3d.net@nanog.org> on behalf of Bryan Holloway <bryan@shout.net> *Sent:* 19 January 2021 18:18 *To:* William Herrin <bill@herrin.us> *Cc:* NANOG list <nanog@nanog.org> *Subject:* Re: Hosting recommendations ... ? Perhaps I'm missing something, but in your #1 example "Cloud", what prevents me from running a Proxmox ISO (which is more or less Debian) vs. a "standard" Debian install on the provider's virtual server?

If I can, I've succeeded. That is the sort of hosting provider I'm looking for, if they exist.

#2 would be suitable, but it seems to be that if leased bare-metal dies, it will be some time for ETR. Less desirable, but I'm open to ideas.

#3 I do now. Trying to move away from that.

On 1/19/21 5:44 PM, William Herrin wrote:

...

On Tue, Jan 19, 2021 at 8:31 AM Bryan Holloway <bryan@shout.net> wrote:

...

I would like to stop personally dealing with bare-metal. That's what I'm doing now.

Hi Bryan,

Cloud = you get virtual servers with virtual storage, generally adjustable to meet your needs. You manage the operating systems and storage within the virtual environment. You DO NOT manage the host operating systems or hypervisors.

Bare metal = you lease physical equipment. You manage all software on the equipment including any hypervisors needed to run virtual servers. You DO NOT deal with hardware break/fix, that problem belongs to the service provider.

Colocation = You lease space in a data center. You provide physical equipment in your custom configuration.

With this terminology, at least one of your requirements is unmeetable for contradicting the others. So I ask again for clarification: which of these do you seek?

Regards, Bill Herrin

I would look at https://www.webhostingtalk.com forums and specifically the "Dedicated Hosting Offers" forums https://www.webhostingtalk.com/forumdisplay.php?f=36 for providers and deals. On Tue, Jan 19, 2021 at 1:34 PM Forrest Christian (List Account) < lists@packetflux.com> wrote:

Maybe this will help:

I use vultr. I have also brought my own address space and am announcing it to them from one of their instances (vm's) with BGP. They are set up such that you can use a private AS if you don't have your own and are ok with them announcing this from their AS (after they strip off the private as).

If you told me right this second that you needed a Ubuntu Server spun up, I could be logged into the interface, select a location/size and have it running within 60 seconds - with the complete Ubuntu install done. Most of my instances are running under their $5 plan which is 25GB of disk space, 1 CPU, 1G of memory, and 1TB of transfer. Oh, and one or two are on the $6 plan which has slightly more disk and is on a faster CPU. They obviously have various options that go up from there all the way to dedicated servers. In a couple of locations, they also have a smaller IPv6-only server for $2.50.

I haven't found anything I need anything larger than their $5 or $6 plan for, but I also am not running any heavy workloads. Basically a static webpage, a ticketing system, a couple of DNS servers, and the like. I'm currently testing voip(Asterisk) on the platform as well, and based on testing so far will likely be moving it into full production in the next few weeks.

I used ubuntu as an example, they also have all of the mainstream linux distros and the BSD's and I think in some locations windows available. They also have a pretty good selection of preconfigured applications (aka common CMS'es, eCommerce platforms, etc). All of these are in the ~60 seconds to spin up category. You can also upload your own ISO or use a pre-uploaded one from their library of less commonly used applications/operating systems.

I've been running on the platform for about a year after having enough of a shared hosting provider for some of the stuff and running on my own metal in a datacenter for other stuff. So far I'm very happy with them.

If you want to try them out, I noticed as I logged in to look up what the $5 service includes that they're running a a "refer a friend and they get $100 to test the service" promo. I normally don't pass on referral codes but I figured if you're evaluating this you might want the $100 credit (for up to 30 days of service). If so, the link for that offer is https://www.vultr.com/?ref=8776996-6G .

On Tue, Jan 19, 2021 at 10:39 AM Bryan Holloway <bryan@shout.net> wrote:

...

You make an excellent point, Martijn ... (and I suspect this is what Bill was pointing out ...)

Virtualization on top of virtualization is inherently not the best idea.

I guess I'm looking for flexibility in the sense of being able to spin up additional VMs at my leisure. In which case #2 could be suitable in the right environment.

... in which case SLAs would be tantamount to success ...

So ... that said, any recommendations? :)

On 1/19/21 6:32 PM, Martijn Schmidt wrote:

...

For #1, are you trying to do "Cloud-ception" e.g. running your own proxmox virtualization on top of an already virtual machine, so that you're basically two layers deep?

For #2, of course you need to be able to survive a hardware failure (using RAID1 or some flavour of DRBD for example) but having to think about such things is the "trade-off" of having access to the bare-metal layer.. it does have advantages, for example if you want to install your own virtualization layer without any involvement from the hosting provider. You'd usually have agreements with the hosting provider about how/when hardware replacements would be done.

Best regards, Martijn ------------------------------------------------------------------------ *From:* NANOG <nanog-bounces+martijnschmidt=i3d.net@nanog.org> on behalf of Bryan Holloway <bryan@shout.net> *Sent:* 19 January 2021 18:18 *To:* William Herrin <bill@herrin.us> *Cc:* NANOG list <nanog@nanog.org> *Subject:* Re: Hosting recommendations ... ? Perhaps I'm missing something, but in your #1 example "Cloud", what prevents me from running a Proxmox ISO (which is more or less Debian) vs. a "standard" Debian install on the provider's virtual server?

If I can, I've succeeded. That is the sort of hosting provider I'm looking for, if they exist.

#2 would be suitable, but it seems to be that if leased bare-metal dies, it will be some time for ETR. Less desirable, but I'm open to ideas.

#3 I do now. Trying to move away from that.

On 1/19/21 5:44 PM, William Herrin wrote:

...

On Tue, Jan 19, 2021 at 8:31 AM Bryan Holloway <bryan@shout.net> wrote:

...

I would like to stop personally dealing with bare-metal. That's what I'm doing now.

Hi Bryan,

Cloud = you get virtual servers with virtual storage, generally adjustable to meet your needs. You manage the operating systems and storage within the virtual environment. You DO NOT manage the host operating systems or hypervisors.

Bare metal = you lease physical equipment. You manage all software on the equipment including any hypervisors needed to run virtual servers. You DO NOT deal with hardware break/fix, that problem belongs to the service provider.

Colocation = You lease space in a data center. You provide physical equipment in your custom configuration.

With this terminology, at least one of your requirements is unmeetable for contradicting the others. So I ask again for clarification: which of these do you seek?

Regards, Bill Herrin

-- - Forrest

Proxmox specifically isn't much more than a wrapper for standard Linux KVM, which can support nested virtualization. In my limited experience with nesting, it doesn't work half bad as one would expect, but I haven't used it in a stressed environment with anything substantial running that way. On Tue, Jan 19, 2021 at 1:53 PM William Herrin <bill@herrin.us> wrote:

On Tue, Jan 19, 2021 at 9:18 AM Bryan Holloway <bryan@shout.net> wrote:

...

Perhaps I'm missing something, but in your #1 example "Cloud", what prevents me from running a Proxmox ISO (which is more or less Debian) vs. a "standard" Debian install on the provider's virtual server?

Hi Bryan,

I haven't used Proxmox but from a 60 second glance through Google that looks like you're asking for nested virtualization. If it works at all, you'd take a double-hit on everything that wants to run in ring 0, a double-hit on virtualized I/O and a double-hit for OS overhead making the result more than a little sluggish. Kinda has "bad idea" written all over it.

As I understand it, you can "cat /sys/module/kvm*/parameters/nested" in one of the service provider's VMs and if the answer is "1" or "Y" for the CPU type which matches the exposed CPU then what you're asking for will probably work. For some definition of work anyway.

I use Vultr for my primary BGP exit and have found it largely painless. The VMs I have there DO NOT support nested virtualization. They do claim a bare metal offering but it's currently listed as sold out in all of their data centers. They also claim to provide mountable block storage for compute instances up to 10TB per, but I haven't worked with that feature, I presume it only applies to virtual servers, and it looks like it's only available in one of their data centers.

Regards, Bill Herrin

-- Hire me! https://bill.herrin.us/resume/

On Tue, Jan 19, 2021 at 10:50:31AM -0800, William Herrin wrote:

I use Vultr for my primary BGP exit and have found it largely painless. The VMs I have there DO NOT support nested virtualization.

That's odd, I have run Xen in standard Vultr VMs, and it works. It's not fast, though. Thanks, Ward.

On 1/19/21 6:33 PM, Brandon Martin wrote:

On 1/19/21 11:44 AM, William Herrin wrote:

...

Cloud = you get virtual servers with virtual storage, generally adjustable to meet your needs. You manage the operating systems and storage within the virtual environment. You DO NOT manage the host operating systems or hypervisors.

It's worth pointing out that nested virtualization is a thing these days, and some providers might even support it!  That means you could buy one large instance and sub-divide it yourself into multiple VMs if you want to.

In practice, unless you need that flexibility to dynamically spin the VMs up and down with various specs AND don't want to or cannot use a provider's API for that, I'm not sure why you'd want to if you didn't have to for some crazy reason.

I'm very curious about your assertion: Is nested virtualization really a thing? I mean, I'm not exactly trying to render Pixar's latest movie ... just trying to push some bits around (light web-sites, some e-mail ...) It just seems inherently prone to issues. Could you back this up with any white-papers or documentation on the subject? I'm genuinely interested ... - bryan

On Tue, Jan 19, 2021 at 3:27 PM Brandon Martin <lists.nanog@monmotha.net> wrote:

On 1/19/21 12:56 PM, Bryan Holloway wrote:

...

I'm very curious about your assertion:

Is nested virtualization really a thing?

I mean, I'm not exactly trying to render Pixar's latest movie ... just trying to push some bits around (light web-sites, some e-mail ...)

It just seems inherently prone to issues.

Could you back this up with any white-papers or documentation on the subject? I'm genuinely interested ...

With KVM, if you have a recent kernel and qemu, it pretty much "just works" on supported hardware. AFAIK Xen supports "Xen on Xen", too, but I haven't used it and don't know much about it.

The use case is pretty much exactly this. You (the product consumer) are handed a product that amounts to a virtual machine on somebody else's $BIGBOX. You want to deploy multiple virtual machines where you have direct control over their lifecycle, configuration, etc. and can bring in additional I/O resources, etc. at the hypervisor level (consider that, with KVM, the Linux kernel basically IS the hypervisor). So, you run one or more VMs inside the top level VM that you're handed.

It's full of lots of little wiggles and can be a pain to maintain if you have visibility into both levels of the equation, but it does seem to work and is surprisingly performant.

Nested virtualization is, however, pretty widely deployed for production workloads. Take for instance Juniper's vSRX and vMX products. Both require nested virtualization support as they run a Wind River Linux instance with KVM directly on your virtual machine which may be on vmware, KVM, Hyper-V, etc, and then Junos running nested within the WRL KVM virtual machine. So it's not like this is something that's uncommon or not trustworthy, lots of us are doing it every day with no serious problems and entirely acceptable performance on modern hardware. I run a decent fleet of vSRX's on Hyper-V and it works well. YMMV as always based on your own platform, but I don't think that nested virtualization is something that we should be steering clear of at this point. - mdh Matt Harris|Infrastructure Lead Engineer 816-256-5446|Direct Looking for something? Helpdesk Portal|Email Support|Billing Portal We build and deliver end-to-end IT solutions.

----- Original Message -----

From: "Keith Medcalf" <kmedcalf@dessus.com>

...

Is nested virtualization really a thing?

Real Computers have been running VMs inside VMs for about 50 years. Bringing this technology to "bitty boxes" is a recent thing.

Sure, but VM is a bit more mature than KVM. :-) Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274

On 1/19/21 9:33 AM, Brandon Martin wrote:

On 1/19/21 11:44 AM, William Herrin wrote:

...

Cloud = you get virtual servers with virtual storage, generally adjustable to meet your needs. You manage the operating systems and storage within the virtual environment. You DO NOT manage the host operating systems or hypervisors.

It's worth pointing out that nested virtualization is a thing these days, and some providers might even support it!  That means you could buy one large instance and sub-divide it yourself into multiple VMs if you want to.

In practice, unless you need that flexibility to dynamically spin the VMs up and down with various specs AND don't want to or cannot use a provider's API for that, I'm not sure why you'd want to if you didn't have to for some crazy reason.

Except for the problem that you pay a premium for larger vm's. you can't compete with t instances on aws, for starters. Mike