On Mon, Jun 3, 2019 at 1:09 PM Fletcher Kittredge <fkittred@gwi.net> wrote:

Here is your checklist in descending order of importance:

market opportunity finding the right partners (see below) financial sales and marketing organizational capacity and HR legal, regulatory capital acquisition security ... ... ... technical including equipment selection, routing policy, filtering, etc

It is a stone cold lock that the success of your new ISP will governed by factors other than technical. Your most important task is to find competent financial and marketing people you can respect and trust. If the market opportunity exists and you find them, you will succeed. If you don't, all the technical excellence in the world won't help you. The road is littered with technically excellent companies that failed.

Indeed, but you *also* need to have some technical clue. Two or three years ago a friend and I tried to start a local wireless ISP -- I was doing this purely as a "My home Internet access sucks, and I'll happily donate time, equipment, IP space and some startup capital to fix this" play -- unfortunately it turns out that he and I had very different ideas on, well, basically everything. I wanted an actual architecture / design, and diagrams and routin' and such. He was much more of "We don't need a list of IPs, if I ping it and can't reach it it must be free" / "routing is too hard, let's just put it all in a switch and... um... NAT!". I wanted a plan, and was willing to put in the time and effort to build Ansible / Puppet / an NMS / AAA, etc, he was more seat-of-the-pants. But yes, even if we had good technology this would have failed - there was no real business plan (other than "The current provider is really bad, if we build something else, people will be breaking down the door to sign up"), no real marketing plan (see previous), etc. He was also a bit of a gun nut, and so would arrive at customers with a (holstered) firearm belted on -- even in Virginia this was not a winning business move. Starting a successful ISP is this day and age is hard - make sure that, if you do it, you and whoever you are doing this with are compatible, are both committed, and have similar views on things... W

On Mon, Jun 3, 2019 at 8:05 AM Mehmet Akcin <mehmet@akcin.net> wrote:

...

hi there,

I know there are folks from lots of small ISPs here and I wanted to check-in on asking few advice points as I am involved building an ISP from green-field.

Usually, it's pretty straight forward to cover high-level important things, filters, routing policies, etc.but we all know the devil is in the details.

I am putting together a public DOs and DONTs blog post and would love to hear from those who have built ISPs and have recommendations from Billing to Interconnection, Routing policy to Out of the band & console setup, Software recommendations, etc. Bottom line is that I would like to publish a checklist with these recommendations which I hope will be useful for all.

thanks in advance for your help and recommendation.

Mehmet

-- Fletcher Kittredge GWI 207-602-1134 www.gwi.net

-- I don't think the execution is relevant when it was obviously a bad idea in the first place. This is like putting rabid weasels in your pants, and later expressing regret at having chosen those particular rabid weasels and that pair of pants. ---maf

On Mon, Jun 3, 2019 at 2:55 PM Fletcher Kittredge <fkittred@gwi.net> wrote:

I would respectfully point out that my point about the importance of finding the right partners. For you, sounds like it was good to have opportunity to get out of this venture.

Oh, goodness yes -- however, I *still* have barely working Internet access at that location (it's basically a weekend home) -- I've somewhat started down the Jared Mauch "buy used vibratory plow, trench (house is off a dirt road, neighbors won't have an issue with right of way if I provide them free bits!), install fiber" path. My driveway is @1/4 mile, the dirt road is 0.6 miles and at the end of the road there is (what used to be) Quest fiber. Unfortunately the fiber was originally run for Mount Weather (https://en.wikipedia.org/wiki/Mount_Weather_Emergency_Operations_Center) and no-one I've asked seems to know who controls it now, and or swears that it doesn't exist (although the local Miss Utility / VA811 will come mark it if you call). Some dark nights, when trying to use the network while everyone is using NetFlix, and the RTT for my SSH sessions starts exceeding 1,500ms I strongly consider walking to the end of the road with a nice, sharp shovel, and then talking to the splicers when they come repair the damage :-) W

On Mon, Jun 3, 2019 at 2:40 PM Warren Kumari <warren@kumari.net> wrote:

...

On Mon, Jun 3, 2019 at 1:09 PM Fletcher Kittredge <fkittred@gwi.net> wrote:

...

Here is your checklist in descending order of importance:

market opportunity finding the right partners (see below) financial sales and marketing organizational capacity and HR legal, regulatory capital acquisition security ... ... ... technical including equipment selection, routing policy, filtering, etc

It is a stone cold lock that the success of your new ISP will governed by factors other than technical. Your most important task is to find competent financial and marketing people you can respect and trust. If the market opportunity exists and you find them, you will succeed. If you don't, all the technical excellence in the world won't help you. The road is littered with technically excellent companies that failed.

Indeed, but you *also* need to have some technical clue. Two or three years ago a friend and I tried to start a local wireless ISP -- I was doing this purely as a "My home Internet access sucks, and I'll happily donate time, equipment, IP space and some startup capital to fix this" play -- unfortunately it turns out that he and I had very different ideas on, well, basically everything. I wanted an actual architecture / design, and diagrams and routin' and such. He was much more of "We don't need a list of IPs, if I ping it and can't reach it it must be free" / "routing is too hard, let's just put it all in a switch and... um... NAT!". I wanted a plan, and was willing to put in the time and effort to build Ansible / Puppet / an NMS / AAA, etc, he was more seat-of-the-pants.

But yes, even if we had good technology this would have failed - there was no real business plan (other than "The current provider is really bad, if we build something else, people will be breaking down the door to sign up"), no real marketing plan (see previous), etc.

He was also a bit of a gun nut, and so would arrive at customers with a (holstered) firearm belted on -- even in Virginia this was not a winning business move.

Starting a successful ISP is this day and age is hard - make sure that, if you do it, you and whoever you are doing this with are compatible, are both committed, and have similar views on things...

W

...

On Mon, Jun 3, 2019 at 8:05 AM Mehmet Akcin <mehmet@akcin.net> wrote:

...

hi there,

I know there are folks from lots of small ISPs here and I wanted to check-in on asking few advice points as I am involved building an ISP from green-field.

Usually, it's pretty straight forward to cover high-level important things, filters, routing policies, etc.but we all know the devil is in the details.

I am putting together a public DOs and DONTs blog post and would love to hear from those who have built ISPs and have recommendations from Billing to Interconnection, Routing policy to Out of the band & console setup, Software recommendations, etc. Bottom line is that I would like to publish a checklist with these recommendations which I hope will be useful for all.

thanks in advance for your help and recommendation.

Mehmet

-- Fletcher Kittredge GWI 207-602-1134 www.gwi.net

-- I don't think the execution is relevant when it was obviously a bad idea in the first place. This is like putting rabid weasels in your pants, and later expressing regret at having chosen those particular rabid weasels and that pair of pants. ---maf

-- Fletcher Kittredge GWI 207-602-1134 www.gwi.net

-- I don't think the execution is relevant when it was obviously a bad idea in the first place. This is like putting rabid weasels in your pants, and later expressing regret at having chosen those particular rabid weasels and that pair of pants. ---maf

On 3/Jun/19 15:41, Fletcher Kittredge wrote:

Here is your checklist in descending order of importance:

1. market opportunity 2. finding the right partners (see below) 3. financial 4. sales and marketing 5. organizational capacity and HR 6. legal, regulatory 7. capital acquisition 8. security 9. ... 10. ... 11. ... 12. technical including equipment selection, routing policy, filtering, etc

13. Don't run Mikrotik. I'm kidding... I think :-)... Mark.

triggered :) On Tue, Jun 4, 2019 at 11:31 AM Bryan Holloway <bryan@shout.net> wrote:

On 6/4/19 9:20 AM, Mark Tinka wrote:

...

On 3/Jun/19 15:41, Fletcher Kittredge wrote:

...

Here is your checklist in descending order of importance:

1. market opportunity 2. finding the right partners (see below) 3. financial 4. sales and marketing 5. organizational capacity and HR 6. legal, regulatory 7. capital acquisition 8. security 9. ... 10. ... 11. ... 12. technical including equipment selection, routing policy, filtering, etc

13. Don't run Mikrotik.

I'm kidding... I think :-)...

Mark.

14. Go with K56flex, not X2.

On Mon, Jun 03, 2019 at 01:48:33PM +0000, Mel Beckman wrote:

I’m constantly amazed at the number of even medium-sized ISPs that have no network monitoring. An NMS should go in as the first software component — before billing starts and the provider is on the hook to deliver.

often people are using tools like quickbooks to start, these don't support integration with networking tools. You see tools like Sonar or powercode in use. Some of this is changing with newer tools like UNMS and UCRM in some spaces, but often these are vendor locked or don't integrate well.

The second lacking component is a ticket system, which is silly because turnkey cloud services are not expensive, and open source solutions abound for budget-limited operators.

The number of people who can't do sysadmin functions is high. there's a reason SaaS is a thing, but the costs are often enough to force someone to roll their own. Take something like powercode with a $1/subscriber fee which adds up quickly.

The third component failure is security, including weak and default (!) passwords, failure to use real certificates, and the complete lack of 2FA or MFA. Security also requires data surveillance, in the form of net flow analysis.

Much of this is because hardware has defaults that aren't sane or lack some ZTP or provisioning that you can do. How do you do this with UBNT, Tik or other cost optimized hardware?

The “two guys and a router” business model must be upgraded with more planning and a cohesive operating plan.

Most large networks are run with small teams, while usually more than 2 it's often not more than 10 to do the arch + eng work necessary. If you have more, they're often doing installer work not actual eng work. - Jared

...

On Jun 3, 2019, at 5:05 AM, Mehmet Akcin <mehmet@akcin.net> wrote:

hi there,

I know there are folks from lots of small ISPs here and I wanted to check-in on asking few advice points as I am involved building an ISP from green-field.

Usually, it's pretty straight forward to cover high-level important things, filters, routing policies, etc.but we all know the devil is in the details.

I am putting together a public DOs and DONTs blog post and would love to hear from those who have built ISPs and have recommendations from Billing to Interconnection, Routing policy to Out of the band & console setup, Software recommendations, etc. Bottom line is that I would like to publish a checklist with these recommendations which I hope will be useful for all.

thanks in advance for your help and recommendation.

Mehmet

-- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.

On Wed, Jun 5, 2019 at 5:44 AM William Waites <ww@styx.org> wrote:

It's not enough to have monitoring and a ticket system. You need to pay attention to them, care for them and feed them. I can't count the number of ticket systems full of ancient and irrelevant things or monitoring systems that people have forgotten about or don't know how to add new stuff to. Even the cycle of,

Some points to consider when monitoring your network: 1. Beware early automation. If you write a generator to go and monitor all your stuff without addressing how operators will change things one-off (which is hard to design well) the other operators will find the monitoring system unusable. Which means they won't update it when stuff is added and changed. Making it quickly useless. 2. Careful aggregating alarms. That big green or red light is useless. The operator has to be able to start with the alarm and immediately trace back to exactly what tests and results bubbled up in to the aggregate and from there to the malfunctioning component. If you lose this information during the aggregation process, you're just producing noise. 2. Every alarm must be actionable. When the light goes red, what -exactly- do you want the operator to do as a result? Don't create an alarm until you can offer a detailed and specific answer, and link that answer to the alarm so the operator doesn't have to hunt for it. Regards, Bill Herrin -- William Herrin bill@herrin.us https://bill.herrin.us/

On 6/3/19 9:56 AM, Jon Lewis wrote:

3) Don't advertise one transit provider's routes to another.  Each should    be filtering your routes, but you never know.  Come up with, and use    BGP communities to control route propagation.  As you grow, it sucks    having to update prefix-list filters in multiple places every time    something changes...like a new customer with their own IPs.

To reiterate all this, FILTER EVERYTHING. To start with, explicitly specify in a route-map or similar everything you want to advertise. I usually create a separate route-map for each transit/peer and include what I want to advertise via prefix lists (for my IP space) and/or communities (for downstream BGP-speaking customers if anticipated). When you turn on the session, check what you're squawking AND WHAT YOU'RE FILTERING. You shouldn't be filtering anything you don't expect. Belt + suspenders. The same goes for anything you accept. Obviously for a blended full transit BGP edge router, you're probably going to accept almost everything. But if you only want default + on-net, try to filter using communities from the peer, etc. Again, right when you turn on the session, "sh ip bgp ... filtered" of whatever's equivalent on your platform. If you're filtering something you don't expect to be receiving at all, figure out where the misunderstanding or misconfiguration lies. And of course it goes without saying that, if you've got BGP speaking customers, you filter the heck out of them. Use ROAs and/or RPKI if you can to automatically generate filter lists. Encourage your upstreams to do the same if they're filtering you (and they probably are, or at least should be, if you're new). Remember that you are responsible for every route you advertise, at the end of the day, even if you only advertised it because a downstream network made a boo-boo and you didn't filter it. Filters are useful on your IGP, too, but there's so many ways to set all that up that it's a bit more difficult to come up with nearly universal best practices. Generally speaking, be careful with redistribution, never distribute BGP into IGP or vice versa unless you have a really, really good reason to, and consider filters between both IGP areas/regions or protocols (e.g. RIP coming into OSPF) as well as on redistributions of static/connected to prevent simple typos on a static route or interface configuration from taking down more than just local stuff. It's way, way easier to remove or relax filters later if they prove more of an operational hazard than asset than it is to add or tighten them if they prove insufficient. -- Brandon Martin

This Gem is fantastic by the way, https://nsrc.org/workshops/2015/apricot2015/raw-attachment/wiki/Track1Agenda... On Tue, Jun 4, 2019 at 5:57 AM Warren Kumari <warren@kumari.net> wrote:

On Mon, Jun 3, 2019 at 11:34 PM Brandon Martin <lists.nanog@monmotha.net> wrote:

...

On 6/3/19 9:56 AM, Jon Lewis wrote:

...

3) Don't advertise one transit provider's routes to another. Each

should

...

...

be filtering your routes, but you never know. Come up with, and

use

...

BGP communities to control route propagation. As you grow, it

sucks

...

having to update prefix-list filters in multiple places every time something changes...like a new customer with their own IPs.

To reiterate all this, FILTER EVERYTHING.

To start with, explicitly specify in a route-map or similar everything you want to advertise. I usually create a separate route-map for each transit/peer and include what I want to advertise via prefix lists (for my IP space) and/or communities (for downstream BGP-speaking customers if anticipated).

I think a related *principle* is: "Build everything as though you are expecting to scale."

This doesn't mean "spend lots of money to buy huge [routers|servers|commercial software|<etc>], but rather "when you plan your addressing structure and routing policies and monitoring and device config generation and... keep the in mind the question "If this suddenly takes off, and I hire N more people to run this, can I explain to them how it works? Do I have documentation I can point them at or is it stuck in my head / on the devices? If I need to add another M customers in the next month, can I do that easily?".

This is related to the FILTER EVERYTHING -- when you turn up a new customer / peer / transit / whatever, you shouldn't be sitting around trying to figure out how you will write their route-map / policy-options -- this leads to weird one-offs, and quick hacks. Instead you should have policies already largely designed and simply plug in their prefixes (or, better yet, use bgpq3 or similar to build and populate these). Obviously there will be some cases where a new connection does require some special handling, but that *should* just be a plugin/chain in an existing policy-statement. Related to this is how you end up naming things -- I recently found 9 variants of firewall-filters which basically do:

filter ACCEPT { term ACCEPT { then accept; } } named things like: ACCEPT, ACEPT, Accept, Allow, Permit_all, AcceptAll, dontdrop [0].

Obviously, there is a tension in the "design for scale" - while it would be great to design a complete automation system so that everything from installing a new customer to a new sites is simply typing 'make <thing>' and having everything pull from a database, at some point you will need to actually build a network, or you'll never have customers :-) Just keep in mind that "Am I building myself into a corner here?". E.g it only takes 10 or 15 minutes to install something like NetBox to keep track of addresses (and prefixes and racks and connections and ...) -- stuffing this in a spreadsheet might save you a few minutes *now*, but will this scale? Can $new_person easily figure it out?

W [0]: My personal favorite is: filter Accept_All { term Accept { then { count dropped; reject; } } term filter_<customer> { from { prefix-list { <customer>; } } then accept; } term NEXT { then log; } }

Presumably this all made sense to <name_removed_to_protect_inoccent> when they stuck it in at 3AM to deal with some crazy issue, but...

...

When you turn on the session, check what you're squawking AND WHAT YOU'RE FILTERING. You shouldn't be filtering anything you don't expect. Belt + suspenders.

The same goes for anything you accept. Obviously for a blended full transit BGP edge router, you're probably going to accept almost everything. But if you only want default + on-net, try to filter using communities from the peer, etc. Again, right when you turn on the session, "sh ip bgp ... filtered" of whatever's equivalent on your platform. If you're filtering something you don't expect to be receiving at all, figure out where the misunderstanding or misconfiguration lies.

And of course it goes without saying that, if you've got BGP speaking customers, you filter the heck out of them. Use ROAs and/or RPKI if you can to automatically generate filter lists. Encourage your upstreams to do the same if they're filtering you (and they probably are, or at least should be, if you're new). Remember that you are responsible for every route you advertise, at the end of the day, even if you only advertised it because a downstream network made a boo-boo and you didn't filter it.

Filters are useful on your IGP, too, but there's so many ways to set all that up that it's a bit more difficult to come up with nearly universal best practices. Generally speaking, be careful with redistribution, never distribute BGP into IGP or vice versa unless you have a really, really good reason to, and consider filters between both IGP areas/regions or protocols (e.g. RIP coming into OSPF) as well as on redistributions of static/connected to prevent simple typos on a static route or interface configuration from taking down more than just local stuff.

It's way, way easier to remove or relax filters later if they prove more of an operational hazard than asset than it is to add or tighten them if they prove insufficient. -- Brandon Martin

-- I don't think the execution is relevant when it was obviously a bad idea in the first place. This is like putting rabid weasels in your pants, and later expressing regret at having chosen those particular rabid weasels and that pair of pants. ---maf

For some reason our network looks nothing like that. He either needs to define what a PoP is or define a module that does not have expensive gear like two redundant routers at every location. Regards Baldur tir. 4. jun. 2019 15.46 skrev Mehmet Akcin <mehmet@akcin.net>:

This Gem is fantastic by the way,

https://nsrc.org/workshops/2015/apricot2015/raw-attachment/wiki/Track1Agenda...

On Tue, Jun 4, 2019 at 5:57 AM Warren Kumari <warren@kumari.net> wrote:

...

On Mon, Jun 3, 2019 at 11:34 PM Brandon Martin <lists.nanog@monmotha.net> wrote:

...

On 6/3/19 9:56 AM, Jon Lewis wrote:

...

3) Don't advertise one transit provider's routes to another. Each

should

...

...

be filtering your routes, but you never know. Come up with, and

use

...

BGP communities to control route propagation. As you grow, it

sucks

...

having to update prefix-list filters in multiple places every time something changes...like a new customer with their own IPs.

To reiterate all this, FILTER EVERYTHING.

To start with, explicitly specify in a route-map or similar everything you want to advertise. I usually create a separate route-map for each transit/peer and include what I want to advertise via prefix lists (for my IP space) and/or communities (for downstream BGP-speaking customers if anticipated).

I think a related *principle* is: "Build everything as though you are expecting to scale."

This doesn't mean "spend lots of money to buy huge [routers|servers|commercial software|<etc>], but rather "when you plan your addressing structure and routing policies and monitoring and device config generation and... keep the in mind the question "If this suddenly takes off, and I hire N more people to run this, can I explain to them how it works? Do I have documentation I can point them at or is it stuck in my head / on the devices? If I need to add another M customers in the next month, can I do that easily?".

This is related to the FILTER EVERYTHING -- when you turn up a new customer / peer / transit / whatever, you shouldn't be sitting around trying to figure out how you will write their route-map / policy-options -- this leads to weird one-offs, and quick hacks. Instead you should have policies already largely designed and simply plug in their prefixes (or, better yet, use bgpq3 or similar to build and populate these). Obviously there will be some cases where a new connection does require some special handling, but that *should* just be a plugin/chain in an existing policy-statement. Related to this is how you end up naming things -- I recently found 9 variants of firewall-filters which basically do:

filter ACCEPT { term ACCEPT { then accept; } } named things like: ACCEPT, ACEPT, Accept, Allow, Permit_all, AcceptAll, dontdrop [0].

Obviously, there is a tension in the "design for scale" - while it would be great to design a complete automation system so that everything from installing a new customer to a new sites is simply typing 'make <thing>' and having everything pull from a database, at some point you will need to actually build a network, or you'll never have customers :-) Just keep in mind that "Am I building myself into a corner here?". E.g it only takes 10 or 15 minutes to install something like NetBox to keep track of addresses (and prefixes and racks and connections and ...) -- stuffing this in a spreadsheet might save you a few minutes *now*, but will this scale? Can $new_person easily figure it out?

W [0]: My personal favorite is: filter Accept_All { term Accept { then { count dropped; reject; } } term filter_<customer> { from { prefix-list { <customer>; } } then accept; } term NEXT { then log; } }

Presumably this all made sense to <name_removed_to_protect_inoccent> when they stuck it in at 3AM to deal with some crazy issue, but...

...

When you turn on the session, check what you're squawking AND WHAT YOU'RE FILTERING. You shouldn't be filtering anything you don't expect. Belt + suspenders.

The same goes for anything you accept. Obviously for a blended full transit BGP edge router, you're probably going to accept almost everything. But if you only want default + on-net, try to filter using communities from the peer, etc. Again, right when you turn on the session, "sh ip bgp ... filtered" of whatever's equivalent on your platform. If you're filtering something you don't expect to be receiving at all, figure out where the misunderstanding or misconfiguration lies.

And of course it goes without saying that, if you've got BGP speaking customers, you filter the heck out of them. Use ROAs and/or RPKI if you can to automatically generate filter lists. Encourage your upstreams to do the same if they're filtering you (and they probably are, or at least should be, if you're new). Remember that you are responsible for every route you advertise, at the end of the day, even if you only advertised it because a downstream network made a boo-boo and you didn't filter it.

Filters are useful on your IGP, too, but there's so many ways to set all that up that it's a bit more difficult to come up with nearly universal best practices. Generally speaking, be careful with redistribution, never distribute BGP into IGP or vice versa unless you have a really, really good reason to, and consider filters between both IGP areas/regions or protocols (e.g. RIP coming into OSPF) as well as on redistributions of static/connected to prevent simple typos on a static route or interface configuration from taking down more than just local stuff.

It's way, way easier to remove or relax filters later if they prove more of an operational hazard than asset than it is to add or tighten them if they prove insufficient. -- Brandon Martin

-- I don't think the execution is relevant when it was obviously a bad idea in the first place. This is like putting rabid weasels in your pants, and later expressing regret at having chosen those particular rabid weasels and that pair of pants. ---maf

https://startyourownisp.com/ this is also pretty good, just thought i would share.. On Mon, Jun 3, 2019 at 7:19 AM Mikael Abrahamsson <swmike@swm.pp.se> wrote:

On Mon, 3 Jun 2019, Mehmet Akcin wrote:

...

I am putting together a public DOs and DONTs blog post and would love to hear from those who have built ISPs and have recommendations from Billing to Interconnection, Routing policy to Out of the band & console setup, Software recommendations, etc. Bottom line is that I would like to publish a checklist with these recommendations which I hope will be useful for all.

The "ISP Essentials" publication is still valid in a lot of cases. It might not cover everything but gets a lot of the basics right.

ftp://ftp.securecomp.net/EBook/Cisco%AE%20ISP%20Essentials.pdf

-- Mikael Abrahamsson email: swmike@swm.pp.se