I'm looking for a cost-effective, highly reliable, high bandwidth, low latency solution for a link in Canada between Vancouver and Toronto.

Tall order.

I have a couple of ideas:

1) Droping some type of high-speed interface into the frame-relay cloud of a third party network like AT&T, Sprint, MetroNet, or whoever provides this type of service. I think this would definetly be a scalable solution as it would allow us to put POPs anywhere we like in between and it would be as easy as running a circuit back into the frame cloud.

You also should consider the availability of access to include proximity to the frame switch, comprising an access point into the frame cloud. If the nearest Frame Access point is 100 km away, your cost of access is likely significantly higher than if you are close to the switch. Some NSPs provide subsidization of the loop back to the L2/L3 NE point, but most do not.

The only issues with this is a) What are these networks like?

Generally they are comprised of a sparse mesh of FR/ATM switches, analagous to the backbones of most ISPs, w/ less NE redundancy, and more hops.

Are they reliable?

Relative to what? The uptime on most FR/ATM switches is probably much higher than most IP routers. The micro level reliability of the FR/ATM networks is typically 1 - 2 notches above those of typical IP routers. They are also typically less dynamic and more 'rigid'. The stdev on the quality tends to vary significantly, as BB issues on the L2 FR NSP cause PVCs to reroute, introducing step-wise latency/loss changes. From my experience, these seem to be much less pronounced on native IP networks, than on IP connectivity via the 3rd party L2 FR/ATM provider.

Are they fast?

Most L2 FR networks are at DS3 speeds. Most ATM neworks are hybrid DS3/OC-12. To the best of my knowledge noone has yet deployed OC-48. Many/most L2 providers [ATM/FR] offer FR and ATM to customers, and use ATM as a backbone transport mechanism for FR. This is done by 'emulating/transporting' FR PVCs over ATM PVCs.

Is their a risk of them not being able to provide us with the bandwidth we paid for without performance loss due to lack of capacity (QoS)?

Yes. It is common for the L2 FR/ATM NSP to oversubscribe their network, above and beyond that 'promised' w/ SCR/PCR/CIR contracts.

2) Assuming we buy our own fibre:

It is unlikely you would purchase your own glass between cities, the cost would most likely be many millions of dollars $US. This presents an interesting market system, where the supply is quite controlled by folks who have bought up Rights-Of-Way [ROW]. The ROW tends to be tremendously expensive, due to the expanse of distance, tremendous growth in DWDM technology==high value of ROW, and municipalitys being reticent to allow carriers to dig up streets. I'll assume for info below that you mean purchase DIM capacity or purchase SONET circuits from point A to Z.

Do we run a direct link between Toronto and Vancouver directly or drop a POP in between; somewhere in Saskatchewan or Manitoba and running fibre out of there to each?

If you have business only in the cities at the end, [Toronto and Vancouver] then putting a POP in the midle is unlikely to help you, as it just adds a point of potential problems. If you have business in the middle cities, or semi-uniformly distributed across the country, building POPs across the country is a good thing. As you consider building a densely meshed network, you are likely to find that putting an L3 POP in the middle of the networks allows you to make a 'T' point or 'intersection' point where you can bring in additional bandwidth to the backbone fabric. This will allow you to cut costs.

I may like the direct route from A to B but how pricy is that and how scalable is that?

Typical circuits are measured in '$/DS0s/mile' A T3 is 28*24 DS0s, typical pricing is in the ballpark of ~2 cents/ds0 mile, or the ballpark of $10/mile for a DS3, plus a 'termination' charge for each node intersecting the circuit. Typically adding nodes to the circuit does not significantly increase the cost; ie; adding manitoba or winchester does not change your per-mile costing, only add a relatively small fixed cost to the monthly circuit charge.

If we plan do put POPs into some some cities along the way we'd have to run pipes from those POPs back into either Toronto of Vancouver.

Right.

If we go the route of A to B to C that would probably be a bit smarter as the haul from POP to POP wouldn't be so large.

Right.

Is it common practice for the National ISP to lay their own pipe or is it more common practice to obtain the link from a third party?

It is most common for them to be purchased and use their parent company's fiber :-) I would dichotomize NSPs into 2 categories - facilities based and non facilities based. Facilities based NSPs own their own fiber, and usually have 75-100% of the backbone circuits on bandwidth owned by their company. Non Facilities based NSPs by definition obtain their bandwidth from a third-party. I would estimate that 95% of NSPs are non-facilities based. I would estimate that 70-90% of 'Internet Traffic' in the US travels over facilities based NSPs.

How do the UUNets, and BBNPlanets do it?

You'll have to ask them. Most people would agree that the current economic models forbid an NSP from being successful on their own long term as a nationwide NSP unless they are facilities based.

They are not telcos so laying a pipe is not as easily justified.

Your two examples actually are facilities based, UUNET=WCOM; BBNPlanet=GTE.

Do they buy fibre between A and B or do they even go through some sort of telco-like carrier?!

Both. And for additional confusion, most (facility based) carriers enjoy incestuous fiber swaps and cross-leasing of capacity.

If anyone has any ideas, opinions, experiences, or something to sell me I'd love to hear from you!

I've got lots of things to sell you, but this is not the forum :-) I hope this perspective helps you and perhaps others. -alan