This was my experience with Arrowpoint:
(BTW, we ended up abandoning the Architecture outlined below. It was
originally designed with certain goals in mind and decided to go another
route.)
###########################INTERNAL MEMO ####################
Date: 29 December 1999
To: #########, President
#########, Sr. Vice President Technology
From: ##########, Chief Systems Architect
RE: Arrowpoint Communications CS100 / CS800 Solution
The review and testing of the Arrowpoint Communication products CS100 and
CS800 as possible solutions for the CompanyXYZ distributed virtual hosting
platform has been concluded and this product line has been determined to be
insufficient to our defined goals. This conclusion is based on development
testing, production service, information provided by the upper levels of
technical support & senior development staff, and Arrowpoint defined
development timelines. Additional facts pertaining to Arrowpoint’s sales
practices are referenced here for consideration in future product
assessments.
Architectural Goals for Product
Outline of SystemXYZ Architectural Goals as initially provided to Arrowpoint
Communications. These goals were repeated in their entirety to various
technicians, developers, and technical managers:
Two clusters (one unix, one NT) initially served by one CS100
device. Each Cluster will initially retain two servers but should be able
to scale up to at least 16 servers per cluster. Each Cluster is expected to
support 2000 websites for a total 4000 websites.
Due to the unique architecture of the platform, each server will
maintain unique bindings for each website instance. DNS portrayal to the
Internet will be propogated as one (1) single IP to represent each cluster.
The Arrowpoint device will intercept the intended address and based on layer
4 HTTP/1.1 compliant headers will determine redistribution based on the
‘Host’ key-value pair of the HTTP header. The redistribution rules will
utilize NAT to redirect request to local IP:Port binding unique to each
website/server instance.
Rule utilization is as follows:
2 Clusters x 2000 sites = 4000 HTTP Services
2 Clusters x 2 Servers x 2000 IP:Ports = 8000 rules
Persistance methods: Load (primary), cookie (on request)
Expected Throughput for both clusters: 8mbps to 3gbps
Product Claims
White papers made available online for full release versions and beta
versions detailed that expectations were well within defined product
guidelines. These technical outlines were reinforced with product
documentation upon delivery. Discussions with the attending salesman and
sales engineers confirmed that the product could comfortably achieve the
defined goals. Based upon discussions regarding development cycles and
product extensibility, it was determined that product testing was desirable.
Testing Results
Testing and review consisted of several phases including manageability,
resource utilization, network throughput, reliability, and scalability of
two versions of full release software and several beta versions.
Manageability: All CLI based management is straightforward. Configuration
is for the most part logical and heirarchal. Later in testing, it became
apparent that some features needed some massaging. Stalling in scrolling
through large configurations a know bug being addressed in version to be
release in mid-2000 using “collapse-able” configurations.
Resource Utilization: Concerns first arose when 40 sites (20 per cluster x
2 clusters) were configured and tested. Utilization climbed and stayed at
38% CPU utilization. This test included NAT, round robin, and approximately
35mbps of traffic. Future testing would indicate that CPU utilization rose
in conjunction with rule sets and load. Addressing this issue became the
focus of the next four months of testing, product revisions, and major
concerns.
During the following months, various concessions were made to the SystemXYZ
platform to accommodate redefinitions of the CS100 resource availability.
Despite that Arrowpoint sales and sales engineers identified certain ranges
that could be expanded to accommodate larger rule sets than those defined
with in the technical specification papers, it quickly became apparent that
the 5000 rule sets defined in the white sheets was an inflated number.
Apparently, modifications in the software that added functionality also
introduced additional memory requirements and these were not reflected in
the white sheets. Over a period of six months, we went from assurances that
the CS100 product could accommodate 5000+ rule sets, down to the major
concerns of Arrowpoint offering to give us three additional CS100’s to
support a total of 250 rule sets after determining that the product could
reliably support 250 ruleset at that level. Additionally, under this
configuration, certain methods of determining service status were no longer
available. This meant that service status was only determined via ICMP
calls which can indicate whether a server is available on a network, but
which are poor indicators of a running WWW server. A review of future code
revisions indicated that by MID-2000, a partial solution would probably be
made available.
A review of the CS800 solution offers an impractical remedy for the resource
demands. The CS800 offers only a 100% to 200% increase in processing
capabilities due to the software architecture’s memory demands.
Explorations of memory expansion capabilities with Arrowpoint development
members resulted in the consensus that the boards they base the architecture
on could not physically accommodate additional memory that would be required
to meet. The CS800 uses up to two of these same boards of up to 256MB RAM
each. This revelation indicated that our concept of the CS800 as an upgrade
path was invalid since all experience with the product was indicating that
512MB would be required to run 250 services with 2 ruleset each.
Network Throughput: As a matter of course, two moderate volume Anonymous
FTP servers where introduced to the CS100. The two servers only serviced
FTP and telnet access (for maintenance). The CS100 was configured for 20
basic WWW / Round robin rulesets for use by SystemXYZ platform developers
and the handfull of rulesets required to support the two ftp servers. The
FTP servers were returns to standard TCP/IP Round Robin DNS outside of the
CS100 environment after several code revisions, a full release code, and
several undefined CS100 device crashes resulting in either no restoration of
services or partial restoration of service. Through two full releases and
various in-between beta releases, a bug was apparently allowed to persist
that did not release closed sessions which resulted in up to 250 stale ftp
sessions that only cleared on power cycling (but apparently not in
crashing). The claim for 5GBPS on the switch fabric was never achieved due
to our own resource limitations in benchmarking. However, 35mbps of WWW
traffic did test successfully using the resources defined above.
Reliability: Throughout the six month review and testing of the CS100
product various codes, both full releases and betas, were explored. Beta
code was not considered when regarding the stability of the product with the
exception of understanding Arrowpoint's pursuit of addressing any ongoing
stability issues in release versions. The first release version that was
provided was stable. The beta versions of the following release were
understandably unstable, but did not apparently address an “undefined”
crashing problem that continued to occur in the full release. It has been
the consensus that some type of memory leak occurs that causes the crashing,
however, up to the time of our decision to pursue other solutions; the issue
of the device crashing with or without load had not been resolved.
Scalability: Due to the problems with other facets of the Arrowpoint
product, scalability testing was never reached.
Product Development Goals
Arrowpoint personnel are courteous professionals who devote a great deal of
time to seeking acceptable solutions. The level of this devotion was one of
the reasons that attracted us to Arrowpoint and kept us pursuing the product
for six months. Arrowpoint had indicated their willingness to incorporate
enhancements to their product based on feedback from SystemXYZ and there is
no doubt that Arrowpoint would have accommodated us to the limit of their
hardware. Discussions with Senior Developers of the Arrowpoint product line
outlined their development cycle for the next year which, unfortunately, did
not include utilizing or exploring solutions to the memory barrier that was
physically limited by the circuit boards they used for the CS100, CS150, and
CS800 products. Additionally, existing memory would be consumed by new
functions that were being added to the code.
Summary Conclusion
Despite both Arrowpoint’s and SystemXYZ’s best efforts, the Arrowpoint
product line is not well suited to the SystemXYZ virtual webhosting platform
or for recommendation to CompanyXYZ clients interested in load balancing
solutions. It may come to pass that Arrowpoint will find a solution to
their physical memory limitations beyond their currently defined development
cycle that will result in their ability to support the number of rulesets
outlined in their white sheets. As of this writing, the supportable rule
sets sharply dropped from 5,000 on one box to 250 across four boxes.
Although Arrowpoint offered to provide the three additional boxes at no
additional fee, subsequent box requirements would be associated with a fee,
not too mention the cost of the rackspace for the additional devices.
Additionally, coordination of the management of four devices to service 250
sites would introduce it’s own instability factors.
The current combination of the additional inferred costs, loss in critical
supported feature sets, and the projected development cycle that does not
include resolution to memory constraints outweighs any prospects for
continuing to pursue the Arrowpoint products as a practical solution. Once
the product has survived its infancy, the feature set that Arrowpoint has
outlined will certainly be of interest to many hosting companies.
As for recommendations to clients, there are a wide variety of load
balancing platforms that offer minimal feature sets that provide acceptable
load balancing features at a much lower pricepoint and with reputations for
stability, at minimum.
