[ga] Costs of running necessary parts of DNS

[Karl Auerbach <karl@CaveBear.com>]

Everybody talks about DNS stability but nobody defines it.

I assert that with respect to ICANN, that DNS stability means that its
system of root servers operates with acceptable reliability, acceptable
reachibility, and with acceptibly accurate contents.  In addition, DNS
stability means that updates to the information in the root zone and meta
information about the root zone (contact information) is updated
reasonably quickly and for a reasonable cost, taking into account the need
to properly validate the authenticity of the update.

I leave it open to debate whether technical stability requires that ICANN
create minimal operating standards, such as data escrow formats, annual
audits for adequate preservation of business assets, etc for the DNS
layers beneath the root.  That raises the basic question of to what degree
ICANN is a consumer protection body.

(Notice that I did not include trademark concerns - I categorically reject
the assertion that such concerns are part of technical stability.)

Getting back to the root layer -

About two months ago I undertook to make a quick (one day) inquiry how
much it would cost to establish an entirely new set of root servers,
deploy them around the world, and maintain them.  (I did not include the
cost of root zone file maintaince itself.)

The people I worked with on this have many, many years of experience in
establishing and running 24x7 high availability/high volume services all
over the world.  These are people who I consider the professional's
professionals both in terms of servers and in terms of networking.

We defined a basic server-group consisting of 8 distinct servers (running
4 different operating systems to provide common-mode attack resistance)
sitting behind redundant load balancers/firewalls, with remote control of
things such as power and console access.  The design was such that each of
these server-groups could be expanded manyfold simply by adding more
individual servers.  And individual servers in each server-group could be
removed (for maintainance or replacement) or added without taking down the
entire server-group.  The basic server-group would fit in less than 1/2 of
a standard 19" rack making it nice for fork-lift installation as a unit.

The whole server-group is designed to be controlled from afar - including
taking individual servers offline, working on their software, and power
cycling them - via secure communications.

We made the assumption (one based on practical experience) that there are
high-quality co-lo's with excellent connectivity and acceptable physical
security and environmental (power, temperatature, humidy) systems.  Also
based on practical experience, we assumed that we could hire local
"trusted hands" to do those few jobs that actually required physical
access to th equipment.

We also assumed that monitoring and management would be performed from
existing management centers by existing personnel, thus incurring only
incremental costs.

What does it all cost:

The server-groups ran to between $35,000 and $50,000 each to build.
These are one-time costs.

(No we didn't figure in the life expectency of the equipment and build a
depreciation/replacement fund - a five year period isn't unreasonable, but
it could probably be rather longer and still be quite realistic.  Given
the architecture of these server-groups, they could evolve over time
without the need for a fork-lift [replace the entire thing] upgrade.)

Installation varied widely and consisted mainly of shipping from a central
construction site (we've had several years positive experience with this
build-wrap-and-ship technique), plus either hiring on-site "trusted hands"
or going there and doing it.  Obviously the cost range for this would vary
widely depending on location.  But it was a non-trivial expense amounting
to at least several $1,000s of per server cluster.

So we're talking about something on the order of $750,000 to $1,000,000
one time cost to establish 12 new root server-groups.  We actually wanted
to go further and create any-cast clusters to produce 96 server-groups
with 8 server-groups for each of the 12 IP addresses that are assigned to
the root servers (one address per logical server.)

In terms of recurring costs, there are several components:

Co-lo costs were low.  We estimated high at $1000/per server-group per
month.  In other words about $150,000/year.

Bandwidth costs could potential be rather higher.  There are contradictory
reports of the relative bandwidth of the incoming and outgoing DNS
streams.  We made the worst-case assumption, that outgoing bandwidth would
be higher than incoming, and concluded that co-lo payments for bandwidth
could amount to more than $1,000/month per site.  This is a highly
variable cost component and is subject to negotition with the co-lo
operators.

(The bandwidth to transmit root zone files for update purposes is
negligable - a compressed version of a root zone file is about 15K bytes -
smaller than the gif/jpg buttons on many web-pages.)

Day-to-day management and operations costs were very low - remember we
excluded the costs of maintaining, preparing, and triple-checking root
zone file contents and the dissemination (and subsequent triple checking
by the recipient) - and amounted to essentially a web of automated, low
cost "is it running at all", "is it reachable from diverse locations", and
"is it returning sane answers in a timely manner" polling.  But we decided
to be very conservative and threw in several FTEs (full time employee).

Because we assumed incremental use of existing management/operations
centers, the incremental cost for building/office space was low - nearly
negligable compared to the cost of the FTE's.  (Because we were looking at
overloading an exisiting system, we obtained an automatic benefit of
multiple sites around the globe for management/operations.)

We estimated with a broad brush that mangement/operation costs would be on
the order of between $500,000 to $1,000,000 year.  (We figured that much
of the time the FTEs that gave rise to this expense would be twiddling
their thumbs.)

On-site costs for "trusted hands" was assumed to be fairly rare and we
estimated that $1000 per month per server-group would cover it.  In other
words for 12 server-groups we figured about $150,000 per year.

One more thing, we figured on having enough liquid assets on hand that we
could cover emergency disaster recovery situations.  Since we were working
with the context of an existing worldwide enterprise (you would all
recognize the name, but I'm not going to say who), this financial
resiliancy wasn't a big issue.

>From a political point of view, we knew that this single-provider plan
would have a hard time.  But we did the exercise in order to get some
notion of the concrete costs of undertaking to provide DNS root
infrastructures.

Obviously much cost efficiency in these estimates was gained by making use
of existing management/operation infrastructures and by using high-quality
co-los rather than constructing special purpose centers and hiring
dedicated operations staff.

And as I mentioned several times, we did not deal with the costs of
maintaining a root zone file.  But our gut feelings were that that is a
job requiring much less than one FTE.

Obviously there are many "guestimates" here, but they are educated ones.
There are probably items we forgot - this was a very informal effort done
very quickly.

This is what DNS technical "stability" is all about.  I started talking
about this stuff from my first day on ICANN's board.  I've met stone walls
that were more responsive.  (And you won't find any of my materials on
this on ICANN's web site, ICANN's management has apparently chosen a
"non-publish" policy with respect to anything I write, even though it
routinely publishes even the most banal of statements from other Directors
or "staff".)

		--karl--


--
This message was passed to you via the ga@dnso.org list.
Send mail to majordomo@dnso.org to unsubscribe
("unsubscribe ga" in the body of the message).
Archives at http://www.dnso.org/archives.html