Esoteric Curio - Comments

Theo Schlossnagle: Scalability and concessions

nospam@example.com (Theo Schlossnagle) — Thu, 03 Jul 2008 23:19:08 -0400

Dan, well played. You make my point stick. Those are all excellent questions. And clearly the "one large database" has a very different usage pattern than the horizontally partitioned one. My point in all of this: ask the question before jumping to the answer. For datapoints, the "one big database" averages about 6k transactions/second and peaks around 20k/second. Very little batched -- that's done on the "one big datawarehouse". As for the horizontally partitioned one... I'd say less transaction right now, but the growth predictions led us down the path of a partitioned approach.

Dan Creswell: Scalability and concessions

nospam@example.com (Dan Creswell) — Thu, 03 Jul 2008 14:13:26 -0400

Mmmm, so you have some interesting raw data - wouldn't mind a little bit more if you're game: ...."[You] Can't use just one database" -- this is a conclusion you should arrive at after analysis. We have one client that supports 10 million users on a cluster of partitioned databases. We have another that supports 35 millions users on one database without issue and room for growth..... How many operations are being performed concurrently by each sets of users? Are the loads write intensive or read intensive? How much data do these operations typically touch? .....My bank supports many more users and transactions than LinkedIn -- and it damn well better have FKs and 100% integrity. So, while you still may partition in such a fashion that requires a loss of enforced integrity, the decision should be a heavy one..... What's your definition for a transaction? Are these transactions processed in real-time? Do you batch them? Are the loads write intensive or read intensive? How much data do these operations typically touch?

Jannis: Scalability and concessions

nospam@example.com (Jannis) — Thu, 03 Jul 2008 07:45:21 -0400

Thank you, that last sentence made your point very clear

Theo Schlossnagle: Reconnoiter and another platform

nospam@example.com (Theo Schlossnagle) — Wed, 02 Jul 2008 10:55:16 -0400

I looked a little bit. Their tool looks to have a lot of differences in approach. None of the differences are wrong and their approach seems pretty well thought out. We choose to use an RDBMS. This is not as efficient (but scales with partitioning easily), but allows for complex querying of data. The distributed, disconnected data collection was foundational for Reconnoiter with a strong approach toward performance. Additionally, the front-end of reconnoiter is designed to be highly interactive and simple to use -- I didn't look into the orbitz stuff enough to know that they don't do this. Their stuff looks really nice.

ashbhai: Reconnoiter and another platform

nospam@example.com (ashbhai) — Tue, 01 Jul 2008 23:20:54 -0400

Theo, How different is Reconnoiter from the monitoring tools released by Orbitz? Orbitz Open Sources Monitoring Tools ERMA and Graphite http://www.infoq.com/news/2008/06/orbitz-opensource-erma

ian Holsman: Reconnoiter and another platform

nospam@example.com (ian Holsman) — Tue, 01 Jul 2008 20:44:29 -0400

Looks good Theo. you might want to investigate some stream processing tools like http://esper.codehaus.org/ to help you with anomaly detection

Theo Schlossnagle: Dissecting today's Internet traffic spikes

nospam@example.com (Theo Schlossnagle) — Tue, 01 Jul 2008 08:59:04 -0400

In response: 1. That was actually the point. The old rule of thumb is to maintain about 70% utilization and I'm saying it is time to rethink that paradigm. You echo my conclusion with using large instances with low utilization "just in case." However, if other VMs on the same hardware spike at the same time, you are limited due to over saturation of the underlying hardware. Heavy use machines typically use all of the hardware anyway. 2. It takes us far less than 60 minutes to bring online new systems from bare-metal, app install, test and integrate into the load balancer. So, it's really the first few minutes I'm interested in solving, the rest is canned. I think elastic computing make sense the smaller you are. If I have 4 hosts, the ebb and flow is more dramatic and clouds can help a lot. When I already have more than a few racks of servers, I have my own elastic computing if I designed the infrastructure correctly. The premium on fully taxed VMs is pretty significant on the wallet.

ElasticHosts: Dissecting today's Internet traffic spikes

nospam@example.com (ElasticHosts) — Tue, 01 Jul 2008 08:05:46 -0400

Theo identifies a real issue, but I believe cloud computing systems do provide a little more help than he gives them credit for. A user of a cloud service such as ElasticHosts or the others has two fallbacks: 1) For the initial response to the spike, the user can indeed set up with "warm" capacity - and this works very well in a cloud computing environment. e.g. they can size their virtual servers for large CPU/RAM with a low base utilization - while they have low usage the hypervisor will share capacity with other VMs, but it is instantly available when the spike hits. 2) Although the spike hits instantly, the higher level of traffic lasts for hours afterwards. With a good monitoring and a good cloud vendor, the user will be aware and able to provision additional capacity within the first hour. This doesn't tackle the initial part of the spike (handled above), but does mean that many of the NYT/Digg readers will actually get a great experience when they arrive later in the day.

Theo Schlossnagle: Dissecting today's Internet traffic spikes

nospam@example.com (Theo Schlossnagle) — Mon, 30 Jun 2008 12:23:15 -0400

We use Reconnoiter to trend most of this stuff. http://labs.omniti.com/trac/reconnoiter

Peter: Dissecting today's Internet traffic spikes

nospam@example.com (Peter) — Mon, 30 Jun 2008 08:18:07 -0400

Not about traffic spikes per se, but how are you rendering the graphs? They're pretty gorgeous

Andrew Binstock: Dissecting today's Internet traffic spikes

nospam@example.com (Andrew Binstock) — Sun, 29 Jun 2008 22:38:54 -0400

The quotation about premature optimization being the root of all evil that you attribute to Knuth was actually first said by C. A. R. Hoare (see: en.wikipedia.org/wiki/C._A._R._Hoare). Knuth was quoting him.

Chris Parente: Dissecting today's Internet traffic spikes

nospam@example.com (Chris Parente) — Sun, 29 Jun 2008 16:37:05 -0400

Very interesting post. A couple of weeks ago there was a survey that shed some light on what carriers are doing to control spikes -- and that could affect net neutrality: http://cparente.wordpress.com/2008/06/21/will-the-internet-break-under-peak-load/

Theo Schlossnagle: Dissecting today's Internet traffic spikes

nospam@example.com (Theo Schlossnagle) — Sun, 29 Jun 2008 16:35:31 -0400

Well, Times was an article that included links to the site. The article was one level off the home page (it was a section page). So, there you see the level increase and stay increased for about 20 hours or so (with some die down due to Times traffic behaviour). The Digg stuff, has a much shorter time frame. It is the life of an entry on Digg on the front-page. Anywhere from a few minutes to a few hours, but rarely longer than that.

PR NY: Dissecting today's Internet traffic spikes

nospam@example.com (PR NY) — Sun, 29 Jun 2008 10:02:53 -0400

What were the events that caused those extreme spikes (Times & Digg) on that graph? Why did they only last for only such an extreme short period of time?

Jan: Dissecting today's Internet traffic spikes

nospam@example.com (Jan) — Mon, 23 Jun 2008 17:43:25 -0400

What I always find interesting is how much of the traffic "sticks". We do TV advertising and have huge spikes within 3-5 seconds of the spot running. After maybe 20-30 seconds everything is back to normal again, with maybe +5-10% traffic (a normal variation).