Why Twitter dedicated servers for Justin Bieber, 3% of the load

There are 247 news articles about Twitter having dedicated servers for Justin Bieber which is 3% of Twitter traffic.

image

CNET broke the news with citing a tweet.

Report: Justin Bieber is 3 percent of Twitter

by Chris Matyszczyk

If you believe that Twitter is full of inane, immature narcissism, here's one in your solar plexus.

For an allegation has reached my eyes and baffled them into blindness. The allegation is that, at any given moment, at any given movement of your lungs and toes, 3 percent of Twitter's infrastructure is dedicated solely to the one person who most defines our hopes and our times.

I am not speaking of Kim Kardashian, nor of Rep. Jack Kimble. I am speaking of the one person who can unite men and women, young and old, sane and slightly less so: Justin Bieber.

A fascinatingly hopeful post at Gizmodo offers that Dustin Curtis, a designer of some repute, was told by an employee at Twitter that 3 percent of the company's infrastructure is dedicated to the little man with the unreal voice and the even more unreal hair.

(Credit: Screenshot: Chris Matyszczyk/CNET)

Read more: http://news.cnet.com/8301-17852_3-20015781-71.html#ixzz0ywhpUh60

What I haven't seen anyone attempt to explain is why would there be Racks of Servers dedicated?

This is common sense with no information from Twitter folks.

When you want speed in your servers you want to keep your whole search index in RAM in a server. Google, eBay, and Amazon are good examples of companies who use this method in their search servers to support keyword lookup and result match-up.  Any time you go to disk or another server you will slow down the search results by magnitudes.

The costs to put your whole search index in RAM is too high, so break down the problem.  Analyzing the flow of tweets, Twitter figured out many people are focused on a few areas, and probably stick to that topic watching the tweets.

Second Life was started with the idea of virtual reality and avatars, but many people are too lazy to keep up their 3D appearance.  Tweets are much more efficient to support a virtual presence.

In the same way that Saudi Arabia wanted to get access to RIM's encrypted data the information and analytics available in Justin Bieber's servers can be useful.  And, Twitter is probably figuring out ways to monetize access to analytics.

Which brings up an interesting point is Twitter the next service which governments want access to crawl?  For many people, e-mail and blogging is in the past.  And, Twitter is their main method to communicate.

There was a rumor of part of the reason why Twitter chose Salt Lake City is to be near the NSA's new data center in Salt Lake City. 

Room to grow: a Twitter data center

Wednesday, July 21, 2010

Later this year, Twitter is moving our technical operations infrastructure into a new, custom-built data center in the Salt Lake City area. We're excited about the move for several reasons.
First, Twitter's user base has continued to grow steadily in 2010, with over 300,000 people a day signing up for new accounts on an average day. Keeping pace with these users and their Twitter activity presents some unique and complex engineering challenges (as John Adams, our lead engineer for application services, noted in a speech last month at the O'Reilly Velocity conference). Having dedicated data centers will give us more capacity to accommodate this growth in users and activity on Twitter.
Second, Twitter will have full control over network and systems configuration, with a much larger footprint in a building designed specifically around our unique power and cooling needs. Twitter will be able to define and manage to a finer grained SLA on the service as we are managing and monitoring at all layers. The data center will house a mixed-vendor environment for servers running open source OS and applications.
Importantly, having our own data center will give us the flexibility to more quickly make adjustments as our infrastructure needs change.
Finally, Twitter's custom data center is built for high availability and redundancy in our network and systems infrastructure. This first Twitter managed data center is being designed with a multi-homed network solution for greater reliability and capacity. We will continue to work with NTT America to operate our current footprint, and plan to bring additional Twitter managed data centers online over the next 24 months.

The probability of the NSA talking to Twitter at some point of time is almost certainty.

Is Twitter's dedicated servers a help for the NSA?  There are Twitter servers dedicated to President Obama.  And Oprah too. :-)  Are the number of dedicated servers the new status symbol and measure of popularity?  It used to be records sold and movie revenue.  Is it now the billions of tweets?