Twitter lives

June 24, 2008 8:24 PM

The rumors are true. Outage-plagued Twitter really has been raising money and today announced that they'd closed their funding round. Venture firm Spark Capital doled out some cash as did Amazon.com CEO Jeff Bezos. Twitter, however, ain't saying how much money was raised. (It's reportedly around $15 million.) CNET helpfully points out how the press release from Twitter that announced the news went out of it's way to mention that the company will -- hold the presses -- someday be supported by an actual revenue model: "To reach our goal, Twitter must be reliable and robust. Private funding gives us the runway we need to stay focused on the infrastructure that will help our business take flight."
posted by not_the_water (81 comments total)

Or one could be on the google backed jaiku.
posted by BrotherCaine at 8:43 PM on June 24

*bet*
posted by BrotherCaine at 8:43 PM on June 24

Eating soup? Research shows that moms want to know.
posted by nola at 8:43 PM on June 24

To reach our goal, Twitter must be reliable and robust.

Against my own expectations mere months ago, I have grown to heart the Twitter -- nay, be thoroughly addicted to it, but goddamn, guys, buy some servers or something. Your site goes down more often than your mom.
posted by middleclasstool at 8:45 PM on June 24

The problems of scaling with Ruby on Rails.

I think they are the largest project using Rails.
posted by bhnyc at 8:54 PM on June 24

I like soup.
posted by Burhanistan at 8:56 PM on June 24 [1 favorite]

I don't make any money either... can I have $15M?
posted by pompomtom at 8:59 PM on June 24 [7 favorites]

What, exactly, is Twitter's business model? Provide a free service, and then not sell advertising?
posted by Electrius at 9:04 PM on June 24

Their problems have absolutely nothing to do with Rails, and they are nowhere near the largest project using Rails. (I say this as a detractor of the Rails model, and I think Ruby has some serious problems)

Their problems stem almost solely from having a system with the architecture of a one-off blog site — a system that is now expected to be a messaging system supporting hundreds of thousands of concurrent users.

They built a multiuser blog site that is expected to behave like a persistent Jabber server.
posted by blasdelf at 9:04 PM on June 24 [2 favorites]

I think they are the largest project using Rails.

Easy, now.

Actually, yellowpages.com and scribd.com have more traffic with less downtime. The word on the street is that Twitter's problems lie within it's XMPP messaging and some crazy MySql replication (or lack thereof).
posted by cdmwebs at 9:14 PM on June 24

How lazy are programmers these days? It's kind of unreal, I mean IRC and email servers have had the ability to process the kind of message loads that twitter does decades ago. And twitter is such a simple system, it seems like something a good programmer could sit down and write in a couple of weeks, including the ability to handle millions of messages a day on modern computers.

And these guys need millions of dollars of hardware, or millions of dollars to rewrite their code? Insane.

Why doesn't someone write a p2p version of twitter that lets people host their own identity online, which would distribute messages between servers. Maybe we could call it "Simple Message Transfer Protocol" or something.
posted by delmoi at 9:16 PM on June 24 [10 favorites]

i think twitter is just a ploy to see how gullible VCs can be. clearly, it has no other purpose.
posted by klanawa at 9:28 PM on June 24

Private funding gives us the runway we need to stay focused on the infrastructure that will help our business take flight.

Someone is paid a lot of money to write that sentence. Think about that.
posted by smackfu at 9:46 PM on June 24 [4 favorites]

and they are nowhere near the largest project using Rails

Easy, now.

huh? that link says they are about the 2nd biggest rails site

I like rails. i program in it. twitter is one of the biggest rails sites. it has scaling problems due partly due to ruby on rails. this isn't a secret. i'm sure it also has other architectural problems.
posted by bhnyc at 9:48 PM on June 24

Venture firm Spark Capital doled out some cash ...

Spark Capital is here in Boston (founded by Todd Dagres [1 | 2] and Dennis Miller [1 | 2] ).

They've been one of the very few VC firms which are making 'early stage' investments in consumer-based web companies.

Their other investments.
posted by ericb at 9:49 PM on June 24

I just don't get Twitter.

I mean, honestly...

What are you doing? I'm sitting on the toilet squeezing out a great big turd. Happy now?



/sheesh



Mephisto
posted by Mephisto at 9:50 PM on June 24 [4 favorites]

it seems like something a good programmer could sit down and write in a couple of weeks, including the ability to handle millions of messages a day on modern computers.

Things always seem simple to those who don't have to worry about them; just because a problem has been solved doesn't mean it's effortless to implement.

Two weeks and one person is not enough time to design, implement, test, fix, test, and release a solid, 1.0-quality product used by hundreds of thousands (current) or millions (target) of people. It's pretty arrogant to think otherwise, and it shows little understanding of the amount of effort that goes into good software.

millions of dollars to rewrite their code? Insane.

It's not "millions to rewrite code", it's "millions to stay afloat while there is no revenue stream". $15 million for a company in the Valley—one without a real revenue model, mind you, other than private funding—over the course of a couple of years is basically nothing. Good developers, IT support personnel, managers, office space, hardware, and even counsel expenditures . . . this stuff adds up quickly. And a good CEO will make sure the company has more than a single year of operating funds when there's no incoming cash other than VC bank.
posted by Mikey-San at 9:53 PM on June 24 [2 favorites]

I'm typing a comment but watching it display down in live preview instead of the text window where the cursor is.
posted by Burhanistan at 9:54 PM on June 24

Note that I'm not saying Twitter didn't fuck up when they first designed the system. They clearly made some bad decisions at the starting line. I'm also not making any statement about Twitter's usefulness, either. But it's too easy to take shots at the challenges they face without stopping to attempt to understand them first.
posted by Mikey-San at 9:56 PM on June 24

It feels like their runway has a conveyor belt on it.
posted by maxwelton at 10:06 PM on June 24

It feels like their runway has a conveyor belt on it.

Given that, will those little birdies ever get the whale off the conveyor?
posted by Mikey-San at 10:12 PM on June 24

I, too, have grown to like Twitter, although I like the concept more than the execution. There's no reason why something so simple should be so top-heavy, failure-prone and unreliable.

Pretty much everything you can do with Twitter you could do with XMPP presence notifications, and you'd get the benefits of decentralization. You could easily have lots of interoperable Twitter-like services, using regular [e-mail] addressing, and none would be affected by a failure in another except that the statuses for those users wouldn't update.

What Twitter does offer, and what really makes it a usable service, is a very easy-to-use interface. It really shouldn't be that hard to set up something similar that aggregates the status messages from XMPP or SMTP and provides an HTTP API, but nobody ever did or to my knowledge has. It's not a challenging or interesting enough problem to attract programmers working on their own time, and there's not an obvious enough revenue stream to attract commercial interest.

And thus we're stuck with Twitter, at least for the moment.
posted by Kadin2048 at 10:14 PM on June 24

And thus we're stuck with Twitter, at least for the moment.

It's not something ubiquitous like garbage pickup or EMS/911. It's not something that we are "stuck" with.
posted by Burhanistan at 10:27 PM on June 24

Twitter is yet another way people fail to come to grips with the fact that the universe really doesn't care about what you're doing right now.
posted by Justinian at 10:34 PM on June 24 [11 favorites]

/removes Mephisto's ENTER key and points out to him that it's very bizarre that after so many posts in the last hour he still doesn't realize he doesn't need to sign his own posts.

dobbs
posted by dobbs at 10:41 PM on June 24 [1 favorite]

Applications infrastructure and the software architecture in it is generally only 1/4 of a given infrastructures uptime. Equally important is the actual infrastructure itself - network, storage, systems and the operational procedures around the infrastructure. You can have a fantastically stable infrastructure and code base but be very low on the operational maturity model and no one would even know about your infrastructure.

Twitter is an interesting product and presents some unique engineering challenges, people flinging turds at folks attempting to provide a very downtime visible HA application usually have no clue as to the level of effort and costs associated with providing the service.

There is a distinct difference in the technical challenges in engineering a specific service and operating/growing that service in a manner that is responsive and that scales.
posted by iamabot at 10:51 PM on June 24

Twitter is yet another way people fail to come to grips with the fact that the universe really doesn't care about what you're doing right now.

I just favorited that comment RIGHT NOW.
posted by Blazecock Pileon at 11:08 PM on June 24

some unique engineering challenges

So, maybe you could tell us exactly what those "unique engineering challenges" are, and how they differ from, say, Facebook's status/precence handling or Google's Gmail, which both have about 50 times as many users.
posted by effbot at 11:14 PM on June 24 [1 favorite]

I'm thinking about favouriting Blazecock Pileon's message, but perhaps I'm just not hip enough for this realetime minutiae thing.
posted by pompomtom at 11:21 PM on June 24

(And I wrote that comment right at that time, which explains the lack of proof-reading, which I did right now. To no avail.)
posted by effbot at 11:33 PM on June 24

us exactly what

Maybe you could tell "us exactly what" you expect to get as an answer to an inane question ?

Was this it ? Perhaps you could highlight what challenges your two referenced organizations could offer to twitter, but I suspect you're exactly what I was referencing in my first comment.
posted by iamabot at 11:38 PM on June 24

Twitter: Why? What's the point?
Of course, I also predicted this "camera in a cell phone" thing was silly and wouldn't catch on....
posted by cccorlew at 11:40 PM on June 24

So, maybe you could tell us exactly what those "unique engineering challenges" are

It's Rails, you wouldn't understand.
posted by Blazecock Pileon at 11:43 PM on June 24 [1 favorite]

It's Rails, you wouldn't understand.

rake turd:fling
posted by ryoshu at 12:14 AM on June 25

It was depressing when one of my friends -- who, I'd like to point out, runs a website vastly larger than Twitter -- pointed out that, at 140 bytes a message, all of Twitter could be run from a single laptop, if it was engineered properly.

Perf matters. People forget that Moore's Law finally died.
posted by effugas at 12:19 AM on June 25

No, Really. Twitter isn't that complex, and rails doesn't suck, those that complain about scaling are just fucking parrots. It really should be '2 weeks to do 1.0'. Their fuckup was using MySQL as the model store. If they designed it in PHP, Java, Python, ASP, Perl, whatever -- they'd have the same problems if they used a MySQL backend.

The main DB load seems to be loading the homepage. If you had 2000 friends, the architecture, AFAIK, would go and do 2000 (albiet cached) DB hits to find their most recent posts and sort by time. FUCKTARDED. Everyone's been telling them to use the mailbox architecture for a long, long time. In simple terms, every account should be equivalent to an email inbox, and anytime someone posts, it should initiate a queue and CC it to everyone that subscribes to them. DONE.
posted by amuseDetachment at 12:37 AM on June 25 [1 favorite]

Oh and by the way, for those bitching about Rails: I never ever hear from those complaining the real primary flaw of Rails: It's not threadsafe. This means that the max connections on a web server is dependent upon the amount of memory on that server, as if you're doing a rails app that's killing 100MB/thread, you're not going to be running that much on 2GB, i.e. let's hope that the cache server has an extremely high hit rate. This is the real scaling difficulty of Rails, but all I hear is people bitching about the speed of ActiveRecord or some such BS.

On the other hand, I write stuff so much faster and happier on Rails that this downside is totally worth it.
posted by amuseDetachment at 12:46 AM on June 25

This is not a pedantic question or request. If anyone here can explain the appeal of Twitter (for the Twitter and the Twittee), I would love to know. I just don't get it.
posted by kcalder at 1:21 AM on June 25

Maybe you could tell "us exactly what" you expect to get as an answer to an inane question ?

You claim that Twitter's engineering challenges are unique. I wonder what makes them unique, given that others have solved seemingly identical problems on a much larger scale. You resort to personal attacks. Do you work for them, perhaps? That would answer my question.
posted by effbot at 1:25 AM on June 25

"If anyone here can explain the appeal of Twitter (for the Twitter and the Twittee), I would love to know. I just don't get it."

I use a service called diabetic days to log and analyze my blood sugars and insulin doses, and I use Twitter to forward the messages to DD so that they are archived when DD drops the ball or goes out of business. I'm pretty sure there are other non-social networking uses for Twitter and Jaiku, and more will be developed over time.

I'd like to set up some kind of nutrition database that can be queried via SMS using the API for Twitter or Jaiku, and also maybe a contact lookup process for contacts that are not stored in my cell phone.
posted by BrotherCaine at 1:55 AM on June 25

you're not going to be running that much on 2GB,

Yeah, but 2 GB is nothing these days. Name brand 64-bit Lintel servers will set you back a few grand with 8 GB. Memory is pretty damn cheap.

The fact Ruby doesn't currently have bytecode reuse probably hurts far more than anything else for long-running processes.
posted by rodgerd at 2:27 AM on June 25

The whole Twitter downtime saga has prompted vast amounts of ill-informed advice/speculation, ranging from OMGrails/mysqlSUX! to claims that anyone with half a brain can set up a scalable messaging system in five minutes. I'm amazed at how patient the Twitter staff have been in responding to all the nonsense.

They've admitted that their initial architecture was completely wrong for a messaging system, and they should've been far quicker to rectify it and allow them to scale, but anyone who thinks sorting out the mess is easy just doesn't understand what's involved. They've got a tough job ahead.
posted by malevolent at 2:35 AM on June 25

Ok, so taking all of the above hoo-ha as read, surely we can all agree that there *is* a more stable, scalable architecture that Twitter is aware of an building towards? Because it's not like the problems they're having are localised and un-obvious to laymen. Has anyone at Twitter ever come out and said, "Yeah, it's fucked. We're redeveloping the motherfucker from scratch in parallel and then when the day comes, we'll flip the switch and everyone will migrate to the New Hotness."

Or does it take every ounce of manpower they have just to support the cruddy old system?
posted by Jofus at 3:10 AM on June 25

Twitter is at its best when you realize that you can have lazy-reader group discussions with it. It is less about being a blog (although that's part of it) and not so much about announcing monster turds (though some do) and rather most about a non-real time conversation; unlike a blog, you tweet for your friends, not yourself, and their responses roll the cycle of primate social bonding. It's a non-real-time party-line phone call you can participate in as meatspace permits.
posted by seanmpuckett at 4:12 AM on June 25 [1 favorite]

This is not a pedantic question or request. If anyone here can explain the appeal of Twitter (for the Twitter and the Twittee), I would love to know. I just don't get it.
My friends and I use it over SMS, and it's a way of texting all your friends at once. With an "am bored, anybody coming out?", or to get entertaining messages from them. It's pub one-liners, without the pub. Really nice, sometimes.

Also, at big events it comes into its own -- great for finding people, finding what's good. At WWDC, if you followed a few key people, you could find out what sessions are good right now, where the parties were, get open invitations and so on. It's fantastic for that, actually, and I can't think of another service that would substitute

But what I really want to know is HOW is it going to make money? I mean, seriously. How?
posted by bonaldi at 4:16 AM on June 25

Twitter : Jaiku :: Friendster : Orkut

I think the hardest scaling problem Twitter will face is growing it's active userbase beyond sycophantic hipster webnerds. That and cashflow — they could put ads on their webpage, but the vast majority of their traffic is via other people's software.
posted by blasdelf at 4:21 AM on June 25

> I think the hardest scaling problem Twitter will face is growing it's active userbase beyond sycophantic hipster webnerds.

The SHNs were jaded with Twitter last year. These days most of the garbage is from followwhores and SEOs autofriending everything in sight, and who can probably be blamed for some of Twitter's outages. The platform was designed to cope with the traffic of hundreds of thousands of concurrent users but not the thousands of alpha-followwhores whose accounts demand the latest updates of hundreds of thousands of users every time their own homepage is hit.

Twitter is background signal. I keep an instant messenger window open see the latest announcements of a couple dozen friends and acquaintances. This is lowest-investment personal networking; I can keep in touch in as hands-off a way as I want, with a scant fraction of the attention necessary to manage accounts on LinkedIn or Facebook.

Twitter also provides a lot of fun for the stats and data visualization geeks. For the cool toys it enables (through free APIs), Twitter deserves some props.
posted by ardgedee at 5:05 AM on June 25

smackfu: "Private funding gives us the runway we need to stay focused on the infrastructure that will help our business take flight.

Someone is paid a lot of money to write that sentence. Think about that."

You have to give marketing folk who write press-releases like that some slack. Could you write crap like that all day without going completely fucking insane? I cringe every time I read stuff like that from our marketing/sales people or upper management but if it's that bad to read, think how much worse it must be to write?
posted by octothorpe at 5:29 AM on June 25 [1 favorite]

Many responses in one. Please accept my apology, I'm still not sure how best to avoid long comments...



blasdelf: Their problems stem almost solely from having a system...supporting hundreds of thousands of concurrent users.

This estimation of concurrency doesn't jibe with mine. How are you arriving at this conclusion? (Twitter's use seems higher to me.)

delmoi: twitter is such a simple system, it seems like something a good programmer could sit down and write in a couple of weeks

Delmoi, I respect many of your opinions on this site. Your opinion about Twitter's system sounds definitive but this assertion seems a little off to me, though. Do you have concrete data about your conclusion? Have you seen their code? Their network topology? Their database schema? Do you know their message volume? What's your opinion about their message queue solution? Do you know more technical information about Twitter than the platform they've (reportedly) chosen for their web service? I'm suspicious, hopefully understandably.

Kadin2048: Pretty much everything you can do with Twitter you could do with XMPP presence notifications...

Wow...really? I'm curious about this, I'd love to know what you mean by this. What's your assessment of "everything" Twitter is doing?

effugas: one of my friends...runs a website vastly larger than Twitter -- pointed out that, at 140 bytes a message, all of Twitter could be run from a single laptop, if it was engineered properly.

A website vastly larger than Twitter? Damn. Wondering by which metric: 7-day active users? Page views? Unique visitors? API traffic? SMS messages sent? Number of HTTP requests? Would you mind saying which site? (I can totally understand if you'd prefer not to say, I'm just curious.)

kcalder: If anyone here can explain the appeal of Twitter (for the Twitter and the Twittee), I would love to know.

Here's an attempt based on a local set of feedback from Twitter and Google Reader and FriendFeed users but I hope it's taken with a HUGE grain of salt as I have no conducted no specific testing on Twitter usage.

1) For a significantly large group of people, timely updates are important for certain relationships and events e.g. families, close-knit friends, emergencies.
2) Many people in this group have cell phones and can receive and send these afore-mentioned important updates with more immediacy than via a website.
3) Simultaneously updating a public website (or archiving) those updates is important to these people and is somewhat compelling/interesting to a larger set of closely-related people.

effbot: I wonder what makes them unique, given that others have solved seemingly identical problems on a much larger scale.

I mean...I'm kind of uncomfortable describing any problem as wholly unique, but doesn't it seems reasonable to say that...

- sending millions of messages to phones
- while simultaneously updating a frontend cache for a website that supports a huge number of visitors
- while also supporting a massive number of API calls from third-party clients

...is not easily described as similar to a lot of other products? (I also think it's accurate to describe it as a difficult task for a new product.)

If you don't think that's reasonably difficult or unusual, I'd be very interested in hearing an explanation of your engineering solution or similar other solutions (and the number of resources you'd assign or have been assigned and I'd benefit from your backing this up with specific experiences) while supporting a legacy system built on a CMS.

I'm not being snarky. Seriously, that would be excellent to hear about. And incredibly useful for anyone studying problems in a similar vein.

As far as I can tell, Twitter seems to have less than a dozen engineers working full-time. Is that enough? What constitutes enough? (And since you asked iamabot about whether they work for Twitter, I should also mention that I don't work for Twitter - I just think they have some interesting problems to solve.)

Jofus: Has anyone at Twitter ever come out and said, "Yeah, it's fucked. We're redeveloping the motherfucker from scratch in parallel...

Yes. There are some answers on the Twitter development blog where they say things like

"Our key problems have been primarily architectural and growing our infrastructure to keep up with our growth"

and

"Our direction going forward is to replace our existing system, component-by-component, with parts that are designed from the ground up to meet the requirements that have emerged as Twitter has grown. First and foremost amongst those requirements is stability. We're planning for a gradual transition; our existing system will be maintained while new parts are built, and old parts swapped out for new as they're completed."

Disclosure: Until last Friday I worked at Google on similar systems, mainly Google Reader. The Gmail and Reader teams have talked a lot about Twitter's scaling issues, especially those that seem to echo problems we've faced. We often have come to the conclusion that there's a difficult set of problems for Twitter to solve given their legacy and ones we wouldn't wish on our enemies (or friends who have delusions of grandeur) but I'm keeping an open mind... maybe some of you have crafted a ground-breaking solution based on Twitter's exact requirements?

I haven't. Partly since I make huge mistakes in every engineering decision I've made. Also. 'Cause I don't know those requirements in exact enough detail.
posted by massless at 5:32 AM on June 25 [15 favorites]

they'd have the same problems if they used a MySQL backend

Yes, but MySQL doesn't have the annoying, self-important crowd of fanboys that Rails has.

People are starting to love the "fail whale", because they find comeuppance satisfying. What's really happening becomes less important than constructing a narrative to bring the social stresses back into balance.
posted by gimonca at 5:32 AM on June 25

The real problem with Twitter for me is: how can I find the people I want to follow? Let people add tags to their profile. Oh, but solve the "whatever it is that's holding you back but I'm willing to be agnostic about" problems first.
posted by gimonca at 5:41 AM on June 25

1) For a significantly large group of people, timely updates are important for certain relationships and events e.g. families, close-knit friends, emergencies.

That use-case is inappropriate for SMS.

I have built systems in the same space as Twitter, although they probably haven't hit the traffic peaks Twitter has. And I've certainly never hit the peaks of Google Reader. IMO, if you want to see engineering that works, look at Facebook, which seems to boil down to "scale free where possible, feeds everywhere, caching everywhere".

The long-polling-with-Erlang they're using for the new Facebook chat is especially nice. If I was starting from scratch, I think that's what I'd be using, although not over HTTP internally, obviously.
posted by Leon at 5:52 AM on June 25

Post from Twitter's tech team:

We appreciate the creativity that the technical community has offered up in thinking about our issues, but our issues won't be resolved in an afternoon's blogging.

-- Twittering about archictecture
posted by bonaldi at 5:53 AM on June 25

That use-case is inappropriate for SMS.
I think possibly the users decide that, and they might not do an evaluation of the possible solutions available in the messaging space first.
posted by bonaldi at 5:54 AM on June 25 [1 favorite]

I think possibly the users decide that, and they might not do an evaluation of the possible solutions available in the messaging space first.

I totally agree that users bend technology to their will, and I think that's great - in fact, SMS itself is one of the better examples of that. But the fact remains that you can't rely on SMS; there are basic constraints built into the system that aren't going to change.

After the Virgina Tech shootings, a bunch of snake-oil salesmen crawled out from under their rocks offering variations on "emergency alerts via SMS". Despite the fact that sending 5000 messages via a handful of cells would take hours. I forget the exact bandwidth available to SMS, but it isn't much. It's very easy to clog a cell, or even an entire city.
posted by Leon at 6:06 AM on June 25 [1 favorite]

Or one could bet on the google backed jaiku.

Well seeing as how you can't even sign up for Jaiku now, and haven't been able to for the last half a year, i'm not sure it's the best thing ever.
posted by chunking express at 6:38 AM on June 25 [1 favorite]

If anyone here can explain the appeal of Twitter

My wife recently started twittering after she came up with a very practical use. In the past, if she's at work, she could just email me if she wanted to tell me something. There's a good chance her new job will block her email program though, so she's going to text info (I don't have/don't want a cell phone) to her twitter account so I can see it.
posted by drezdn at 7:12 AM on June 25

My girlfriend thinks Twitter is dumb. She's also a huge Liz Phair fan.

A few weeks ago when Liz Phair tickets went on sale and sold out in a flash of combustion, I shared my defeat on Twitter. Resigned, I went to lunch. When I returned I learned that a second Liz Phair show was announced, went on sale and took 30 minutes to sell out. But yet again I was too late. Fortunately, thanks to Twitter, a friend of mine saw my Tweet and bought two extra tickets for us.
posted by yeti at 7:33 AM on June 25

It's the Twits!™
posted by anthill at 8:09 AM on June 25

H-H-H-Hits from the Twits!™
posted by anthill at 8:10 AM on June 25

In case anyone is interested I am currently in the very early funding rounds of my own Twitter-alike startup. The idea came to me when I was in the bathroom and my friend texted me, saying "Yo, totally grunting one out right now" and I wrote back saying "Word? Me too. I don't remember eating eggs yesterday" and then my other friend texted me and said "Oh god it's like a soup can." I was forwarding texts back and fourth and I thought, dammit there has to be a way to update all my friends at once about how it's going for me when I hit the head. I have a plan to make money by the way, some kind of small, waterproof, always on texting device that can be mounted to the wall in front of the toilet, create the market and then the product baby, w00t!

I can't figure out what to call it though.
posted by Divine_Wino at 8:58 AM on June 25 [7 favorites]

If you don't think that's reasonably difficult or unusual, I'd be very interested in hearing an explanation of your engineering solution or similar other solutions

Well, the company I work for has built systems that process two thirds of all text messages being sent globally, and I know plenty of people who work on hardware for the telecom industry, so big numbers don't scare me that easily... but sure, I probably wouldn't try to build something like this with a bunch of LAMP hackers.

(the "gotta support the old CMS system" angle feels a bit odd, btw. surely there has to be a way to move clusters of big users over pretty quickly, once you have a new best-practices system in place?)
posted by effbot at 9:14 AM on June 25

you can't rely on SMS

Snuh? Isn't reliance subjectively experienced? If there was evidence that millions of people currently use SMS for communication they find important would you change your mind?

I know plenty of people who work on hardware for the telecom industry, so big numbers don't scare me that easily

I've seen massive bandwidth usage, data storage, and hardware use at Google. The fact that Twitter is a potentially complicated system to build doesn't seem correlative to the fact that telecoms or Google exist.

I'm still more interested in just hearing your technical thoughts, tho. Could you describe the solution you would use to creating a system that has the features of Twitter and scales to its current use? Also, how would you evaluate the current use of Twitter to approach this problem?

the "gotta support the old CMS system" angle feels a bit odd, btw. surely there has to be a way to move clusters of big users over pretty quickly, once you have a new best-practices system in place

How long would it take you to build the new best-practices system?
posted by massless at 10:38 AM on June 25

Do you work for them, perhaps?

No, but I design, implement and operate infrastructure for a living.
posted by iamabot at 12:49 PM on June 25

The fact that Twitter is a potentially complicated system to build doesn't seem correlative to the fact that telecoms or Google exist.

What I'm trying to figure out here is if it's potentially complicated because they've had trouble building/scaling it, or because it's potentially complicated in absolute engineering terms? Cannot say I'm getting anywhere, really.

(Btw, if the architecture descriptions I've seen aren't horribly outdated, things feel a bit... underdimensioned. I've worked with systems with more iron in their off-line test systems than Twitter has had in their production system. Is lack of funding a part of the story here, perhaps?)
posted by effbot at 1:49 PM on June 25

And twitter is such a simple system, it seems like something a good programmer could sit down and write in a couple of weeks

Or in 30 minutes, if you're a tad optimistic and ignore 95% of what the Twitter application can actually do :-)
posted by effbot at 1:56 PM on June 25 [1 favorite]

Metafilter analyzing twitters's infrastructure and problems is like two blind men arguing about what the color blue tastes like.
posted by iamabot at 2:41 PM on June 25

Metafilter analyzing twitters's infrastructure and problems is like two blind men arguing about what the color blue tastes like.

It must be rather interesting to walk around and think that you're the only person in the world who's ever worked on non-trivial computer systems.
posted by effbot at 2:52 PM on June 25 [2 favorites]

Snuh? Isn't reliance subjectively experienced?

Experienced, yes. But reliability can most certainly be objectively measured (assuming you define properly what the system is supposed to do). For example, the old telco 5-9's was based on a particular set of assumptions/constraints which are specified, and from there you can measure the reliability.

Building "carrier grade" systems is extremely hard, and 99% of websites don't bother. That's not necessarily a wrong decision, as if it takes double the resources to achieve a 98%->99% reliability increase, that may not be worth it if your system doesn't need it. On the other hand, if people expect it to be reliabile in all but the most extreme circumstances, as they do non-VOIP telephones, that extra effort is worth it.

(VOIP is an excellent example - it's great for a lot of reasons, but it's demonstrably less reliable than POTS. Relying on it for, say, 911 is dubious)
posted by wildcrdj at 3:02 PM on June 25

1. Develop web 2.0 app limited to 160 characters.
2. Get venture capital funding.
3. Sell ringtones.
!Profit
posted by acro at 3:21 PM on June 25 [1 favorite]

Snuh? Isn't reliance subjectively experienced? If there was evidence that millions of people currently use SMS for communication they find important would you change your mind?

Not really. That last few tenths of a percent is hard. SMS reliability in the UK is under 99.8%, and because of the architecture the time when it's most needed is also the time it's most like to fail.
posted by Leon at 3:29 PM on June 25

3. Sell ringtones.

Nah. Ringtone market disappeared overnight when phones became mp3-capable. More likely it's the "no cash flow, but lots of signups, maybe Vodafone will buy us" model. ("Free SMS on the web" has happened dozens of times in Europe over the past decade. No company has ever made it pay.)
posted by Leon at 3:33 PM on June 25

Twitter+muxtape <> sms+ringtones?
posted by acro at 4:05 PM on June 25

- But reliability can most certainly be objectively measured (assuming you define properly what the system is supposed to do).

- That last few tenths of a percent is hard.

Yup, reliability theory and assessment is one heck of a difficult and necessary discipline in engineering and can be used to objectively measure a system.

When I first mentioned use cases for Twitter, though, I wasn't talking about engineering functions but about its appeal. Leon said that the use case I outlined was inappropriate and further mentioned that "you can't rely" on SMS.

That's definitely the point of my "snuh." :) I was surprised to hear Leon say that it's a fact that SMS is unreliable and that timely updates for certain relationships and events (e.g. families, close-knit friends, emergencies) isn't an appropriate use-case for it and I'm curious about how that conclusion's been reached. I'm talking about the perception of current use by users and not the IEEE term of 'reliability' and so it's possible Leon and I are talking past each other. (If so, sorry about that.)

- when [SMS reliability is] most needed is also the time it's most like to fail

Are scale-busting conditions really the time SMS is most needed? Do you think that because of mass emergencies? What about local emergencies? Very local emergencies? What percentage of messages being sent right now are not vitally important to their senders and recipients?

the old telco 5-9's was based on a particular set of assumptions/constraints which are specified

However...if we're gonna chat about subjectivity... :) Seems like the standard by which a system is considered reliable requires a bunch of decisions that can incur subjective reasoning: the length of time of ensuring proper function, the functions that need to be considered proper, the manner in which the functions are tested...but that's kind of off-topic.

On-topic? I'm glad they have funding and I'm fascinated by what will happen to Twitter as a result.
posted by massless at 5:41 PM on June 25

> Wow...really? I'm curious about this, I'd love to know what you mean by this. What's your assessment of "everything" Twitter is doing?

Well, first off, I said "pretty much everything," not "everything," because I don't know enough about SMS integration to really talk about that side of things. But by "pretty much everything," I mean the core functionality, which to me is providing a way to subscribe to and monitor the status updates provided by a number of selected other users.

I think that's a pretty safe statement because that's one of the things that XMPP does. I mean, it stands for Extensible Messaging and Presence; what we're talking about is pretty much bare-bones presence notifications with a web interface. But really, I'm not even sure that XMPP wouldn't be overkill; you could probably come up with an even simpler way of pinging messages from server to server on a "subscribed push" basis if you wanted to, since you're not really using any of the 'messaging' parts of XMPP.

Where I think Twitter runs into trouble is with its centralization. If it wasn't a centralized service, they wouldn't have scalability problems, and nobody would be arguing over whether it was their choice of Rails, or MySQL, or Linux, or whatever, that was messed up. It wouldn't matter. In a decentralized solution where you put the servers out on the edges, which was the whole idea behind Jabber/XMPP, you wouldn't have any SPOF that could take out the whole network.

What I think would be a lot better than Twitter would be a sort of Twitter-in-a-box; a small application stack that would run on a per-domain server (just like email) that would have its own web interface and IM/SMS gateways if so desired, that would then interoperate with other servers. Rather than just using a globally unique username, you'd use username@domain addresses (just like email), and any server could go down without affecting the entire network (just like ... you know where I'm going).

With a decentralized architecture, the traffic handled by any individual node would be a minuscule fraction of the traffic Twitter has to deal with today. In the same way that a small mailserver doesn't need a lot of HA optimization, a small Twitterish presence server wouldn't either.

Twitter is a good idea, but it suffers because it's a communications infrastructure project that's being run as though it's a web site. Can you imagine what it would be like if email was done the same way, with one "Email.com" web-based service for the whole world? It wouldn't matter what choice of technology they used, it would be almost guaranteed to suck. That's Twitter's problem in a nutshell, and I suspect it will only get worse as they get more popular.
posted by Kadin2048 at 5:44 PM on June 25 [1 favorite]

It appears to me that Twitter is only around to bridge the functionality gap between SMS and IM. Otherwise the market share would all be on IM applications. Any architecture analysis of Twitter should therefore take into account SMS gateways, which might make decentralization difficult or pointless.
posted by BrotherCaine at 5:50 PM on June 25

It must be rather interesting to walk around and think that you're the only person in the world who's ever worked on non-trivial computer systems.

It must be rather interesting to walk around and think that you're the only person in the world who thinks running an infrastructure you have no first hand knowledge of takes no effort.
posted by iamabot at 5:59 PM on June 25

Metafilter: our issues won't be resolved in an afternoon's blogging
posted by britain at 8:33 PM on June 25

you have no first hand knowledge of takes no effort

OTOH, the stuff they write on their blog about the technical issues really do sound like someone trying to scale up from a small solution to a large one, without really knowing how.
posted by smackfu at 9:02 PM on June 25

Well I guess we're back to one of the points in the post, hopefully the cash influx will help them transition to a more robust infrastructure, code base and operations plan.
posted by iamabot at 9:33 PM on June 25

Kadin2048 - Thanks for responding. Sorry about overlooking your "pretty much everything" qualification earlier.

> Rather than just using a globally unique username

Namespaced identity and decentralization would have been interesting to see at the outset of their development, though I'm curious if useful adoption and critical mass would have occurred given the added weight in the user's experience of either a) deciding which domain/namespace to adopt or b) getting assigned a namespace and c) trying to type the longer names when following or replying.

> It really shouldn't be that hard to set up something similar that aggregates the status messages from XMPP or SMTP and provides an HTTP API

Setting that up seems moderately easy. Scaling that seems like...well...a scaling problem. :)
posted by massless at 7:01 AM on June 26