Twitter Said To Be Abandoning Ruby on Rails

let’s call that a derailment

PHP or bust

They should go to PHP. Lots of experience to tap into to get a scalable solution.

Rails sucks. I never liked it, and never recommended it.

I presume this is all back-end work? No impact on UI or functionality, right?

I will go one further..

Because of the similarity of Ruby on Rails and Cake PHP in the ORM layer and etc its more likely to be a move to PHP using the Cake MVC web framework.

The other choice is not Enterprise Java but say Plain Old Java Objects using the Groovy Graiils framework that works with Java.

Sometimes huge growing pains force change sooner than planned. Hope the transition goes smoothly. I’m just now getting into Twitter and although not an addict, yet, really enjoy the connecting that goes with the site.

Cindy

I thought database abstraction and transparency, almost as if you could avoid thinking about databases in traditional terms, was a big new bonus in Rails, why would you need to optimize the database?

Interesting post from Gigaom on rails getting some enterprise-level support: http://gigaom.com/2008/04/30/r.....l-support/

@5 - Front-end’s all HTML, CSS and JavaScript. You could duplicate the UI and functionality in FORTRAN, if you really wanted to.

Oh no, this is going to cause absolute mayhem. Michael, do you realise what you’ve just unleashed? This is more potent than a DHH post on GENERATING PROFITS or four day works weeks.

I’m betting they won’t side with a PHP or Java framework, but that’s not going to stop the upcoming rage virus epidemic.

PHP?

Django a much better choice here.

they should try python & Django… their competitor Pownce uses it!

lol.. plus they’d be able to run it on Google App Engine…

Perl all the way!!

I’m not very surprised, things happen and they don’t always have an explication. I just like Twitter and nothing more.

They should just go with IIS7 and sql server 2008. I know it is the “in thing” to not pay for your webserver os and db engine, but why not build on something that is used in the enterprise? You can cache to your heart’s desire in asp.net, and there are hundreds of thousands of people learning C# in college right now. Seems like a no brainer to me.

They should go with C. This is a messaging protocol, not a Web app people. Would AOL Instant messenger implement their app in Rails? That would be preposterous. This really says very little about the Rails framework, if you ask me.

They could rewrite twitter in like a day.

PHP FTW. All I have to say.

You know. I know this probably won’t set well with everyone, but besides PHP and Java, I would recommend Twitter really look at using Asp.Net MVC framework.

It isn’t a secret that http://www.plentyoffish.com runs his whole entire operation on 2 front-end servers and one DB server, and he’s doing what like 30,000,000 page views an hour or a day or something ridiculous like that.

Compiled code beats scripting in speeds any day of the week.

You know what, have ‘em use IronRuby with the Asp.Net MVC framework, that way they can write ruby and still get performance.

My $.02.

-TF

Goodbye Rails 

Maybe it’s time for Twitter to leverage Amazon’s E2C which has no uptime agreement. At least Twitter would have someone new to blame for endless, unscheduled outages.

That’s May Fool

@16 - PHP, ROR, Python, etc. are used in the enterprise, unless you don’t consider Amazon.com, Google, and Yahoo! to be “enterprisey” enough.

There was a no-brainer in your post, just not the one you thought.

How do we know the recent three day service outage only affected large users? What about the little people?

Twitter is easy to implement using an ActiveMQ (Java) cluster for the subscriptions, PHP for the front end/API, and another Java farm for message delivery. You can put all your eggs in one basket. They are probably going down because they are pushing around too much data on the backend. Subscriptions should just be pushing around the index for messages. They should have a memcached farm as well for messages. Oh Well.

What about jruby on glassfish?

@20: “Compiled code beats scripting in speeds any day of the week. ”

That’s why you use an opcode cache.

Depends on their strategy here, but there are quite a few other ways they can go. The RR monolithic strategy can be tough to cope with when you need to do things like asynchronous operations. Other platforms are just better equipped to do those sorts of things, without your environment ending up as a hodge-podge of technologies.

I’m *sure* they’re having conversations about the front-end vs. back-end vs. middle-tier vs. db. Or, I hope they are anyway.

I see lots of people suggesting a PHP solution, but I think that’s a limiting path for 2 reasons.

First, Twitter is increasingly global, and (I believe) PHP still lacks Unicode support. Unicode support would make life much easier for them.

Second, for a site that gets as much traffic as Twitter, they’d benefit immensely from a language that handles multi-threading well. Sorry, that’s just not what PHP does.

Go Java, but be smart about it

earle: PHP is a template language meaning that it’s a language that works within a template system (in this case, HTML). That makes building smaller templates very easy and fast. That’s one of the reasons that you really want to divvy up business logic from display logic. Django is a template system within a language, which may seem like a minor quibble, but it requires data preconditioning in order to use it, meaning that templating is something that has to be kept different and requires a different syntax.

It really depends on how they’ve structured things for their site. Both choices are valid, but it’s a bad idea to dismiss either of them out of hand.

@Matt Baron: agreed this is a messaging app. Sticking a db into the flow and distribution of messages is a non-starter for anything with real aspirations to scale.

Just about any language can work fine … it’s the flow of msgs and what you do with them that matters.

Anyhow [osted some more on this yesterday.

@26 Your technology doesn’t support compilation so your defense in using it to scale is use an adon? I don’t get the argument. I / me / developers everywhere shouldn’t have to fight with a technology that doesn’t scale and bolt something else on to solve a problem. I should be able to focus on my code, write it, and move on, not worrying about having to one day at some point install some other third party tool to make it go faster.

Ridiculous argument. Wake up.

PHP and Cake FTW!

@26 - that thought scared me - are they really *moving* around multiple copies of tweets? I’ve seen people seemingly in awe of being able to “deliver” that many tweets to people (the twitter spewage charts a few days ago). There should only be one copy of a tweet in a database table, not dupes. I honestly don’t think they’d have been that dumb.

Also, not really sure why they’re having the scaling problems. I’m not a huge RoR fan, but I don’t really care to bash on it that much. I’ve been around to see crap code on every major platform out there - I know people who can make *any* system *not* scale. However, if they’re trying to keep everything within the confines of the RoR framework, that may be the problem. Frameworks are great, but there are times when you need to bypass the confines of the system, usually for performance reasons. The trick is knowing when and how to do it, documenting it, training people on why you deviated, etc.

I suspect if they just rebuild with the same architecture and mindset, just a new language, they’ll have the same problems. Likewise, if they rebuilt from the ground up in RoR (or just Ruby) they would likely be able to make it scale much better with all the lessons learned. Of course, given that many of the original brains are now gone, perhaps that wouldn’t be as useful, because those experiences are now gone too… ???

ha! gosling is probably wetting his pants laughing right now (remember last year when he said ruby couldn’t scale for the enterprise? well, here you go…)

Just because a site has scalability issues, it does mean Rails is the culprit. Most of the time when Sites are not scaling it is database which is the culprit, mainly for not being totally optimized.
Twitter is growing faster than they expected and can’t really blame someone here because the service is free and beggar’s(users) can’t choose.

PHP only limits people of lack of creativity. They will probably find the correct language for the application, but I would not say that PHP is not the right language for this.

They are obviously going migrate to Termite/Gambit and replace Starling with ejabberd, I mean, what else could possible handle the load?

Come on people, as if the web front end has anything to do with twitter’s particular technology challenges.

A default install of PHP isn’t the best thing in the world, but once you tweak some settings and add a few extensions like eAccelerator, it’s real fast. It could power Twitter easy.

Clicky’s tracking server handles almost 20 million hits per day to PHP, and it’s far from top of the line. I realize Twitter would have quite a bit more traffic than that, but the point is that 20M/day is still a LOT of traffic, and PHP does it without a sweat.

@30 - Eric:

Multithreading and application scaling - at the server farm level - have pretty much nothing to do with each other. One might almost argue *against* that approach, as the ’shared nothing’ approach of PHP has been proved as scalable as any other approach out there. IIRC, friendster was originally in Java and *just* *couldn’t* *scale*. No matter how much people tried to second guess them, they eventually moved over to PHP and they now scale. Wordpress seems to scale alright as well.

I’m not saying Java *can’t* do it, but there seem to be enough examples of high profile PHP scaling sites to say that it can’t do it, or that the problem at hand inherently requires a threading model.

Loren Feldman is now an expert on web technologies and a professional douche bag what a skilled guy.

Can we get some BRAINF_CK up in here?

http://esolangs.org/wiki/Brainfuck

All the Rails people are going to cry and say it’s not Rails fault, it’s the developers fault.

I wonder if internal politics is partially driving the move, made easy by their past technical issues. I’ve seen it before, and sometimes the new solution doesn’t actually address the root cause. I hope this decision is based on rational thought instead of finding a expedient scapegoat.

Also, given the background of Steve the new guy (ex-google) i suspect Python or Java would be on the cards before PHP. Likely though, they’d investigate JRuby on Rails before anything else (if it hasn’t been done already). Last I read it was able to run 100% of the Rails stack without issue, and was faster than the standard C Ruby in many instances. http://antoniocangiano.com/200.....-shootout/ shows it faster than the Ruby 1.8.6, though slower than Ruby 1.9.

There are times when I don’t think wasting over 8 years of my life building enterprise financial systems where messaging was a routine part of the conversation was a waste of time. This is one of them, amateurs! Use a real messaging platform (Microsoft or Java (ActiveMQ). No wonder the tech team was fired! PHP is good for the web facing stuff and API calls. Twitter the web app or API calls should be reading data from a DB, PHP is more than fine and horizontally scales, so does the DB. All the fancy messaging magic is what is killing them and that should be able to queue load and scale horizontally as well to meet QoS targets.

We had the same problem that Twitter is having. RoR just doesn’t scale and it’s not ready yet.

Large-scale social networking sites sites, such as Facebook, have chosen PHP because of its execution model and because it’s a specialised scripting language designed for the Web.

Nothing beats Linux, Apache and mod_php. That’s why more than 20 million sites use it. That’s why System Administrators install it and Software Engineers use it.

Twitter’s lead developer and architect left the company. It’s time to scale, it’s time to use PHP.

PHP and CodeIgniter FTW.

http://codeigniter.com

I’m seriously interested in seeing if you will have enough butts and balls to post a retraction like a man

Partially agree with Matt. They should recode parts of their codebase in C and use some scripting language (Ruby’s ok) when appropriate.

Real Life Case of C rewrite: Rocketmail (way back in the day and became Yahoo Mail) was built in Python and then rewritten in C as things started to scale.

Twitter shouldn’t leave Rails. They’ve done a lot for the community by going through the growing pains that hopefully others (including myself) won’t have to. I hope they can keep it going and get rock solid with it.

Last time I looked at ActiveMQ it was all hype, not much implemented and buggy. Maybe it’s better now. I would try Joram.

On another note, I guess Bezos won’t be investing in Twitter. We all know how much of a Rails fan he is.

I’m not surprised about Twitter/Ruby. Simply because Ruby-on-Rails is non-scalable trendy piece of shit for weekend projects.

I know it’s matter of personal taste, but based on my big experience if I were Twitter’s VPE I’d create infrastructure on custom built C++ servers or on Java with layer of in-memory database.

This might be the opportunity Powce was waiting for to try a come back: my experience with large software projects and that they fail 75% of the time, special if they bring a new person in.

Outages can occur because of operations too. In fact in large scale com servers like twitter it should be expected. How many here work for a fortune 100 that has good exchange uptime? How do we know someone didnt accidently init 6 a production box thinking they were still ssh@ /dev? Every app & architecure design has its tradeoffs & it usually comes down to how much of a life or how much sleep your DBAs are willing to give up. You could have the best design that is so novel your dba doesn’t understand how to keep it humming. Sounds like they need ITIL more than a new interpreter or framework

This is nuts. This app is fundamentally a messaging problem, & a modest one at that.

Putting a DB into the flow IS THE PROBLEM. It’s not the language, it’s not politics, it’s not even about friendships or past contributions to a community.

It’s simply that you’re asking a DB to act like a router, something which it is pretty bad at doing.

Why be surprised when it doesn’t work?

Quite frankly I’m glad this has happened since I sometimes get the impression the Ruby on Rails community are better than everybody else which is simply not the case.

Ruby on Rails looks to me like it was written for all those unemployed Visual Basic programmers looking to be re-skilled in web development work.

I would laugh if after a couple of years the folks at Twitter decide to use something like Seaside (Smalltalk), because by then they will have realised that was probably the best tool for the job.

Personally I’m still promoting Ruby, since it’s got a lot more going for it longer term unlike Ruby on Rails IMHO.

With Yahoo! developers jumping ship, there will be no shortage of solid PHP developers with large-scaling experience!!

Good move on Twitter

ROR is a tool and it has it’s place, but it’s no where as mature as java. There are so much high technology stuff around java that Ruby is not going to catchup to it.

I think the java move is smart. One of the largest online poker site is on java and uses clustering very effectively

They have to switch canoes in the middle of the stream to send a message to the moneymen that they’re serious about building teh awesome.

Remove the DB and rewrite, using a messaging protocol using jabber / IRC.

There’s an O(n^n) problem here when users send messages to every other user through a database. If you replace that with the idea of subscriptions and use a decent messaging protocol, the problem goes away.

Relational databases are not the solution to all problems.

I agree with #56.

Was just thinking the opposite today. http://verni.wordpress.com/200.....-to-offer/

Surprise! Using a framework like Rails to do something as simple as Twitter didn’t work very well.

I agree with the poster who said this should be written in C. They could still use Rails to build out the frontend, or frankly anything else. The workhorses behind the scenes still needs to be something lower-level, faster, with greater opportunities for optimization.

This isn’t an indication of Rails’ failing. This is an indication of a team of people choosing the wrong tool for the job, plain and simple.

Look at Facebook. PHP frontend to a primarily C++ backend.

php or python is the way to go for content oriented sites. java is o.k but require development skills and more $.

“Readers tired of Twitter articles said to be abandoning TechCrunch.”

The Greatest Enemy of Knowledge is not Ignorance, but Illusion of Knowledge
- Stephen Hawking

That is what is happening here in these comments, everyone giving their opinion about how to scale twitter and what languages to use to make it better when they don’t even know what the real problem is.

Loren Feldman: Nice turban asshole. Welcome to the real world.

I doubt all those apps have the amount of database hits that twitter does. Maybe they should look at ironPython and go the .net route. If .net can power mySpace I am damn sure it can power twitter.

Why the change… old school hires… nuf said

“…We’re thrilled to announce two talented new team members. John Kalucki is an experienced distributed systems architect well versed in relational messaging as the former co-founder of San Francisco based SQLstream. Steve Jenson is a familiar face for a few of us here at Twitter HQ because he’s a former Google software engineer known for his work scaling Blogger and Blogspot–a service which tens of millions of people use on a regular basis…”

I cannot believe you idiots brought up Zed Freaking Shaw again.

Hasn’t that shrill halfwit gotten enough ink by now?

This is great…Finally Twitter surrender to PHP or Java…

Ruby On Rails—Write less, suffer more

Apparently this site can’t scale either.

That’s the 3rd connection timeout I’ve had in the past 10 minutes.

What’s wrong here?

Twitter can be done scalable in almost any platform. The only real problem with RoR is it’s relative lack of maturity, meaning any high-traffic site will have to find it’s own solutions. That learning process also includes regularly ending up with dead-end solutions and trying again. Not something everyone can afford.

And in general, the best strategy for a highly scalable site is ’small pieces, loosely joined’, leaving room for mixing and matching several technologies. This is why hanging a project like that on any monolithic framework (like Rails) is a not such a good idea.

The most likely solution for Twitter will be to step by step refactor pieces of the platform with better suited technologies, not completely ditching RoR.

Bad. fucking. idea.

@64 - your saying PHP and Python DON’T require any development skills?

Real professionals are technologically agnostic.

They pick the right tools for the right job. If it becomes apparent that they need to switch tools, then great, choose a better tool for your current scenario.

It’s not rocket science people, really…

Bashing a language or framework just highlights your ignorance.

You don’t have to use the same arrow every time, that’s what a quiver is for.

This is hilarious. What part of Rails, exactly, doesn’t scale?

Is it the database that doesn’t scale? The front-end web server? The IM connections? The db connection pool? The outgoing email servers? The API connections? The SMS gateway?

Rails is just a Ruby app. Which, in an application as simple as Twitter (big, but simple) is just like any other app. You can run as many instances of it as you like.

The scaling problems Twitter has are related to their data, the different gateways they connect to and the need to push new content instantly. PHP can’t send ten thousand updates per second over IM, SMS, email and HTML any faster than something written in Ruby.

You wanna stop messing about and run an Enterprise Java system on Sun UNIX. That’s what you wanna do.

Boy the FUD floodgates have really opened up on this one. Given how many different language and platform suggestions people have made on this thread, maybe there is one generalization we can make: it’s not the language’s fault. Scalability is not something inherent to a language. Your app is not going to scale because it’s written on Java, .NET, PHP, or C.

Also, many people on this thread appear to be confused as to the difference between scalability and performance. One language may have a linear performance benefit over another, but it does not make it any more scalable. Nobody is going to argue that C has better performance than Ruby. Because C is statically compiled and ruby is run time interpreted.

However rewriting twitter in C and keeping the same rough architecture is only going to allow them to reduce the # of servers by a linear factor. It is likely that there is something else wrong with their architecture on a conceptual level that is preventing easy scalability.

People, don’t you get it? This is just BS, Twitter is not giving up on Rails, at least not for the front end.

Wait and see who was right.

TechCrunch == FoxNews of the web.

-Matt

Using any sort of general framework for something at the scale of Twitter is insanity. It makes more sense to develop your own HTTP daemon and cut things down to the lowest level possible, especially for the backend / AJAX-y request side of things. Why get a big framework or bulky runtime in the way? It’s not so relevant now, but Feed Digest was architected to handle 1 billion requests a month on a single server (and made it to 500m at one point) following this low level model. It’s possible.

Twitter using Ruby on Rails is like NASA using Visual Basic. It’s nonsense.

Ruby On Rails - “Write less, suffer more.”

RoR losing a major account - it’s all mostly downhill from here for them, likely

@Loren Feldmen - have a more pleasant face, you look like shit

As much as I dislike Rails, and frameworks in general, if they have problems scaling it for something like Twitter, the problem isn’t Rails but their architecture.

The problem of scaling that kind of message distribution is “solved” and has been for many years. Heck, they could even do it by setting up a cluster of Jabber servers and a set of “dumb”, largely stateless frontends.

Facebook uses APC with PHP, works good and fast. Switch.

If the language would just be properly updated or provide a compiled solution or a better memory caching system everything would be fine. BUT NOOO. We have to have 100-200MB of memory for each rails app. lul.

For speed go for Django

OK we are just going to have to take predictions on this one. The market is opening at 60% that they announce they are dropping Rails by the end of July. Something makes me think that forecast is going down:
http://www.hubdub.com/e/Market....._6663/view

Wikipedia Architects: http://upload.wikimedia.org/wi.....-04-12.png

Twitter is currently hiring (twitter.com/help/jobs). Check out some of qualities they want in their engineers:

- Experience with Ruby on Rails stack performance tuning
- Experience with JVM-based application stacks
- Code using primarily Java, Ruby, C/C++ and Scala
- Expert knowledge developing and debugging in Java and C/C++ on Unix
- Knowledge of python and ruby

I wouldn’t quite say they were “abandoning rails.”

@77. Of course you can add a new instance but surely it need money. The problem now is rails apps wasting a lot of resource. So, it is easy for your server suffer overload. PHP is not the only alternative as we have Java and many more.

Twitter is stupid and so is RoR fad.

Come on, twitter isnt exactly a complex bit of code. With a couple of million in the bank as well it’s not even a scaling problem. I see no one mentioning message queues which would be the heart of a decent messaging system. Anyone here actually used message queues?

It’s not the ‘framework’. that has nothing to do with scaling. it’s the way objects are marshaled. There is a different byte order for every persistent object in Active Record. Plus, the toy databases and the table structures, migrations (changes to the schemas) etc., all add to the problem.

People should read this to understand the problem: (There are better development stacks) http://www.avibryant.com/?p=16

I wonder if Apache ActiveMQ is part of the new mix (if not already)? That would be neat to read about in a soon-to-be high profile site that does heavy lifting service behind the scenes.

Twitter is dead.

@loren and others, plz share your computer science qualificaitons — graduating summa cum laude from egomaniac U doesn’t cut it

Ah, ad revenue down _again_ this week?

I can’t believe it.

If you want to know the real deal with Twitter, then why not ask them? I just asked two of their staffers and they said you’re full of crap. I can quote them on that and I’ll gladly put you in direct contact with them.

As for all of the asinine comments in this thread, please do learn something about web development, specifically these “superior alternatives” you’re suggesting. Saying Rails doesn’t scale isn’t fashionable anymore and it doesn’t make you look smart. I’ve been involved with massively scaling Rails on a number of sites and, though it’s been painful at times, it works.

As I said before: PHP went through this same trial. Java went through this same trial. Python is on the downside of this same trial. _Someone_ had to take it to a big enough scale before someone cared enough to make it scale that far. Twitter has done this for Rails: they’ve pushed it to the limits and taken it beyond them to new heights. They’ve motivated other people to do so too, as many of the new updates to Rails have greatly improved performance and scalability. As Twitter grows or another site surpasses it, we’ll see the same growth again. It just takes time.

@77.: I sort of agree with you. Twitter should be IO constrained, not limited by framework scaling issues that can’t be reasonably easily overcome. That said, scaling _Ruby_ to that level isn’t trivial - you do need to know far more about Ruby than the typical Rails developer will.

We learned that the hard way at Edgeio when we ran into “fun” Ruby quirks like the ridiculous performance hit you get with the combination of many sockets open and many threads running at the same time (solution: architect your app to use multiple processes instead), various memory leaks (yes, Ruby can leak like a sieve if you do the wrong things), lockups (the MySQL c-client lib can hang your Ruby script indefinitely if you don’t take care) and a number of other issues that you first start running into when you get to a certain size.

That said, we had a messaging middleware server written in Ruby, using multiplexing to handle IO that was mostly IO bound (the most we ever processed was around 2-3 million messages a day, and we never exceeded around 10% CPU usage on a single 2GHz Xeon core peak) that would’ve been trivial to scale by clustering them in a mesh, so it’s certainly doable.

As I said elsewhere, this isn’t a Rails issue, or a Ruby issue. Ruby is by far my preferred language, but if it’s easier for them to find people who can do it properly in another language than Ruby, then it might still be worth switching for them.

AOLserver (with TCL) FTW!

Actually, it might just be that the tubes got filled.

Ev says no, they are not leaving RoR.

Kambiz

Michael, you are high as a kite.

http://twitter.com/ev/statuses/801530348

Evan said on twitter:

FWIW: Twitter currently has no plans to abandon RoR. Lots of our code is not in RoR, already, though. Maybe that’s why people are confused.

Haha, me, Coda, and Adam all jumped on this update within 10 mins from Evan posting the clarification.

Twitter’s scalability problem is probably of unique proportions that even a stock message queue solution cannot be easily rolled out. The problem will get worse as more people follow one another, leading to difficulty in partitioning users into distinct groups.

They’ll need to employ different strategies for dealing with different Twitter users. For instance, partition users based on the rate they poll twitter (e.g. Power Users vs. experimenters), timezone (people have to sleep!).

epic lulz

I got my Daily Twitter, now where the hell is my daily Seesmic?

francisyeboah60.media-toolbar.com

i heard facebook has problems scaling PHP….they are adding hardware like crazy to scale…..may be java, c++ might work better for massive operations

SELL.

They obviously can’t keep it going sustainably.

They need to sell it, and become junior engineers within a company that has the code chops to make Twitter stable and relevant.