Some lift & CircleShare ...

David Pollak's Blog

Some lift & CircleShare performance testing Mon, Jul 23. 2007

It's been a while since I made a blog post. I've been heads-down working on CircleShare.

We're using lift to run the server part of CircleShare. First, you may be asking what CircleShare is... well...
CircleShare occupies the center of your digital photography experience. CircleShare makes it super simple to securely share photos without logging onto web sites, protect your memories, harmonize digital content between your computers, and publish your photos to numerous online services including SmugMug. Best of all, CircleShare runs on Mac and Windows.
  • Securely share pictures with friends and family. You can send full resolution pictures to family and friends without worrying about web site passwords, pictures being too big to fit in email or waiting for things to upload or download.
  • CircleShare protects your memories by making copies of your pictures on our servers. Your pictures are secure again hard disk crashes, viruses and accidental erasure.
  • If you’ve got multiple computers in your house, CircleShare will harmonize the pictures on all those machines. No more worrying who downloaded the camera card or wondering which of your computers has the vacation 2003 photos.
  • Publish your photos from your cell phone onto your computer or from your computer to a myriad of online services. No waiting for uploads and downloads or browser crashes because you’re sending too many pictures. CircleShare takes care of the heavy lifting, all you do is drag and drop pictures onto a “Helper” and CircleShare takes care of publishing your photos.
CircleShare is a combination of client and server technology. The clients are native code (Objective-C on Mac and C++/QT on Windows) applications that use an HTTP API to talk to the server (yes, we will be publishing the API and we will encourage folks to build applications that access CircleShare.)

We want to be sure that CircleShare is stable and scalable, so I spent about a week building code and an environment to test the scalability of CircleShare.

I created an Amazon EC2 instance to run Butterfly (the internal name of the CircleShare server) and MySQL. Then I built some Scala code to access Butterfly. The code logs into Butterfly and every 30 seconds performs 2 REST calls (one to check status and this requires no DB access and one to get the next command to execute and this requires a DB query.)

A side note about EC2 instances... they are supposed to be a 1.7Ghz x86 instance with 1.7GB of RAM. They are virtual instance and the actual performance depends on the amount of CPU usage on the actual machine serving the instances. During high CPU usage, I noticed that all the instances slowed down so there was some bleed of CPU utilization across the instances I fired up.

So, anyway, what we've got is Jetty or Tomcat (it turns out that Tomcat 6 has about 20% better throughput than Jetty and we decided to use Tomcat 6 in production.) (an update: The folks from Mortbay are working with us to reproduce the performance envelope that we saw and we'll hopefully see a better Jetty.) Running inside Tomcat is lift. Running on lift is Butterfly. The same virtual machine is running a MySQL 5 instance.

I fired up a simulated batch of 2,000 clients running on another EC2 instance. Things worked like a charm. There was sub 1ms ping times between EC2 instances. Butterfly was servicing the non-DB calls in < 1ms. The "get next command" calls were taking some time to execute after 15 minutes, the time started increasing and finally, MySQL started reporting dropped connections.

I added some DB connection pooling and tuned the number of simultaneous connections (the right number turned out to be 70.) Then I restarted with 2,000 simulated clients. Things ran like a charm. I sparked up a second EC2 instance and 2,000 more clients.

I kept doing this until I got to 4 EC2 instance, each running 2,000 simulated clients. This is 520+ requests per second and the EC2 instance running Butterfly was spending about 30% CPU in the Java process about 40% CPU in MySQL and 20% of its time in System IO/Wait. At 8,500 simultaneous clients, the system started dropping connections (this point came earlier when using Jetty) and was generally saturated.

I then put Butterfly on our production hardware (a quad-core Intel machine running @ 2.something Ghz with a hardware RAID card that has buffering.) At 10,000 simulated users (>650 requests per second) the server had a load average of about 0.40 and the Java and MySQL processes we using about 15% CPU (of the 400% available).

To date, I have not done any material performance tuning of either lift or Butterfly. I was pleasantly surprised that both performance the way I had hoped they'd performance. As expected, the DB seems to be the gating item in performance. As we scale CircleShare, I've got plenty of performance headroom using Scala Actors like I did with Skittr.

More information as we have more experience with lift and Butterfly.






Posted by David Pollak in Scala Comments: (3) Trackbacks: (0)

Trackbacks
Trackback specific URI for this entry

No Trackbacks

Comments
Display comments as (Linear | Threaded)

#1 - Joel Meyer said:
2007-07-24 11:15 - (Reply)

I've been following your blog because of an interest in lift, but the application you're working on really excites me as well. I'd love to have an application that would sync the photo libraries on the Mac laptops we use at home and back them up somewhere.

The performance numbers are also interesting. I'm really curious what type of hit an application takes when running on EC2 because of the virtualized hardware. Without an apples-to-apples hardware spec comparison (1.7Ghz virtualized to 1.7Ghz real, etc) it's hard to draw any conclusions, but the numbers above make me think that it's non-trivial. Perhaps when you consider the cost of hosting the hardware yourself, acquiring bandwidth, etc vs renting it all from Amazon (even though you have to rent more) EC2 is a compelling alternative.

#1.1 - David Pollak said:
2007-07-24 11:36 - (Reply)

We use Amazon S3 for storage. We will use EC2 for certain features we're planning on offering in the future. However, there are some challenges to using EC2 for persistent database serving and some other stuff so we chose to go with a hybrid model (some at Amazon, some on our own hardware.) This model works well for other companies including SmugMug.

#2 - Derek Chen-Becker 2007-08-07 10:22 - (Reply)

Very cool. It's nice to finally see what you've been up to and it's really nice to see how well Scala is holding up. I'm on Ubuntu, so I can't wait for the API to be published :-)

Derek

#3 - Olivier said:
2007-08-26 09:43 - (Reply)

Hi!
I have taken great interest reading all your blog about scala/lift.
After years of java (and mostly WebObjects), I have recently felt the need to learn new things. Flex and ruby/rails were first.
Then I heard about Scala (about 2 months ago). But the first look at the code examples was not as easy to understand as Ruby.
But the poor performances of Ruby (I hope compiled JRuby will improve it) pushed me to have a second look at Scala.
Thank you for putting your benchmarks. They will help me put more effort into understanding Scala's syntax as well as lift (which also seems at first glance not as easy as Rails. But again, considering the advantages, worth spending some time :-)

I have a couple of questions:
1- I understand you are deploying your scala application into Tomcat/Jetty. Wouldn't it be better to have a proper Scala servlet container ? Wouldn't it scale better ? (Is there such a thing).
I am wondering after seeing benchmarks of Erlang server vs. Apache.

2- You are developping your application for both Windows and OS X (because I am on Mac and I like photography...Just don't have time for designing my web site as you can see ;-)
I understand the benefits of Cocoa/Objective-C, but if you are using Qt, would it be not easier for you to use it also for OS X ? Will you be able to maintain the same features for both platforms ?

I wish a bright future to your company. I may have some needs for your first application!

Regards,
Olivier

#3.1 - David Pollak said:
2007-08-26 13:43 - (Reply)

Olivier,

Thanks for the post!

While Scala is a great language for writing and maintaining code, with the exception of Actors, there's no material performance advantage of Scala code over Java code. While there is a Scala Servlet container (Kitten), I'd prefer to leave the development of something as complex as a Servlet container to the folks who do it best and right now, those folks choose to write in Java. Given that Scala can perfectly subclass Java classes, implementing the Servlet in Scala (as lift does) and running it inside the best container around (be that Tomcat or Jetty or something else), that seems to make the most sense.

In terms of CircleShare, we want to maximize the quality of the experience for our Mac and our PC users and thus have chosen to write native apps for each. Cocoa and Objective-C is the best client development environment on the planet. Using that for Mac gives us maximum latitude with the best results on the Mac. We chose Qt on Windows because it sucks less than the native Windows APIs, but in my opinion, Qt still sucks massively and Trolltech is a seriously weak software vendor that has no regard for security in their products and is really pissy to customers who have spent a lot of money on their tools. If we have to decision to make again, I might opt for a Scala-based client compiled with gcj on Windows (and Linux).


Add Comment

Enclosing asterisks marks text as bold (*word*), underscore are made via _word_.
Standard emoticons like :-) and ;-) are converted to images.

To prevent automated Bots from commentspamming, please enter the string you see in the image below in the appropriate input box. Your comment will only be submitted if the strings match. Please ensure that your browser supports and accepts cookies, or your comment cannot be verified correctly.
CAPTCHA 1CAPTCHA 2CAPTCHA 3CAPTCHA 4CAPTCHA 5
 
Submitted comments will be subject to moderation before being displayed.