Benchmarking ActsAsList vs RankedModel vs Resort
Ready for a drag race? Let's see who is the fastest!
Sorting solution for Rails 3 showdown!
Motivation
When we released Resort we were glad to see such good feedback. There was some concern about the efficiency of Resort, so we were also curious of how it actually performs compared to other libraries — here is our first benchmarking experiment.
Benchmarking
These are the participants in our benchmark:
We followed the instructions provided by RankedModel and ActsAsList and wrote a simple script that uses in-memory SQLite and ActiveRecord to create some records and then run the benchmark. If anyone is curious about the setup, this is what we used:
Gear
- Intel i7 920
- 6 GB RAM DDR3-1333
- WD Velociraptor 150 GB @ 10K rpm
Environment
- Mac OS X
- Ruby 1.9.2-p180
Running the benchmarks is quite straightforward:
$ git clone git://github.com/vpereira/acts_as_list.git
$ git clone git://gist.github.com/894185.git list_benchmark
$ cd list_benchmark
$ gem install activerecord rankedmodel resort sqlite3
$ ruby list_benchmark.rb 100 200 300 400 500 1000 2000Results
DISCLAIMER: I'm no benchmarking expert, if you find any error in the script or interpretation please comment and I'll try to fix it ASAP.
The y-axis represents the time in seconds and the x-axis the number of elements of the list. (Ergo, less/lower is better).
On creation, both Resort and ActsAsList have nearly constant performance (meaning that creation does not depend on the current number of elements in the list), while RankedModel performs exponentially because of its algorithm.
This is the worst part of Resort — at least with 2000 elements or more. Hopefully the nature of Resort's ordering algorithm will benefit greatly from upcoming Rails 3.1's Identity Map, so more benchmarks to come! :)
Here we have constant performance for Resort, almost constant for Acts As List, and quite exponential performance for RankedModel.
This seems to be Resort's golden feature! Basically, this is when you do this:
We can see Resort's almost constant performance. Both Acts As List and RankedModel perform exponentially, the former being a bit faster.
Nevertheless, here RankedModel beats both Resort and ActsAsList! Well done.
You can find the detailed results in this gist.
Who's your daddy?
I was really surprised the first time I ran the benchmark, I didn't expect RankedModel to perform so exponentially. After the results, we'll definitely stick using Resort since the ordering overhead is not that singificant (19.85 ms slower than RankedModel at 2000 elements) and in average it performs great, plus it is what really fits our needs (that's why we built it in the first place!).
Roadmap for Resort
While writing the benchmark we found that Resort had some areas to improve, so we've just released 0.2.0 with bugfixes and some performance tweaks. In the future, we're planning to support Identity Map when Rails 3.1 is released (which will improve the ordering algorithm) and try to find a way to paginate the elements. Stay tuned :)
