This month’s Linux Journal has an article about Redis. I read about it while sitting on the shitter, because that’s about all that Linux Journal is useful for. The article itself was crap, but the introduction to this product was at least tempting.
The basics: take memcached and add a disk backing, replication, virtual memory, and some cool additional data structures. Hashes, lists, sets, sorting, joining, transactions, yay! I instantly got a geek boner scanning the feature list. My boner quickly faded to about half mast when I saw the C clients that are available. No consistent hashing? Wait, no server hashing at all? Yet they have a gay Ruby client, fully featured? C is the lowest common denominator when it comes to language support. Start there, then add high level bindings using this low level library. libmemcached got it right. You start on the bottom and work your way up to higher level bindings. It seems that Redis took the wrong approach to this which will result in every language having a different client implementation. This leads to inconsistencies between languages, which is bad news. That on first glance gave me that headed-for-the-toilet feeling for this project.
With a half-mast boner, I decided to do some benchmark comparisons. Perhaps I’ll see some real numbers that might arouse me. Stand back, I’m going to do some science!
The Test Setup
I used the same machine as client and server for both tests to avoid network adding a skew to the results. (This should stress the algorithms themselves) The hardware itself doesn’t matter, it’s an apples to apples comparison. The software does however:
- Redis 2.0.0 rc4
- Memcached 1.4.5 (release)
- Credis (libcredis client) 0.2.2
- Libmemcached 0.31 (I know this is a little outdated, but for this test it works fine)
- Memcache client benchmark app: mc_stress.c
- Redis client benchmark app: redis_stress.c
The clients I wrote are just simple iterations of setting and getting keys and timing the results. Since WP sucks for posting source code snippets I had to stuff these into PDF files. Feel free to recreate my tests and post your results, since that’s what science is all about.
Results
This test varies the key size with a static value of only 3 bytes. (Using 100k unique keys) I’m guessing this will stress the internal hashing algorithms used for processing the key names. As you can see here, Redis came up short by a range of 20-26% the speed of memcache.
Same test, but showing the Multi-GET performance. Redis clearly has some kind of problem here, and I believe it might be due to requesting a large amount of keys in a single MGET operation. 100,000 keys seemed to be the upper limit… but in every case, Redis came up short.
This test stresses the different value sizes. Using a small 10 byte key length, I used varying sizes of value lengths up to around 16KB. Even though memcached advertises a 1MB value limitation, performance drops significantly over 8192 bytes. SET speed was reduced by 100 times and depending on the test, GET speed was either significantly reduced, or reduced by 100 times as well. I suspect that playing around with slab sizes might assist with the speed here, but that becomes quite the pain in the ass. The main point: If you use memcache, do not stuff large values into a single key. Hash them out into namespaces, you’ll be much better off.
Adding to this, Redis boasts a 1GB value size. We would need to graph another relationship of transfer rate to determine if there is an inefficiency in handling larger values, or if it’s just due to the size of the value itself. Memcache shows a similar trend as value size increases, but again Redis is 20-40% slower than memcache.
Here is the same test, showing Multi-GET performance. Similar trend as key length test.
Conclusion
After crunching all of these numbers and screwing around with the annoying intricacies of OpenOffice, I’m giving Redis a big thumbs down. My initial sexual arousal from the feature list is long gone. Granted, Redis might have its place in a large architecture, but certainly not a replacement to memcache. When your site is hammering 20,000 keys per second and memcache latency is heavily dependent on delivery times, it makes no business sense to transparently drop in Redis. The features are neat, and the extra data structures could be used to offload more RDBMS activity… but 20% is just too much to gamble on the heart of your architecture.
Maybe sometime in the future Redis will be up to par with Memcached performance. Or, it could be the extra VM and disk backing is inherently flawed to begin with. In that case, Memcached will always win, and Redis will be only useful in a niche market somewhere between DB caching and NoSQL fanboys.
Addition:
Anyone that would like to post their own benchmarks please do. But remember to test apples with apples. Keep the tests simple and very specific to avoid adding outside variables that would skew results. Try to avoid shit like using VMs as they’re too chaotic for timed tests like this.
At some point in the future when I have time, I’ll try to revisit this test again. (It seems to have become quite popular)
Anyone that would like to post their own benchmarks please do. But remember to test apples with apples. Keep the tests simple and very specific to avoid adding outside variables that would skew results. Try to avoid shit like using VMs as they’re too chaotic for timed tests like this.
At some point in the future when I have time, I’ll try to revisit this test again. (It seems to have become quite popular)
/sys/toilet 
41 Comments
Hello, thanks for giving a try to Redis.
I think in Redis master we resolved the issues with multi-bulk transfers that are now an order of magnitude faster, but I’m trying to reproduce your results.
Anyway I noticed that you did not ran the most meaningful test that you could run, that is, multiple threads asking for random small data.
Single threaded tests with big data are still interesting in some rare application, and I’m trying to understand if there is still something wrong about this in Redis master. But what makes the difference is the ability to the database/cache to resist against many concurrent *small* requests IMHO, in 90% of real world applications.
I’ll take you posted with my results, but I think you have a serious methodology test here.
Benchmarks are *very* hard.
Well, I’m a little star struck that the Redis creator found my little corner of the Intertubes!
I think I’d disagree with you on the concurrency level and the average use case. Granted, most of the modern web languages are threaded by nature. (Rails, Pylons, etc) which naturally creates a threaded client to whichever data backend you choose. That, is a perfect world: connections are pooled and shared, stacks are shared, and there is little overhead for each request.
In the real, dirty world, threading is difficult to implement, and not everyone does it. (I speak for myself, and I might be biased… can I get a whoop whoop from anyone still using CGI?) This is how I use memcache, unfortunately. Lots of single connections hammering the hell out of it. Memcache appears to do a great job at setup and teardown of individual clients, and that is the test I set out to bench Redis against.
Secondly, in regards to key size and value size, I think our sample space is all over the place. You do not know how many times one of our web developers have come to me asking, “Hey admin, what was the limitation on the memcache key size?” … “WHAT? Only 256 bytes? Well I’m going to have to work around that. What is the limit on the value I can do then? … ONLY 1MB!?”
That is the typical conversation. It pains me to think about it, because I already know how memcache performs under large key/val sizes. The point is, we can *never* assume what a real world application is. We can only advertise what a product can do, or can not do.
I realize it’s impossible for Redis to perform with every possible use case, but perhaps it can be *configured* for different use cases? Rather than choose between cache or database for the goals of your project, perhaps you can let the user decide. Drop in memcache replacement with better features? Or speedy (enough) NoSQL solution? Both require a different set of tunables. I believe Redis could really gain popularity if it can do both jobs well.
And yes, benchmarks ARE very hard… but they are a requirement in our field, as scientists. If we make boastful statements like “Redis … is as fast as memcached but with a number of features more.” we should provide the evidence to prove it.
Salvatore has responded to your post here:
http://antirez.com/post/redis-memcached-benchmark.html
I recently tested both memcached and redis using php for my simple application. I found redis to be about 30% faster than memcached in my novice test.
And.. Well played, Salvatore.
Val,
I’d be interested to see your test setup and more details about your results. After all, I’m merely in search of the truth here. If my benchmarks are eventually proven as inaccurate data, then I’ll eat my potty mouth… if that’s even possible.
I’ll be working on a more thorough test setup later. If Redis is indeed faster in my use case, I would switch in a heartbeat. I’m a whore like that.
Not sure I should trust your benchmark. Looking at your source code, I see a bunch of malloc and not a single line of dealloc. I don’t know whether you ran into any segfault issue during your benchmark run.
some projects become successful mostly by using niche keywords for paranoids, like “replication”, “faster”, “much faster”, “much more faster”, “more safety” and by throwing flames at existent software (e.g. the redis devs case against libevent). 99% of the people using or wanting to use replicated caches don’t even need it. sell them dreamcatchers for nightmares they will probably never have and you’ll be in business. replicated caches are the poor man’s solution to speed-and-reliability problem. too lazy to really understand what your data is about? just go replicated caching alltogether … here’s the redis string hack review.
@Doug: his code looks fine to me, he’s just being lazy. He doesn’t need to free his keys/randomvals until the bench is done. And when it does there’s no need to free because the program finishes and the kernel will take care of it. i don’t even know why am i explaning this to you but whatever. systoilet is probably a bofh. he loves to curse developers writing code this way but he also loves seeing himself doing the same thing. he also probably touches himself a lot.
I ran very simple test to set and get 100k unique keys and values against redis-2.2.2 and memcached. Both are running on linux VM(CentOS) and my client code(pasted below) runs on windows desktop.
Redis
Time taken to store 100000 values is = 18954ms
Time taken to load 100000 values is = 18328ms
Memcached
Time taken to store 100000 values is = 797ms
Time taken to retrieve 100000 values is = 38984ms
Jedis jed = new Jedis("localhost", 6379);
int count = 100000;
long startTime = System.currentTimeMillis();
for (int i=0; i<count; i++) {
jed.set("u112-"+i, "v51"+i);
}
long endTime = System.currentTimeMillis();
System.out.println("Time taken to store "+ count + " values is ="+(endTime-startTime)+"ms");
startTime = System.currentTimeMillis();
for (int i=0; i<count; i++) {
client.get("u112-"+i);
}
endTime = System.currentTimeMillis();
System.out.println("Time taken to retrieve "+ count + " values is ="+(endTime-startTime)+"ms");
I used the default redis.conf on the server side and tested this with Jedis client. I was surprised to see that redis took ~18 secs to store 100k values, as opposed to 797ms taken by memcached.
Anyone has better numbers for redis+java?
Tested Redis 2.4.0 – rc5 VS. Memcached 1.4.5
100k keys
REDIS done in 7s
MEMC done in 6s
Storage
REDIS 8174544 bytes = 7.79585266 megabytes
MEMC 800056 bytes = 0.762992859 megabytes
REDIS 90.3% BIGGER !!!
[note] 100k inserts
Anyone that would like to post their own benchmarks please do. But remember to test apples with apples. Keep the tests simple and very specific to avoid adding outside variables that would skew results. Try to avoid shit like using VMs as they’re too chaotic for timed tests like this.
At some point in the future when I have time, I’ll try to revisit this test again. (It seems to have become quite popular)
hi there.
do u nention that memcache works with all CPUs, but redis use just 1?
That’s a good point, no I haven’t. Although, I don’t believe that fact would have skewed my test results. I was using a single threaded client, against a single thread of Redis and a single thread in memcache.
can you do the benchmark against the latest stable redis release (2.4 i think)? i am wondering if you results still hold.
in my mind the biggest drawback of redis is that it isn’t really a distributed cache. although to be fair, they are working on it.
numan salati, can you define “distributed”? Because both redis and memcache have the same idea and there for memcache is also not distributed… ?
Wouldn’t it have been logical to make the line for Redis RED in the line charts? Maybe that came under the “annoying intricacies of OpenOffice” you’re referring to.
A “gay Ruby client”? Really? Are you 15?
Yeah, pretty much. Nobody uses Ruby, so that’s why it’s gay. So suck it, Tony!
greetings from pluto @theadminblagger…we don’t use ruby here. however, we’ve heard that the entire earth is addicted to ruby! #get-out-of-your-room-once-in-awhile
Well, maybe I’ll write up another article on Perl vs Ruby, and offend all of Japan. Oooooo mee sooo sorrry!
I really enjoyed the commentary when reading this post, even though it is very technical, you add some humor to it, I like that. Since this was posted around 08/09/2010 and at the end of the article you said “At some point in the future when I have time, I’ll try to revisit this test again.”
I was just going to ask if you have tried any test recently or if you could possibly, it would be nice to see an updated test
Thanks Jason. I haven’t had time to revisit the Redis test. It’s very low on my massive TODO list as an administrator. Presently though, I like Mongo. A few preliminary tests show it’s a similar speed to memcache, and it fits our architecture a little better.
Since it’s a document database, it’s not an apples-apples comparison with Redis. But, it works great. It’s redundant, horizontally scalable, and seems pretty speedy.
“Yet they have a gay Ruby client, fully featured?”
how about next time you omit the homophobia you bigoted fuck
butthurt itt
Tony, only a gaywad liberal douche would be offended by that statement. How about you leave your whiny comments in Nova Scotia, and just go back to having butt sex with young boys from Newfoundland.
admin =1 liberal douche = 0
I’m gay and not offended. Ruby is gay anyway, and I live in Japan… but pearl is worse.
damn I almost lost your blog in one of those windows ass wipes – buy new laptop – wipe again.
You always crack me up and get the point across. TO me your benchmark is still realistic. is how I still have my real world use case with PHP (threads? what’s that hehe)
Redis can be used as a memcached on steroids because is as fast as memcached but with a number of features more. Like memcached, Redis also supports setting timeouts to keys so that this key will be automatically removed when a given amount of time passes.
Source Memcached vs. Redis
We are developing a .net MVC EF application, we tried accessing memcache using client library which was 2 years old as no one update it. POC passed but we are using SignalR in our application, where it failed. So wondering if anyone is using memcache with .net application using latest client library ? If so, can anyone throw some light ?
” I instantly got a geek boner scanning the feature list.” Loved this ..and I love Redis as well.
What a gem 🙂 It’s such a shame more folks don’t know about this site, this had
everything I needed this morning.
“damn I almost lost your blog in one of those windows ass wipes – buy new laptop – wipe again.
You always crack me up and get the point across. TO me your benchmark is still realistic. is how I still have my real world use case with PHP (threads? what’s that hehe)”
But you know that Apache will spawn different processes when you run your php web app in the real world do you? So its not multi-threaded but multi-process with single threads which is quite the same when it comes to load for redis.
If you’re having trouble with code snippets on WordPress, you could just use the Code Snippets plugin. http://wordpress.org/plugins/code-snippets/
Pls checkout http://www.gridquorum.com/2013/08/redis-vs-bangdb-performance-comparison_16.html . Simple load test for redis and BangDB
Thorbjorn Olesen of Norway and Felipe Aguilar of
Chile split $285,657 for tying for third place at the 2013
Dubai Desert Classic. The lessons aren’t frightening for newbies;
they might create as many failures as they’d want, without ever becoming uncomfortable in front of pals,
loved ones, business associates, or special someone. The hands
and the clubhead travel on an inside-to-out path without any conscious effort on the
player’s part.
And here, ladies and gentlemen, you see the value of proper, deliberate and considered work (by the author of Redis), vs. the shallow, poorly thought-out and plain sloppy work by this dilettante who claims to be a scientist.
Thank you Salvatore, for your elegant and factual rebuttal of this waste of bytes.
The rebuttal in question is here: http://oldblog.antirez.com/post/redis-memcached-benchmark.html
Pros: 1. Clean design with no screws + lightweight. 2. Plug n Play i.e. your OS detects automatically. Online driver download/ manual installation not required. 3. Left & Right buttons with entire palm surface is made a bit rough providing better grip + Scrolling is quiet while L/R buttons are clicky. 4. Size fits for most unless you are a giant. Cons: 1. Scroll button wobbles a bit. Not an issue if you are careful. 2. Not for hardcore gamers/ users who mashes buttons a lot. 3. Plastic feels cheap.
I adore ur massive boobs!
21 Trackbacks/Pingbacks
[…] the sys/toilet guy writes a post benchmarking Redis vs Memcached and concludes that Memcached is faster and Redis doesn’t do so well at setup and […]
[…] При этом скорость работы по многим тестам (тест, тест2) даже превосходит Memcached. Это отличная идея для высоко […]
[…] is one comparison with Memcached, Redis vs Memcached the intro: take memcached and add a disk backing, replication, virtual memory, and some cool […]
[…] on behalf of two kinds of options,and tell us how to make a benchmark to redis and memcached ： 1.Redis vs Memcached 2.On Redis, Memcached, Speed, Benchmarks and The Toilet An update on the Memcached/Redis […]
[…] is also fast. It is comparable in speed to Memcached. A writer did a benchmark comparison, though Salvatore Sanfililipo the lead developer of Redis found the benchmark problematical […]
[…] easier to install (from my experience) and it has the same functions if not more. Memcache is a lot faster though, but has less features. But I have a php wrapper that checks on which environment it’s […]
[…] https://systoilet.wordpress.com/2010/08/09/redis-vs-memcached/ Share this:TwitterFacebookLike this:LikeBe the first to like this post. […]
[…] A comparison of Redis and Memcached can be found here: https://systoilet.wordpress.com/2010/08/09/redis-vs-memcached/ […]
[…] история про «Redis vs Memcached» — часть 1, часть 2, часть 3, часть […]
[…] comparison found that Memcached out-performs […]
[…] I chose a golden oldie, memcached, and an up and coming alternative, redis. There’s some lively discussion online about the performance differences between these two technologies. Ultimately I chose Redis […]
[…] Kulakov, Weavora Chief executive officerWeavora Chief executive officerCould be interesting – https://systoilet.wordpress.com/2…Comment Loading… • Share • Embed • Just now Add […]
[…] you don’t mind a crass writing style, Redis vs Memcached on the Systoilet blog is worth a read from a usability standpoint, but be sure to read the back […]
[…] you don’t mind a crass writing style, Redis vs Memcached on the Systoilet blog is worth a read from a usability standpoint, but be sure to read the back […]
[…] you don’t mind a crass writing style, Redis vs Memcached on the Systoilet blog is worth a read from a usability standpoint, but be sure to read the back […]
[…] Redis vs Memcached | /sys/toilet https://systoilet.wordpress.com/2010/08/09/redis-vs-memcached/ […]
[…] Redis vs Memcached […]
[…] Redis vs Memcached […]
[…] , 키 길이 및 제한을 검색하면 Redis vs. memcached 의 흥미로운 블로그 항목으로 인해 질문에 대한 답변을 시작할 수 […]
[…] thought at first that load testing was useless until I found this 2010 post: it clearly showed that Redis was falling behind Memcached in term of performances. Below a result […]
[…] on Redis, key length and limits nets me an interesting blog entry on Redis vs. memcached which may start to answer your question. The first response to that blog entry appears to have been […]