Keeping memcache consistent

 

As an afterthought someone decided at the last minute, that maybe the architect (me) should be on the architectural review of a product.

Normally for social networking web development, I allow for a little short term inconsistency. This is because only one user has access to modify a thing and that user isn’t likely to do two things at the same time. Because of this, concurrency is almost never a problem and. even if the data gets clobbered, the database at least is consistent and your objects are quickly fixed.

The problem with this particular project is that since a paid good is involved and many users will race to the same data store—inconsistencies can occur and they’d be more harmful than a goto statement. The solution proposed was to build a Java service to keep these eight pieces of data consistent. There was also a release plan in order to estimate the resource allocation for the new service under live site load.

Though late to the meeting, I opened my mouth and said, “You don’t need a Java service to do this. You can do it all in PHP and memcache.”

 

Why they didn’t think it was possible

Long before I joined the company, there was a system to prevent stampeding by having a lock key in memcache. This didn’t work out so well.

Stampeding is what occurs have 100,000 concurrent users and a memcache key for a fairly popular piece of data (say the block list for your web application, or the ad unit for the banner ads) expires (because of a version increment or an expiration). Tons of concurrent processes will see the data is missing and will stampede the database with the same request. Databases are slow—which is why we have memcache in the first place—and your site experiences a very nasty hiccup every time this happens.

One problem was that it is buggy. It used a nonce/semaphore to look a lot like this example . Well like that but if written by someone who just learned object-oriented program, then the locking code was added by someone else who had just read a design patterns book and then the bugs were closed by someone else who was so lazy that they prefer to patch problems on the user interface layer. It looked that way because it was written that way.

I always say that the sites the founders worked on before this had a business plan that was the internet-equivalent of a drive-by shooting .

Our codebase reflected that attitude.

So the reason we got burned by concurrency issues in memcache wasn’t because that it’s fundamentally broken, it was simply because it was easier to rewrite than to pretend that jenga-ing this stuff with a patch on a patch on a patch was going to magically make the code more stable.

And the reason I removed it in the rewrite wasn’t because that it’s that difficult to write, but rather that we were using this locking for every single object that we stored in memcache, even the less busy ones. Anything built in would be abused similarly. Better to not give the developers any rope they can use to hang the site with.

(In the case of stampedes, I figured the key is very likely to be too “hot” anyway. Instead I built a system to allow easy storage of hot keys in the shared memory cache of the webserver, instead of using memcache and needing a network call.)

What was the bug?

None of this is typically understood by anyone else. The reason is simply that the average engineer has been working here 1/20th the time I have. And those that have been working here half the time—pretty much all the rest—only know that I pulled the locking code out.

The bug in our code (and the above link) is that there is a race gap between the memcache::get() and memcache::set() . This is perfectly fine if all you want to do is prevent a stampede since only a few, even on a slow system like PHP, would be in the winners circle of that race. This sort of thing is really bad in the above case.

So what is the solution? The solution is to use a memcache command that is more atomic. The one that fits the bill is memcache::add() which, if a key is already added returns a failure condition.

The codez

For you script kiddies, here is the code. I don’t know if it works since I wrote it without any unit testing .

view source

print ?

01	// {{{ locked_mecache_update($memcache,$key,$updateFunction,$expiryTime,$waitUTime,$maxTries)

02

/**

03	* A function to do ensure only one thing can update a memcache at a time.

04

*

05	* Note that there are issues with the $expiryTime on memcache not being

06	* fine enough, but this is the best I can do. The idea behind this form

07	* of locking is that it takes advantage of the fact that

08	* {@link memcache_add()}'s are atomic in nature.

09

*

10	* It would be possible to be a more interesting limiter (say that limits

11	* updates to no more than 1/second) simply by storing a timestamp or

12	* something of that nature with the lock key (currently stores "1") and

13	* not deleitng the memcache entry.

14

*

15	* @package TGIFramework

16	* @subpackage functions

17	* @copyright 2009 terry chay

18	* @author terry chay <[email protected]>

19	* @param $memcache memcache the memcache object

20	* @param $key string the key to do the update on

21	* @param $updateFunction mixed the function to call that accepts the data

22	*  from memcache and modifies it (use pass by reference).

23	* @param $expiryTime integer time in seconds to allow the key to last before

24	*  it will expire. This should only happen if the process dies during update.

25	*  Choose a number big enough so that $updateFunction will take much less

26	*  time to execute.

27	* @param $waitUTime integer the amount of time in microseconds to wait before

28	*  checking for the lock to release

29	* @param $maxTries integer maximum number of attempts before it gives up

30	*  on the locks. Note that if $maxTries is 0, then it will RickRoll forever

31	*  (never give up). The default number ensures that it will wait for three

32	*  full lock cycles to crash before it gives up also.

33	* @return boolean success or failure

34

*/

35	function locked_memcache_update( $memcache , $key , $updateFunction , $expiryTime =3, $waitUtime =101, $maxTries =100000)

36

{

37	$lock = 'lock:' . $key ;

38

39	// get the lock {{{

40	if ( $maxTries >0) {

41	for ( $tries =0; $tries < $maxTries ; ++ $tries ) {

42	if ( $memcache ->add( $lock ,1,0, $expiryTime )) { break ; }

43	usleep( $waitUtime );

44

}

45	if ( $tries == $maxTries ) {

46	// handle failure case (use exceptions and try-catch if you need to be nice)

47	trigger_error(sprintf( 'Lock failed for key: %s' , $key ), E_USER_NOTICE);

48	return false;

49

}

50

} else {

51	while (! $memcache ->add( $lock ,1,0, $expiryTime )) {

52	usleep( $waitUtime );

53

}

54

}

55

// }}}

56	// modify data in cache {{{

57	$data = $memcache ->get( $key , $flag );

58	call_user_func( $updateFunction , $data ); // update data

59	$memcache ->set( $key , $data , $flag );

60

// }}}

61	// clear the lock

62	$memcache -> delete ( $lock ,0);

63	return true;

64

}

65

// }}}

(Yes, I this commenting is typical when I code, hope I could say the same for you.) The reason it’s a function is so that an engineer has to do work to use it. If they got it for free, they’d abuse it—or, at least that was the worry.

If you need to do locking on the database, then you would have the $updateFunction nest something that will handle a database update. You might want to up the $expiryTime too, but you probably won’t need to—I just chose 3 because Touge did in his original post.

 

 

from: http://terrychay.com/article/keeping-memcache-consistent.shtml