I thought it would be a good idea to add a persistent WordPress object cache to this blog, to reduce page load times and increase resilience to spikes in traffic. So I added the WordPress Memcached Object Cache drop-in (AKA wp-memcached), and enabled the legacy App Engine Memcache API. However, adding the Memcache object cache increased page load times by 10s or more! Looking into it further, while the App Engine diagnostics reported a high (90%+) cache hit ratio, the drop-in was getting 0% cache hits. The drop-in was adding keys to the cache only for them not to be found in subsequent reads. There were also hundreds of SQL queries that did not occur with WordPress’s default (request-scoped) object cache.
With some additional debugging and perusing of the source code of PECL’s official Memcached module, the App Engine SDK, and the wp-memcached code, I figured out what was going on and how to fix it.
wp-memcached assumes that Memcache::get will set its $flags reference parameter to some non-false value if the lookup succeeds. This is fairly reasonable, given that Memcache::get documents the parameter as an int, so even if it was 0 it would not compare === to false.
However, App Engine’s Memcache replacement doesn’t use pass-by-reference for the $flags parameter to get. It also ignores the $flag parameter on the various write operations like add, set, and replace.
This combination of behaviours causes the WordPress object cache to treat every single read from the persistent cache as a miss. In addition, wp-memcached will not fall back to a local (PHP variable) cache, which means that WordPress will run the expensive work (e.g. SQL queries) and Memcache writes after every cache lookup. This results in severe performance degradation compared to the default WordPress object cache, which will only run the expensive work once per request.
Fixes
The most pragmatic fix for me was to rewrite the wp-memcached drop-in so that it:
- Doesn’t rely on the
$flagsparameter ofMemcache::getto detect a lookup success. Note this means the lookup of afalsevalue will appear to fail. - Aggressively use a local cache for all reads, including the conditional-write behaviours for
addandreplace. Not only does this improve performance (a PHP array lookup is much faster than a Memcache RPC, not to mention more reliable in the presence of eventual consistency), but it also gives sane behaviour for writingfalsevalues. I left a scaled-down proposal with the maintainers.
A better fix would be to modify the App Engine Memcache API to return flag values. This comes with some nuance – the flags are used internally by the App Engine PHP SDK to store the type, which could be undesirable to expose to clients. However, this is similar to the PECL Memcache module behaviour, which states “the lowest byte of the int is reserved for pecl/memcache internal usage (e.g. to indicate compression and serialization status)”. I raised a proposal for Memcache::get to set $flags with the maintainers.
Another alternative would be to use a different store from the legacy App Engine Memcache altogether. Google recommends migrating to Redis. However, Redis is very expensive (a minumum of $43 USD/month) compared to Memcache (which is available in the free tier), so this option is undesirable.