memcached opinions

[Luke Macken]

On Wed, Jul 01, 2009 at 05:50:12PM -0700, Toshio Kuratomi wrote:
> On 07/01/2009 05:39 PM, Mike McGrath wrote:
> > On Wed, 1 Jul 2009, Stephen John Smoogen wrote:
> > 
> >> On Wed, Jul 1, 2009 at 6:08 PM, Mike McGrath<mmcgrath at redhat.com> wrote:
> >>> On Wed, 1 Jul 2009, Stephen John Smoogen wrote:
> >>>
> >>>> On Wed, Jul 1, 2009 at 3:33 PM, Mike McGrath<mmcgrath at redhat.com> wrote:
> >>>>> I'm not sure if we have any memcached experience on the list but I thought
> >>>>> I'd ask.  Can anyone explain this:
> >>>>>
> >>>>> http://pastebin.ca/1481219
> >>>>>
> >>>>> Notice how memcached1 has a much higher hit rate and memcached2 has a much
> >>>>> lower hit rate?
> >>>>>
> >>>>
> >>>> The time for memcached1 is 5x less than memcached2 being up. That can
> >>>> have an effect on caching as right after a system comes up its rates
> >>>> are usually much higher and then over time fall off (iirc). I think it
> >>>> would take bringing both up at the same time to figure out if there is
> >>>> a true disparity over caching.
> >>>>
> >>>
> >>> I thought that exact same thing :)
> >>>
> >>> memcahed1:
> >>> STAT uptime 9143
> >>> STAT get_hits 311736
> >>> STAT get_misses 11255
> >>>
> >>> memcached2:
> >>> STAT uptime 9144
> >>> STAT get_hits 49679
> >>> STAT get_misses 11116
> >>>
> >>
> >> Now that shows something not kosher. My guess is some app is not
> >> talking to both? What apps use memcached for what?
> >>
> > 
> > I was just talking to ricky about this a bit in IRC.  So here's the scoop.
> > 
> > We've got mediawiki using memcached for a couple of things, including
> > session data (which is weird and wrong but fast).
> > 
> > The recent addition to the group is Fedora community, specifically in it's
> > implementation of beaker.  I'm going to get ahold of luke tomorrow to
> > verify and test some stuff but I think this line in the config:
> > 
> > beaker.cache.url = memcached1;memcached
> > 
> > I'm not sure how beaker reads that, but I suspect it might be only sending
> > information to memcached1 and ignoring memcached2 altogether.  If this
> > theory holds it'd explain why memcached1 not only has a higher request
> > rate but also a higher hit rate because I suspect fedoracommunity requests
> > some of the same info over and over again compared to the wiki which
> > probably has a broader data pool it pulls from.
> > 
> easy test: reverse that:
> 
> beaker.cache.url = memcached2;memcached1
> 
> and see if the cache hit ratio reverses itself.
> 
> -Toshio

So, I've been poking at this a little bit lately, and I'm thinking there are
some problems with the way Fedora Community is utilizing it's Beaker cache &
memcached.

The fact that something is wrong with the caching setup becomes obvious when
playing with the Bugzilla grid *should* cache the first 5 pages, and be very
snappy, as it is in my local instance.  However, that is not the case, which
makes me think it's not hitting our memcached servers.

I wrote up a little test script on app1::

    from beaker.cache import CacheManager

    memcached1 = CacheManager(type='ext:memcached', url='memcached1', lock_dir='.')
    memcached2 = CacheManager(type='ext:memcached', url='memcached2', lock_dir='.')

    def createfunc(*args):
            print "createfunc(%s)" % (args,)

    def get_value(cache, value):
            cache_1 = memcached1.get_cache(cache)
            cache_2 = memcached2.get_cache(cache)
            result1 = cache_1.get_value(key=value, createfunc=createfunc)
            result2 = cache_2.get_value(key=value, createfunc=createfunc)
            print "memcached1[%s][%s] = %s" % (cache, value, result1)
            print "memcached2[%s][%s] = %s" % (cache, value, result2)
            return result1, result2

    get_value('fedoracommunity_alerts_global', 'today')
    get_value('bodhi', 'dashboard_None')

Which produces::

    memcached1[fedoracommunity_alerts_global][today] = None
    memcached2[fedoracommunity_alerts_global][today] = None
    memcached1[bodhi][dashboard_None] = None
    memcached2[bodhi][dashboard_None] = None

I also tried using netcat to query for these by hand, to no avail.

So, it looks like we need to look a bit deeper into what is going on here.
Either I'm Doing It Wrong with Beaker, or we're hitting a bug somewhere.

luke