Login

Cache Manager

Author:
jerzyk
Posted:
December 10, 2007
Language:
Python
Version:
.96
Score:
1 (after 1 ratings)

I had a problem: too many fetches from the DB. So, how to reduce load on the database without major changes to the code? Cache Manager is the answer. I've managed to reduce number of DB hits as much as 80% in some cases (dictionaries, complex relations). It is using standard cache mechanisms. I'm using it with mathopd.

This is a very simple solution, instead of standard Manager, put this in your model definition as: objects = CacheManager() Then everythere elase in the code instead of all() or get(...) call all_cached() or get_cached().

I've kept original methods intact, to have an dual access, when you really, really must have frest data from the DB, and you can't wait for cache to expire.

This is much easier to work with, then manually getting fetched data from the cache.No change to your existing code 9except model) and voila!

Additionally if you have some data, you would like to store with your serialized object (e.g. related data, dynamic dictionaries), you can do this in the model method '_init_instance_cache').

Drop me an email if you find this useful. :)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
from django.core.cache import cache
from django.db import models

class CacheManager(models.Manager):

    def _getCacheKey(self, id, subset='s'):
        return "%s_%s_%s" % (self.model._meta.object_name, subset, id)

    def all_cached(self):
        if self.__class__.__name__ == 'RelatedManager':
            key = "".join(["%s%s" % el for el in self.core_filters.items()])
            cacheKey = self._getCacheKey(0, key)
        else:
            cacheKey = self._getCacheKey(0, 'all')
        qs = cache.get(cacheKey)
        if qs is None:
            qs = list(self.get_query_set())
            for element in qs:
                if hasattr(element,'_init_instance_cache'):
                    element._init_instance_cache()
            cache.set(cacheKey, qs)
        return qs

    def get_cached(self, *args, **kwargs):
        cacheKey = cat = None
        if self.__class__.__name__ == 'RelatedManager':
            key = "".join(["%s%s" % el for el in self.core_filters.items()])
        else:
            key = ""
        lst = map(lambda x:(x[0],x[1].pk if isinstance(x[1],models.Model) else str(x[1])),kwargs.items())
        key += "_" + "_".join(list(map(unicode,args))+list(map(lambda x:"%s:%s"%x,lst)))
        key = key.replace(" ","")
        cacheKey = self._getCacheKey(key)

        # check if we have a result already ached
        if cacheKey is not None:
            element = cache.get(cacheKey)

        # no results, try to get it from cache
        if element is None:
            element = self.get_query_set().get(*args, **kwargs)
            if hasattr(element,'_init_instance_cache'):
                element._init_instance_cache()
            # if this should be cached, write it to the cache
            if cacheKey is not None:
                cache.set(cacheKey, element)

        return element

More like this

  1. Template tag - list punctuation for a list of items by shapiromatron 9 months, 3 weeks ago
  2. JSONRequestMiddleware adds a .json() method to your HttpRequests by cdcarter 10 months ago
  3. Serializer factory with Django Rest Framework by julio 1 year, 4 months ago
  4. Image compression before saving the new model / work with JPG, PNG by Schleidens 1 year, 5 months ago
  5. Help text hyperlinks by sa2812 1 year, 6 months ago

Comments

dharris (on December 13, 2007):

Can you give an example of how you use "_init_instance_cache"?

#

jerzyk (on December 17, 2007):

Sure, in model class:

    class my_model(Model):
       something = ForeignKey()

       .....
       _cached_data = None

       def get_processed_data(self)
           if self._cached_data is None:
               long_and_extensive_process
               self._cached_data = result
           return self._cached_data

        def _init_instance_cache(self):
            # just to fill field before it will be serialized
            nil = self.get_processed_data()

This is especially useful when you have chain of 2 or more foreign keyb between your models.

#

Please login first before commenting.