Login

Improved generic foreign key manager 2

Author:
Nomalz
Posted:
October 25, 2009
Language:
Python
Version:
1.1
Tags:
foreignkey generic manager query tuning
Score:
3 (after 3 ratings)

This is an improvement on snippet 1079. Please read its description and this blog post for any information.

This is a manager for handling generic foreign key. Generic foreign objects of the same type are fetched together in order to reduce the number of SQL queries.

To use, just assign an instance of GFKManager as the objects attribute of a model that has generic foreign keys. Then: MyModelWithGFKs.objects.filter(...).fetch_generic_relations()

The generic related items will be bulk-fetched to minimize the number of queries.

Improvement: Problem I had with previous version from snippet 1079 : if two or more items shares the same generic foreign object, then only the first one is cached. Next ones generates new unwanted SQL queries. I solved this problem by putting all the needed foreign objects in a temporary data_map dictionary. Then, the objects are distributed to every items, so that if two items shares the same foreign object, it will only be fetched once.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
from django.db.models.query import QuerySet
from django.db.models import Manager
from django.contrib.contenttypes.models import ContentType
from django.contrib.contenttypes.generic import GenericForeignKey

class GFKManager(Manager):
    """
    A manager that returns a GFKQuerySet instead of a regular QuerySet.

    """
    def get_query_set(self):
        return GFKQuerySet(self.model)

class GFKQuerySet(QuerySet):
    """
    A QuerySet with a fetch_generic_relations() method to bulk fetch
    all generic related items.  Similar to select_related(), but for
    generic foreign keys.

    Based on http://www.djangosnippets.org/snippets/984/
    Firstly improved at http://www.djangosnippets.org/snippets/1079/

    """
    def fetch_generic_relations(self):
        qs = self._clone()

        gfk_fields = [g for g in self.model._meta.virtual_fields if isinstance(g, GenericForeignKey)]
        
        ct_map = {}
        item_map = {}
        data_map = {}
        
        for item in qs:
            for gfk in gfk_fields:
                ct_id_field = self.model._meta.get_field(gfk.ct_field).column
                #print "ct_id=%s" % getattr(item, ct_id_field)
                #print "item_id=%s" % getattr(item, gfk.fk_field)
                #print "%s %s" % (ct_id_field, getattr(item, ct_id_field))
                ct_map.setdefault(
                    (getattr(item, ct_id_field)), {}
                    )[getattr(item, gfk.fk_field)] = (gfk.name, item.id)
            item_map[item.id] = item

        for (ct_id), items_ in ct_map.items():
            if (ct_id):
                ct = ContentType.objects.get_for_id(ct_id)
                for o in ct.model_class().objects.select_related().filter(id__in=items_.keys()).all():
                    (gfk_name, item_id) = items_[o.id]
                    data_map[(ct_id, o.id)] = o

        for item in qs:
            for gfk in gfk_fields:
                if (getattr(item, gfk.fk_field) != None):
                    ct_id_field = self.model._meta.get_field(gfk.ct_field).column
                    setattr(item, gfk.name, data_map[(getattr(item, ct_id_field), getattr(item, gfk.fk_field))])

        return qs

More like this

  1. improved generic foreign key manager by carljm 6 years, 8 months ago
  2. Manager method for limiting GenericForeignKey queries by zerok 6 years, 9 months ago
  3. DRY with common model fields (another way) by jmrbcu 7 years, 10 months ago
  4. Improved Pickled Object Field by taavi223 5 years, 9 months ago
  5. Querying on existence of a relationship by ubernostrum 7 years, 10 months ago

Comments

Please login first before commenting.