Login

Batch querysets

Author:
jkocherhans
Posted:
November 6, 2008
Language:
Python
Version:
1.0
Score:
9 (after 9 ratings)

Most of the time when I need to iterate over Whatever.objects.all() in a shell script, my machine promptly reminds me that sometimes even 4GB isn't enough memory to prevent swapping like a mad man, and bringing my laptop to a crawl. I've written 10 bazillion versions of this code. Never again.

Caveats

Note that you'll want to order the queryset, as ordering is not guaranteed by the database and you might end up iterating over some items twice, and some not at all. Also, if your database is being written to in between the time you start and finish your script, you might miss some items or process them twice.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
def batch_qs(qs, batch_size=1000):
    """
    Returns a (start, end, total, queryset) tuple for each batch in the given
    queryset.
    
    Usage:
        # Make sure to order your querset
        article_qs = Article.objects.order_by('id')
        for start, end, total, qs in batch_qs(article_qs):
            print "Now processing %s - %s of %s" % (start + 1, end, total)
            for article in qs:
                print article.body
    """
    total = qs.count()
    for start in range(0, total, batch_size):
        end = min(start + batch_size, total)
        yield (start, end, total, qs[start:end])

More like this

  1. Stuff by NixonDash 1 month ago
  2. Add custom fields to the built-in Group model by jmoppel 3 months, 1 week ago
  3. Month / Year SelectDateWidget based on django SelectDateWidget by pierreben 6 months, 3 weeks ago
  4. Python Django CRUD Example Tutorial by tuts_station 7 months ago
  5. Browser-native date input field by kytta 8 months, 2 weeks ago

Comments

DocTiger (on November 11, 2008):

Saved my day...

#

mingdongt (on December 21, 2016):

Got a question, is ordering necessary?

#

Please login first before commenting.