Login

Batch querysets

Author:
jkocherhans
Posted:
November 6, 2008
Language:
Python
Version:
1.0
Score:
9 (after 9 ratings)

Most of the time when I need to iterate over Whatever.objects.all() in a shell script, my machine promptly reminds me that sometimes even 4GB isn't enough memory to prevent swapping like a mad man, and bringing my laptop to a crawl. I've written 10 bazillion versions of this code. Never again.

Caveats

Note that you'll want to order the queryset, as ordering is not guaranteed by the database and you might end up iterating over some items twice, and some not at all. Also, if your database is being written to in between the time you start and finish your script, you might miss some items or process them twice.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
def batch_qs(qs, batch_size=1000):
    """
    Returns a (start, end, total, queryset) tuple for each batch in the given
    queryset.
    
    Usage:
        # Make sure to order your querset
        article_qs = Article.objects.order_by('id')
        for start, end, total, qs in batch_qs(article_qs):
            print "Now processing %s - %s of %s" % (start + 1, end, total)
            for article in qs:
                print article.body
    """
    total = qs.count()
    for start in range(0, total, batch_size):
        end = min(start + batch_size, total)
        yield (start, end, total, qs[start:end])

More like this

  1. Add custom fields to the built-in Group model by jmoppel 1 month, 2 weeks ago
  2. Month / Year SelectDateWidget based on django SelectDateWidget by pierreben 5 months ago
  3. Python Django CRUD Example Tutorial by tuts_station 5 months, 2 weeks ago
  4. Browser-native date input field by kytta 7 months ago
  5. Generate and render HTML Table by LLyaudet 7 months, 1 week ago

Comments

DocTiger (on November 11, 2008):

Saved my day...

#

mingdongt (on December 21, 2016):

Got a question, is ordering necessary?

#

Please login first before commenting.