Login

Batch querysets

Author:
jkocherhans
Posted:
November 6, 2008
Language:
Python
Version:
1.0
Tags:
queryset batch
Score:
9 (after 9 ratings)

Most of the time when I need to iterate over Whatever.objects.all() in a shell script, my machine promptly reminds me that sometimes even 4GB isn't enough memory to prevent swapping like a mad man, and bringing my laptop to a crawl. I've written 10 bazillion versions of this code. Never again.

Caveats

Note that you'll want to order the queryset, as ordering is not guaranteed by the database and you might end up iterating over some items twice, and some not at all. Also, if your database is being written to in between the time you start and finish your script, you might miss some items or process them twice.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
def batch_qs(qs, batch_size=1000):
    """
    Returns a (start, end, total, queryset) tuple for each batch in the given
    queryset.
    
    Usage:
        # Make sure to order your querset
        article_qs = Article.objects.order_by('id')
        for start, end, total, qs in batch_qs(article_qs):
            print "Now processing %s - %s of %s" % (start + 1, end, total)
            for article in qs:
                print article.body
    """
    total = qs.count()
    for start in range(0, total, batch_size):
        end = min(start + batch_size, total)
        yield (start, end, total, qs[start:end])

More like this

  1. Memory efficient Django Queryset Iterator by WoLpH 5 years, 5 months ago
  2. several_random template filter by hawkeye 8 years, 2 months ago
  3. Sophisticated order_by sorting by mawi 2 years, 7 months ago
  4. Database cleanup by skyjur 5 years ago
  5. Compact list_filter by onlinehero 5 years, 6 months ago

Comments

DocTiger (on November 11, 2008):

Saved my day...

#

Please login first before commenting.