- November 6, 2008
- queryset batch
- 8 (after 8 ratings)
Most of the time when I need to iterate over Whatever.objects.all() in a shell script, my machine promptly reminds me that sometimes even 4GB isn't enough memory to prevent swapping like a mad man, and bringing my laptop to a crawl. I've written 10 bazillion versions of this code. Never again.
Note that you'll want to order the queryset, as ordering is not guaranteed by the database and you might end up iterating over some items twice, and some not at all. Also, if your database is being written to in between the time you start and finish your script, you might miss some items or process them twice.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
def batch_qs(qs, batch_size=1000): """ Returns a (start, end, total, queryset) tuple for each batch in the given queryset. Usage: # Make sure to order your querset article_qs = Article.objects.order_by('id') for start, end, total, qs in batch_qs(article_qs): print "Now processing %s - %s of %s" % (start + 1, end, total) for article in qs: print article.body """ total = qs.count() for start in range(0, total, batch_size): end = min(start + batch_size, total) yield (start, end, total, qs[start:end])