- Author:
- jkocherhans
- Posted:
- November 6, 2008
- Language:
- Python
- Version:
- 1.0
- Score:
- 9 (after 9 ratings)
Most of the time when I need to iterate over Whatever.objects.all() in a shell script, my machine promptly reminds me that sometimes even 4GB isn't enough memory to prevent swapping like a mad man, and bringing my laptop to a crawl. I've written 10 bazillion versions of this code. Never again.
Caveats
Note that you'll want to order the queryset, as ordering is not guaranteed by the database and you might end up iterating over some items twice, and some not at all. Also, if your database is being written to in between the time you start and finish your script, you might miss some items or process them twice.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | def batch_qs(qs, batch_size=1000):
"""
Returns a (start, end, total, queryset) tuple for each batch in the given
queryset.
Usage:
# Make sure to order your querset
article_qs = Article.objects.order_by('id')
for start, end, total, qs in batch_qs(article_qs):
print "Now processing %s - %s of %s" % (start + 1, end, total)
for article in qs:
print article.body
"""
total = qs.count()
for start in range(0, total, batch_size):
end = min(start + batch_size, total)
yield (start, end, total, qs[start:end])
|
More like this
- Template tag - list punctuation for a list of items by shapiromatron 11 months, 2 weeks ago
- JSONRequestMiddleware adds a .json() method to your HttpRequests by cdcarter 11 months, 3 weeks ago
- Serializer factory with Django Rest Framework by julio 1 year, 6 months ago
- Image compression before saving the new model / work with JPG, PNG by Schleidens 1 year, 7 months ago
- Help text hyperlinks by sa2812 1 year, 7 months ago
Comments
Saved my day...
#
Got a question, is ordering necessary?
#
Please login first before commenting.