Batch querysets

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
def batch_qs(qs, batch_size=1000):
    """
    Returns a (start, end, total, queryset) tuple for each batch in the given
    queryset.
    
    Usage:
        # Make sure to order your querset
        article_qs = Article.objects.order_by('id')
        for start, end, total, qs in batch_qs(article_qs):
            print "Now processing %s - %s of %s" % (start + 1, end, total)
            for article in qs:
                print article.body
    """
    total = qs.count()
    for start in range(0, total, batch_size):
        end = min(start + batch_size, total)
        yield (start, end, total, qs[start:end])

More like this

  1. Memory efficient Django Queryset Iterator by WoLpH 4 years, 1 month ago
  2. several_random template filter by hawkeye 6 years, 11 months ago
  3. Sophisticated order_by sorting by mawi 1 year, 4 months ago
  4. Database cleanup by skyjur 3 years, 9 months ago
  5. Compact list_filter by onlinehero 4 years, 3 months ago

Comments

DocTiger (on November 11, 2008):

Saved my day...

#

(Forgotten your password?)