Login

Tag "gc"

Snippet List

Memory efficient Django Queryset Iterator

While checking up on some cronjobs at [YouTellMe](http://www.youtellme.nl/) we had some problems with large cronjobs that took way too much memory. Since Django normally loads all objects into it's memory when iterating over a queryset (even with .iterator, although in that case it's not Django holding it in it's memory, but your database client) I needed a solution that chunks the querysets so they're only keeping a small subset in memory. Example on how to use it: `my_queryset = queryset_iterator(MyItem.objects.all()) for item in my_queryset: item.do_something()` [More info on my blog](http://www.mellowmorning.com/2010/03/03/django-query-set-iterator-for-really-large-querysets/)

  • queryset
  • iterator
  • memory
  • gc
Read More

1 snippet posted so far.