The function slices a queryset into smaller querysets containing chunk_size objects and then yield them. It is used to avoid memory error when processing huge queryset, and also database error due to that the database pulls whole table at once. Concurrent database modification wouldn't make some entries repeated or skipped in this process.
`dumpdata` without `MemoryErrors`, with progress notification. Most of the real work is done by snippets [1400](http://www.djangosnippets.org/snippets/1400/) and [1401](http://www.djangosnippets.org/snippets/1401/). ./manage.py dumpdata_stream --format=xml > big_dump.xml This is basically the stock Django `dumpdata` with a few modifications. Django devs: it's hard to reuse parts of most Django management commands. A little refactoring could go a long way.
Trying `./manage.py dumpdata` on a huge database and getting `MemoryError`s? Here's part of your solution. [Snippet 1400](http://www.djangosnippets.org/snippets/1400/) provides a queryset_foreach utility that we've found very useful. This snippet uses it on a serializer that can output to a stream, such as the XML serializer. Management command coming momentarily...
3 snippets posted so far.