When executing custom sql, the temptation is to use fetchall or fetchone, since the API for fetchmany is a bit awkward. (fetchall makes all records resident in client memory at once; fetchone takes a network round-trip to the DB for each record.)
This snippet, hoisted from django.db.models.sql.query.results_iter, presents a nice, simple iterator over multiple fetchmany calls which hits a sweet spot of minimizing memory and network usage.
1 2 3 4 5 6 7 8 9 10 11 12 | def iter_fetchmany(cursor, size=100):
"""
Handles the awkwardness of iterating using fetchmany, and exposes all records as a (generated) iterable.
Assumes django.db is the connection for the given cursor. FIXME for multi-db support.
"""
from django.db import connection
rowset_iter = iter((lambda: cursor.fetchmany(size)),
connection.features.empty_fetchmany_value)
for rows in rowset_iter:
for row in rows:
yield row
|
More like this
- get_object_or_none by azwdevops 1 month, 1 week ago
- Mask sensitive data from logger by agusmakmun 3 months ago
- Template tag - list punctuation for a list of items by shapiromatron 1 year, 5 months ago
- JSONRequestMiddleware adds a .json() method to your HttpRequests by cdcarter 1 year, 5 months ago
- Serializer factory with Django Rest Framework by julio 2 years ago
Comments
Please login first before commenting.