Login

streaming serializer

Author:
kcarnold
Posted:
March 29, 2009
Language:
Python
Version:
1.0
Score:
1 (after 1 ratings)

Trying ./manage.py dumpdata on a huge database and getting MemoryErrors? Here's part of your solution.

Snippet 1400 provides a queryset_foreach utility that we've found very useful. This snippet uses it on a serializer that can output to a stream, such as the XML serializer.

Management command coming momentarily...

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
from snippet 1400 import queryset_foreach

def serialize_qs4e(serializer, querysets, stream, **options):
    qs4e_options = {'transaction': False, 'batch_size': 50}
    for opt in ['batch_size', 'progress_callback', 'transaction']:
        val = options.pop(opt, None)
        if val is not None: qs4e_options[opt] = val

    serializer.options = options
    serializer.options['stream'] = stream
    serializer.stream = stream
    serializer.selected_fields = options.get("fields")

    def serialize_object(obj):
        #import pdb; pdb.set_trace()

        serializer.start_object(obj)
        for field in obj._meta.local_fields:
            if field.serialize:
                if field.rel is None:
                    if serializer.selected_fields is None or field.attname in serializer.selected_fields:
                        serializer.handle_field(obj, field)
                else:
                    if serializer.selected_fields is None or field.attname[:-3] in serializer.selected_fields:
                        serializer.handle_fk_field(obj, field)
        for field in obj._meta.many_to_many:
            if field.serialize:
                if serializer.selected_fields is None or field.attname in serializer.selected_fields:
                    serializer.handle_m2m_field(obj, field)
        serializer.end_object(obj)

    serializer.start_serialization()
    for queryset in querysets:
        queryset_foreach(queryset, serialize_object, **qs4e_options)
    serializer.end_serialization()

More like this

  1. Template tag - list punctuation for a list of items by shapiromatron 10 months, 3 weeks ago
  2. JSONRequestMiddleware adds a .json() method to your HttpRequests by cdcarter 11 months ago
  3. Serializer factory with Django Rest Framework by julio 1 year, 5 months ago
  4. Image compression before saving the new model / work with JPG, PNG by Schleidens 1 year, 6 months ago
  5. Help text hyperlinks by sa2812 1 year, 7 months ago

Comments

Please login first before commenting.