Django snippets: Overcome the bulk_create() size limitation using SQLite

def safe_bulk_create(objs):
    """Wrapper to overcome the size limitation of standard bulk_create()"""
    if objs:
        BULK_SIZE = 900/len(objs[0].__class__._meta.fields)
        for i in range(0,len(objs),BULK_SIZE):
            objs[0].__class__.objects.bulk_create(objs[i:i+BULK_SIZE])

More like this

Bulk Insert - updated 5/9/2008 by coolie 5 years, 7 months ago
Template tag to create a list from one or more variables and/or literals by davidchambers 2 years, 8 months ago
Password Validation - Require Letters and Numbers - no regex by watchedman 1 year, 8 months ago
PositionField by jpwatts 4 years, 10 months ago
Significant digits filter by joelegner 4 years, 2 months ago

As of django 1.4, the newly introduced bulk_create() function has a strong limitation when using SQLite. Namely it causes an error if you want to create more than 999/F objects, where F is the number of fields in the object type.

The above safe_bulk_create(objs) function solves this issue by splitting the list of objects to appropriately sized bulks.

This solution also works fine with other db backends, and according to my experiments, it causes no significant overhead comparing to using bulk_create() directly.

For more details on the issue, see https://code.djangoproject.com/ticket/17788

Thanks to charettes for pointing out how to calculate the number of fields in an object.

Author:: alpar
Posted:: March 28, 2012
Language:: Python
Django Version:: Not specified
Tags:: bulk_create
Score:: 0 (after 0 ratings)

Django snippets

Overcome the bulk_create() size limitation using SQLite

More like this

Tools

Comments

Snippets

About