Login

Bulk Insert Manager

Author:
ocdcoder
Posted:
February 18, 2011
Language:
Python
Version:
1.2
Tags:
manager bulk insert bulk-insert
Score:
0 (after 0 ratings)

This is a small manager that just adds a "bulk_insert" function. This is very basic, I'm basically throwing it up here because it's simple and works for my current needs. Feedback on improvements (which I know would be a ton) are very welcome.

Some known "gotchas":

  1. This doesn't handle relationships. If, however, you want to do one-to-one or foreignkeys you'll have to use the actual table column name ('whatever_id' typically)

  2. When using this I typically make a bulk_insert call every 500 iterations or so

Some improvements that I think could be good:

  1. Possibly just find the fields from the first object in the objs array and leave the fields argument as optional

  2. Create a bulk_insert_from_file function and use LOAD DATA INFILE for mysql and whatever else supports it

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
from django.db import models
from django.db import connection

class BulkManager(models.Manager):
        def bulk_insert(self, fields, objs):
                qn = connection.ops.quote_name
                cursor = connection.cursor()

                flds = ', '.join([qn(f) for f in fields])
                values_list = [ r[f] for r in objs for f in fields]
                arg_string = ', '.join([u'(' + ', '.join(['%s']*len(fields)) + ')'] * len(objs))                   
                sql = "INSERT INTO %s (%s) VALUES %s" % (self.model._meta.db_table, flds, arg_string,)             
                cursor.execute(sql, values_list)

More like this

Comments

visik7 (on February 21, 2011):

what is the purpose of this thing?

#

Please login first before commenting.