Tag "bulk"

Compare objects list and get a list of object to inserted or updated

**Problem** You have an input `json` with which you will create a list of objects, you have to validate that the object will be created if it not exists, if exists determine whether to upgrade or discard depending of they have not undergone any changes. Solution 1) With the input `json` will be created the list of objects of the class that we insert or updatee 2) Read all fields in the database, using one of the fields as key to creating a dictionary with the objects in the database 3) Compare the objects and determine if it will be updated, inserted or discarded Django problem: by default only compares the level objects using the primary key (id). Compare field by field is the solution to determine if the object has changed. hints: The _state field is present in every object, and it will produce a random memory location, You can find cache fields so you need to remove these begins with underscore `_`. The fields excluded can be fk, and these fields produce field_id, so you will needs to exclude it class Country(models.Model): # country code 'MX' -> Mexico code = models.CharField(max_length=2) name = models.CharField(max_length=15) class Client(models.Model): # id=1, name=pedro, country.code=MX, rfc=12345 name = models.CharField(max_length=100) country = models.ForeignKey(Country) rfc = models.CharField(max_length=13) Country.objects.create(**{'code': 'MX', 'name': 'Mexico'}) # creating the country Client(**{'id':1, 'name':'pedro', 'country': country, 'rfc':12345}) # creating the client obj_db = Client.objects.get(id=1) country = Country.objects.get(code='MX') obj_no_db = Client(**{'id':1, 'name':'pedro', 'country': country, 'rfc':12345}) obj_db == obj_no_db # True obj_no_db = Client(**{'id':1, 'name':'pedro', 'country': country, 'rfc':1}) obj_db == obj_no_db # True # but isn't True because the rfc has change, how can compare field by field obj_db.rfc == obj_no_db.rfc # False, I was expected this result when compare obj_db == obj_no_db because they are not equal **Solution to compare field by field** _obj_1 = [(k,v) for k,v in obj_db.__dict__.items() if k != '_state'] _obj_2 = [(k,v) for k,v in obj_no_db.__dict__.items() if k != '_state'] _obj_1 == _obj_2 # False This is only for one object, and you can include in `__eq__` method in your model, but what happen if you need compare a list of object to bulk for insert or update with `django-bulk-update`. Well my snipped pretends solve that. so **How can use it.** obj_list = [<Object Client>, <Object Client>, <Object Client>, <Object Client>] get_insert_update(Client, 'id', obj_list) exclude_fields = ['country'] get_insert_update(Client, 'id', obj_list, exclude_fields=exclude_fields)

• models
• bulk

Thu 09 Jul 2015 Read More

Bulk Insert Manager

This is a small manager that just adds a "bulk_insert" function. This is very basic, I'm basically throwing it up here because it's simple and works for my current needs. Feedback on improvements (which I know would be a ton) are very welcome. Some known "gotchas": 1. This doesn't handle relationships. If, however, you want to do one-to-one or foreignkeys you'll have to use the actual table column name ('whatever_id' typically) 2. When using this I typically make a bulk_insert call every 500 iterations or so Some improvements that I think could be good: 1. Possibly just find the fields from the first object in the objs array and leave the fields argument as optional 2. Create a bulk_insert_from_file function and use LOAD DATA INFILE for mysql and whatever else supports it

• manager
• bulk
• insert
• bulk-insert

Wed 01 Jun 2011 Read More

django_bulk_save.py - defer saving of django models to bulk SQL commits

When called, this module dynamically alters the behaviour of model.save() on a list of models so that the SQL is returned and aggregated for a bulk commit later on. This is much faster than performing bulk writing operations using the standard model.save(). To use, simply save the code as django_bulk_save.py and replace this idiom: for m in model_list: # modify m ... m.save() # ouch with this one: from django_bulk_save import DeferredBucket deferred = DeferredBucket() for m in model_list: # modify m ... deferred.append(m) deferred.bulk_save() Notes: * - After performing a bulk_save(), the id's of the models do not get automatically updated, so code that depends on models having a pk (e.g. m2m assignments) will need to reload the objects via a queryset. * - post-save signal is not sent. see above. * - This code has not been thoroughly tested, and is not guaranteed (or recommended) for production use. * - It may stop working in newer django versions, or whenever django's model.save() related code gets updated in the future.

• sql
• tool
• dump
• save
• db
• bulk
• util

Sun 11 Jul 2010 Read More

Bulk Insert - updated 5/9/2008

This Update adds requested support for self referential fields. This is useful if you need to compute and store potentially hundreds or thousands of objects and relationships quickly. To perform the inserts it will hit the database 1 plus N/100 times per affected table and where N is the number of rows to be inserted. It will use INSERT or LOAD DATA INFILE on MySQL. Run this on your test database first and make sure all of your field defaults and null values are set appropriately as you could attempt to insert a NULL where it isn't allowed and end up with a partial insert. This code is reasonably well tested and has been used for database pre-loading and operations on live sites. My test suite, however, is focused on my own use cases. Any input, i.e. failures, for creating more tests would be appreciated. Lots of Details in the Doc String. Currently only MySQL, however there is some crude skeleton code to support other databases.

• mysql
• bulk
• insert
• load-data

Tue 01 Apr 2014 Read More