utf8-friendly dumpdata management command (no escape symbols) #4
The version of snippet that works with Django 1.9
- fixtures
- management
- dumpdata
The version of snippet that works with Django 1.9
This is a revised version of https://djangosnippets.org/snippets/2921/
The version of [snippet](http://djangosnippets.org/snippets/2397/) that works with Django 1.5. Kudos to [kmike](http://djangosnippets.org/users/kmike/) for the original snippet.
Inspired and based on https://djangosnippets.org/snippets/918/ Improvements: - Supports natural keys - Uses Django's Collector so hopefully follows reverse relationships - Fixes problem when you don't specify a slice - PEP8 etc.
South documentation [contains a description](http://south.readthedocs.org/en/0.7.6/fixtures.html#fixtures-from-migrations) of the way you can load fixtures inside the data-migrations. def forwards(self, orm): from django.core.management import call_command call_command("loaddata", "my_fixture.json") It seems pretty clear and easy, but in fact it does not work the way you expect from south migrations, because the fixture loading does not engage the **orm** object. So, it allows **loaddata** management command to use standard models loading mechanism, and it would provide the most recent version of the models, obviously, which may not correspond to the schema of the fixture`s data. To be ensured that migration will use appropriate version of the models for fixture loading you could use code like follows: class Migration(DataMigration): def forwards(self, orm): load_fixture('my_fixture.json', orm) class Migration(DataMigration): def forwards(self, orm): with southern_models(orm): call_command("loaddata", "my_fixture.json")
This creates a fixture in the form of a python script. Handles: 1. `ForeignKey` and `ManyToManyField`s (using python variables, not IDs) 2. Self-referencing `ForeignKey` (and M2M) fields 3. Sub-classed models 4. `ContentType` fields 5. Recursive references 6. `AutoField`s are excluded 7. Parent models are only included when no other child model links to it There are a few benefits to this: 1. edit script to create 1,000s of generated entries using `for` loops, python modules etc. 2. little drama with model evolution: foreign keys handled naturally without IDs, new and removed columns are ignored The [runscript command by poelzi](http://code.djangoproject.com/ticket/6243), complements this command very nicely! e.g. $ ./manage.py dumpscript appname > scripts/testdata.py $ ./manage.py reset appname $ ./manage.py runscript testdata
**This script converts a CSV file into a JSON file ready to be imported via `manage.py loaddata` like any other fixture data.** It can be used manually to do a one-time conversion (for placing into a /fixtures folder), or used in a fabric script that automatically converts CSV to JSON live then runs `loaddata` to import as fixture data. To run script: >`csv2json.py input_file_name model_name` > >e.g. csv2json.py airport.csv app_airport.Airport > >Note: input_file_name should be a path relative to where this script is. **Scripts depends on simplejson module.** The module can just be placed in a sub-folder to the script to make it easy to import. If you use the same Python binary that you use for your Django site, you could use the Django import instead: `from django.utils import simplejson` **File Input/Ouptut formats:** Assumes CSV files are saved with LF line endings, and that first line has field values. First column is the model's pk field. Sample CSV input: id,ident,name,city,state 1,00C,Animas Air Park,Durango,CO 6,00V,Meadow Lake,Colorado Springs,CO 7,00W,Lower Granite State,Colfax,WA 12,01J,Hilliard Airpark,Hilliard,FL Output file name is input name + ".json" extension. Sample JSON output: [ { "pk": 1, "model": "app_airport.Airport", "fields": { "name": "Animas Air Park", "city": "Durango", "ident": "00C", "state": "CO", } } ] **Debugging Conversion Problems** If JSON import errors out with "ValidationError: This value must be an integer", you probably have a blank in an Integer field within your CSV file, but if can't figure out, try setting a breakpoint in file: ./django/django/db/models/fields/__init__.py e.g. 688 try: 689 return int(value) 690 except (TypeError, ValueError): 691 import pdb; pdb.set_trace() 692 -> raise exceptions.ValidationError( 693 _("This value must be an integer.")) To figure out what field caused the error, while in the debugger: (Pdb) u <- to go UP the callstack (Pdb) field.name
This lets you load global fixtures from a directory you set as `FIXTURES_ROOT` in your settings.py. For example, setting `FIXTURES_ROOT` to `/path/to/myproject/fixtures/` I tend to like to keep fixtures that should be loaded when a new instance of a site is deployed, but not auto-loaded (so they won't rewrite any data that comes after), in a project-level fixtures directory like this. Sometimes these fixtures can be useful in your test suites, so this is a convenient way to load them. Usage: `load_global_fixtures('sites.json', 'contacts.json')` `load_global_fixtures('sites.json', 'contacts.json', fixtures_root='/some/other/path', verbosity=1)`
This is a simple fixture that is useful for many tests. It contains the following users: * admin * staff * user0 * user1 * user2 * user3 * inactive0 * inactive1 The password of every user is the same as his username, e.g.: admin/admin
This is a YAML version of http://djangosnippets.org/snipptes/2397
This snippet loads data from JSON files into a MongoDB database. The code is related with the other snippet [MongoDB data dump](http://djangosnippets.org/snippets/2872/). To get it working, just create a ``MONGODB_NAME`` variable in settings, holding the name of your Mongo database. This can be edited to fit more your needs. The snippet requires ``Pymongo``, since it uses its bson module and the ``MongoClient``.
This Django management command just dumps data from a given MongoDB collection into a JSON file. To get it working, just create a ``MONGODB_NAME`` variable in settings, holding the name of your Mongo database. This can be edited to fit more your needs. The snippet requires Pymongo, since it uses its ``bson`` module and the ``MongoClient``.
This is a version of [http://djangosnippets.org/snippets/2258/](http://djangosnippets.org/snippets/2258/) that should work with special chars (e.g. quotes) in json data.
Dumps DB data for each application to that application's fixtures directory. For example: $ ./manage.py dump_app_data ... $ hg status M apps/foo/fixtures/dev.json M apps/bar/fixtures/dev.json
I use this script to export a group of models that I want to import later as initial data. It exports them as serialized json, which is perfect for importing later with the loaddata function in manage.py.