Login

Export Django data to datestamped tarball -- choose individual models for handy packaging and archiving

Author:
fish2000
Posted:
September 23, 2010
Language:
Python
Version:
1.1
Tags:
json data django tarball backup python script tar archive datestamp django1.1 export command
Score:
2 (after 2 ratings)

Just like it says -- set it up and run. Use it for server migrations, for project handoffs, in cron jobs, you name it.

I have never had problems exporting models to individual fixtures in this way, and only one bout of trouble re-importing them (and that was, like, an impossible-triangle of OneToOneField dependencies anyway, and they were going from a sqlite file to a postgres schema that totally had inappropriate nullable columns in it). I find that the json files named for what they contain is helpful when and if manage.py does freak out during an import, as the output from loaddata command is so opaque it's autistic, basically.

A trivial refactoring effort could make it into a management command -- it already makes use of the builtin dumpdata command class internally. However I did not feel like overthinking it enough to set it up in a repository (doubtlessly padded with unrelated 'utilities' and explanatory .rst files) and then writing a blog post to sell it to you. That is why you are reading this code here, instead of on GitHub.

Don't get me wrong, GitHub is awesome, and like a consummate host... but not the way I love me some quick and dirty snippet code, these days. Whatever, you say lazy, I say productively relaxed, potato/potahto.

Erm. In any case please do enjoy this model fixture-exporter. Yes.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
#!/usr/bin/env python
# encoding: utf-8
"""
backupmodelstofixtures.py

TARBALL DATABASE FIXTURE DUMPER.

Created by FI$H 2000 on 2010-04-18.
Copyright (c) 2010 OST, LLC.
MIT License. It works, bitches.

Usage: fill in the blanks, run it, and collect your tarballed fixtures.

Tested by me on Debian and Mac OS -- it works in Textmate or from
the prompt. It will tell you what it is doing. On the Mac, it will
use /usr/bin/open -R to courteously reveal the finished tarball
in the Finder once it is done. 

An enterprising Django fan could easily kill five minutes making it
into a management command.
"""

from __future__ import with_statement

import sys
from django.core.management import setup_environ
import settings
setup_environ(settings)

import os, tarfile, shutil
from datetime import datetime
from tempfile import mkdtemp, gettempdir
from django.core.management.commands.dumpdata import Command as Dumper
from django.core.management.base import CommandError

# some typical off-the-shelf models:
from django.contrib.auth.models import User
from django.contrib.sessions.models import Session
from tagging.models import Tag, TaggedItem

# fill your models in here.
from your_project.some_app import AModel, AnotherModel
from your_project.some_other_app import AndStillYetAnotherModel

# format the models as a dict, 
# to match up with the management
# command 'app.model' syntax
appstogo = dict(
	auth=(User,),
	sessions=(Session,),
	admin=(ContentType,LogEntry),
	some_app=(AModel,AnotherModel),
	some_other_app=(AndStillYetAnotherModel,)
	# etc.
)

# it comes out like: 20100923-052013.tgz --
# they will chronologically sort themselves in a file listing.
timestamp = datetime.strftime(
	datetime.now(),
	"%Y%m%d-%H%M%S",
)

# change this to what you prefer and/or what your OS demands
#finaldestination = "/home/you"
finaldestination = "/Users/you/Desktop"

fixdir = mkdtemp(prefix='', suffix=timestamp)
fixtarfilename = "%s.tar.gz" % timestamp
fixtarname = os.path.join(gettempdir(), fixtarfilename)
fixtar = tarfile.open(fixtarname, "w:gz")
os.chdir(fixdir)

print "> Dumping model fixtures to temporary directory %s" % fixdir

print ""
for app, modls in appstogo.items():
	for modl in modls:
		try:
			fix = Dumper.handle(Dumper(), "%s.%s" % (app, modl.__name__.lower()), format='json', indent=4)
		except CommandError, message:
			print "# dumpdata raised a CommandError: %s" % message
		else:
			fixname = "%s_%s.json" % (app, modl.__name__.lower())
			print "+ Dumping all instances of %s to %s..." % (modl.__name__, fixname)
			with open(os.path.join(fixdir, fixname), "w+") as fixh:
				fixh.write(fix)

print ""
print "> Creating fixture archive in %s..." % fixtarname
for fixxx in os.listdir(fixdir):
	fixtar.add(fixxx, recursive=False)
fixtar.close()

print ""
print "> Cleaning up..."
for dirty in os.listdir(fixdir):
	os.remove(dirty)
os.rmdir(fixdir)


print "> Moving fixture archive to %s..." % finaldestination
shutil.move(fixtarname, finaldestination)

if os.path.exists('/usr/bin/open'):
	print "> Revealing..."
	os.system('/usr/bin/open -R %s' % os.path.join(finaldestination, fixtarfilename))


print "> Done."


sys.exit(0)

More like this

  1. CSV to JSON Fixture by briangershon 5 years, 10 months ago
  2. Export models as json by aconbere 8 years, 3 months ago
  3. Proper fixtures loading in south data migrations by JustDelight 2 years, 4 months ago
  4. MongoDB data dump by jose_lpa 2 years, 6 months ago
  5. JSON fixtures of Intl. country codes & dial-codes by dibau_naum_h 4 years, 10 months ago

Comments

Please login first before commenting.