Log the time taken to execute each DB query
Use this snippet to log the time taken to execute a specific DB query.
- django
- db
- logging
Snippet List
ByteSplitterField
When you want to save integers to the db, you usually have the choice between 16-, 32- and 64-bit Integers (also 8- and 24-bit for MySQL). If that doesn't fit your needs and you want to use your db-memory more efficient, this field might be handy to you. Imagine you have 3 numbers, but need only 10 bit to encode each (i.e. from 0 to 1000). Instead of creating 3 smallint-fields (48 bit), you can create one 'ByteSplitterField' which implements 3 'subfields' and automatically encodes them inside a 32 bit integer. You don't have to take care how each 10-bit chunk is encoded into the 32-bit integer, it's all handled by the field (see also field's description). Additionally, the Field offers opportunity to use decimal_places for each of your subfields. These are 'binary decimal places', meaning the integer-content is automatically divided by 2, 4, 8, etc. when you fetch the value from the field. You can also specify how values are rounded ('round' parameter) and what happens when you try to save a value out of range ('overflow' parameter) Not implemented (maybe in the future if I should need it sometime): * signed values. All values are positive right now! * real (10-based) decimal places (actually you could probably directly use DecimalFields here) * further space optimization, i.e. saving into CharField that's length can be chosen byte-wise
- model
- db
- database
- field
- custom
- custom-model-field
- IntegerField
- multibit-field
- model-field
TinyIntegerField
Django lacks support of MySQL's "tinyint" 8-bit-integer datatype. This snippet gives you a TinyIntegerField and a PositiveTinyIntegerField. Falls back Django's SmallIntegerField if a different database-engine is used
- model
- db
- 8-bit-integer
- MySQL
- IntegerField
- tiny-integer
Finding related objects for instances in a queryset
When deleting objects in Django's admin interface, it lists other objects which would be deleted and asks for confirmation. This snippet does the same programmatically. The snippet works in Django 1.3 (more specifically, revision 14507 or later). It uses Django internals which are not a part of the public API, so this might not work with future versions. Usage: `polls/models.py`: from django.db import models class Poll(models.Model): question = models.CharField(max_length=200) def __unicode__(self): return self.question class Choice(models.Model): poll = models.ForeignKey(Poll) choice = models.CharField(max_length=200) def __unicode__(self): return '%s %s' % (self.poll, self.choice) `$ ./manage.py shell` >>> from polls.models import Poll, Choice >>> from datetime import datetime >>> from pprint import pprint >>> poll1 = Poll.objects.create(question='Me?') >>> Choice.objects.create(poll=poll1, choice='Yes') >>> Choice.objects.create(poll=poll1, choice='No') >>> poll2 = Poll.objects.create(question='Really?') >>> Choice.objects.create(poll=poll2, choice='Yes') >>> Choice.objects.create(poll=poll2, choice='No') >>> pprint(get_related(Poll.objects.all())) {<class 'polls.models.Poll'>: [<Poll: Me?>, <Poll: Really?>], <class 'polls.models.Choice'>: [<Choice: Me? Yes>, <Choice: Me? No>, <Choice: Really? Yes>, <Choice: Really? No>]}
- db
- orm
- related
Improved Pickled Object Field (Fixed for Django 1.2)
Small changes to [Snippet 1694](http://djangosnippets.org/snippets/1694/) to that QueryAPI works for django 1.2 and higher. Changes: * Replaced `get_db_prep_value` with `get_prep_value`. * Replaced `get_db_prep_lookup` with modified `get_prep_lookup`.
- model
- db
- orm
- database
- pickle
- object
- field
- type
- pickled
- store
Log all interaction with user to the DB
Due to compliance requirements in the financials industry we needed to log every request a user made to our system, the action taken (view function) and response from the server. I found a lot of other logging solution bit most revolved around debugging and DB query logging. I needed to be able to tell what a user did while being logged in as much detail as I could with out tracking the mouse pointer position on screen. So I created this (, *my first* ,) middleware. Its very simple really. keeping track of a request, view_func and response object in a single model called Record (models.py file included in the code). The fields I used are optimized to what I intend to show in the UI I am planning for this model. Depending on how you use the doc string of your views they can be tapped to explain to the user what each request/func/responce group in a session is meant to do. There were a few gotcha's: 1. I only care about authenticated requests. So I added the 'quest.user.is_authenticated()' test. 2. I did not care about the favicon request so I skipped them. 2. The actual login request is not authenticated while the response is. This caused the process_response/view to look for a record that is not there. So I added the 'except ObjectDoesNotExist' to skip this case. I added one bell: Logging a full HTML reply is wasteful and mostly useless. I added two values in the setting files. LOGALL_LOG_HTML_RESPONSE to toggle if we want to log them or not. And LOGALL_HTML_START to describe what a full HTML starts with. Personally I use the first few characters of my base.html template that all the rest of my templates expend. I simplified the code to the left for readability.
- middleware
- log
- db
- compliance
- financial
orm_tools
This module contains classes that add new behavior to Django's ORM. Classes include: **Session** * Forces QuerySet objects to return identical instances when objects with the same primary key are queried. * Similar to [SQLAlchemy Session](http://www.sqlalchemy.org/docs/orm/session.html) **GraphSaver** * Save entire object graphs at once. * Automatically detects object dependencies and saves them in the correct order. **Collection** * Easier one-to-many relationships. Instructions and more information on [limscoder.com](http://www.limscoder.com/2011/01/django-orm-tools.html).
- session
- db
- orm
model instance to sql insert statement
This function will take a model instance and return an insert statement for it. I use it for extracting a subset of my production data so that I can reproduce problems in a local environment. The quoting is for mysql, you may have to change it depending on your db backend. Also, it assumes the 'default' database.
- sql
- model
- db
django_bulk_save.py - defer saving of django models to bulk SQL commits
When called, this module dynamically alters the behaviour of model.save() on a list of models so that the SQL is returned and aggregated for a bulk commit later on. This is much faster than performing bulk writing operations using the standard model.save(). To use, simply save the code as django_bulk_save.py and replace this idiom: for m in model_list: # modify m ... m.save() # ouch with this one: from django_bulk_save import DeferredBucket deferred = DeferredBucket() for m in model_list: # modify m ... deferred.append(m) deferred.bulk_save() Notes: * - After performing a bulk_save(), the id's of the models do not get automatically updated, so code that depends on models having a pk (e.g. m2m assignments) will need to reload the objects via a queryset. * - post-save signal is not sent. see above. * - This code has not been thoroughly tested, and is not guaranteed (or recommended) for production use. * - It may stop working in newer django versions, or whenever django's model.save() related code gets updated in the future.
- sql
- tool
- dump
- save
- db
- bulk
- util
Extended db cache backend with 'filter() / LIKE' support (and now scheduled cache clean!)
Because the db caching doesn't support atomic operations, it was unsafe to store a list of 'keys' in a single key. So, rather than store the list, I just append each key with a specific tag, and then filter for it later. This means I don't need to worry too much about atomic usage with lists (i.e. queued requests). However - I still can think of many instances where I would need atomic usage, so I will probably implement this later on down the line. Hopefully, the atomic modifications will be accepted by the core devs. This also contains threaded cache cleaning, which means you no longer need to rely on requests to clean the cache (which would have potentially slowed the user query down), and will remove any cache entries past their expiry date every 3 minutes. Enjoy! Cal
- filter
- cache
- db
- backend
- like
grep and delete sqlite tables
This is a slight improvment of a previous snippet of mine: http://www.djangosnippets.org/snippets/1776/ It is an interactive shell script that greps and deletes sqlite tables USAGE: ./pdrop.sh myquery mydbfile
- db
- sqlite
- drop
Integer based MoneyField
It is supposed the aggregation on integer is fast than numeric type in database duo to how they are stored as numeric is represented as string. As money only have 2 decimal place, it can be converted to an Integer despite of its decimal point. The python class decimal.Decimal also has a shortcoming that it is not json serializable(neither cjson nor simplejson). This money field appears as a float number to the front end, so it does not meet the headache as DecimalField. The usage is very simple. In Model: class SomeModel(models.Model): ... income = MoneyField('income') ... Then treat the attribute of the model instance as a normal float type. **Notes** If you try to use aggregation on this field, you have to convert it to float by yourself. Currently I could not find any method to fix this.
- money
- db
- field
Improved Pickled Object Field
[Based on snippet #513 by obeattie.](http://www.djangosnippets.org/snippets/513/) **Update 10/10/09:** [Further development is now occurring on GitHub, thanks to Shrubbery Software.](http://github.com/shrubberysoft/django-picklefield) Incredibly useful for storing just about anything in the database (provided it is Pickle-able, of course) when there isn't a 'proper' field for the job. `PickledObjectField` is database-agnostic, and should work with any database backend you can throw at it. You can pass in any Python object and it will automagically be converted behind the scenes. You never have to manually pickle or unpickle anything. Also works fine when querying; supports `exact`, `in`, and `isnull` lookups. It should be noted, however, that calling `QuerySet.values()` will only return the encoded data, not the original Python object. *Please note that this is supposed to be two files, one fields.py and one tests.py (if you don't care about the unit tests, just use fields.py).* This PickledObjectField has a few improvements over the one in [snippet #513](http://www.djangosnippets.org/snippets/513/). 1. This one solves the `DjangoUnicodeDecodeError` problem when saving an object containing non-ASCII data by base64 encoding the pickled output stream. This ensures that all stored data is ASCII, eliminating the problem. 2. `PickledObjectField` will now optionally use `zlib` to compress (and uncompress) pickled objects on the fly. This can be set per-field using the keyword argument "compress=True". For most items this is probably **not** worth the small performance penalty, but for Models with larger objects, it can be a real space saver. 3. You can also now specify the pickle protocol per-field, using the protocol keyword argument. The default of `2` should always work, unless you are trying to access the data from outside of the Django ORM. 4. Worked around a rare issue when using the `cPickle` and performing lookups of complex data types. In short, `cPickle` would sometimes output different streams for the same object depending on how it was referenced. This of course could cause lookups for complex objects to fail, even when a matching object exists. See the docstrings and tests for more information. 5. You can now use the `isnull` lookup and have it function as expected. A consequence of this is that by default, `PickledObjectField` has `null=True` set (you can of course pass `null=False` if you want to change that). If `null=False` is set (the default for fields), then you wouldn't be able to store a Python `None` value, since `None` values aren't pickled or encoded (this in turn is what makes the `isnull` lookup possible). 6. You can now pass in an object as the default argument for the field without it being converted to a unicode string first. If you pass in a callable though, the field will still call it. It will *not* try to pickle and encode it. 7. You can manually import `dbsafe_encode` and `dbsafe_decode` from fields.py if you want to encode and decode objects yourself. This is mostly useful for decoding values returned from calling `QuerySet.values()`, which are still encoded strings. The tests have been updated to match the added features, but if you find any bugs, please post them in the comments. My goal is to make this an error-proof implementation. **Note:** If you are trying to store other django models in the `PickledObjectField`, please see the comments for a discussion on the problems associated with doing that. The easy solution is to put django models into a list or tuple before assigning them to the `PickledObjectField`. **Update 9/2/09:** Fixed the `value_to_string` method so that serialization should now work as expected. Also added `deepcopy` back into `dbsafe_encode`, fixing #4 above, since `deepcopy` had somehow managed to remove itself. This means that lookups should once again work as expected in **all** situations. Also made the field `editable=False` by default (which I swear I already did once before!) since it is never a good idea to have a `PickledObjectField` be user editable.
- model
- db
- orm
- database
- pickle
- object
- field
- type
- pickled
- store
JSONField
This is a custom field that lets you easily store JSON data in one of your model fields. This is updated to work with Django 1.1. **Example: (models.py)** from django.db import models import JSONField class MyModel(models.Model): info = JSONField() ** Example: (shell)** >>> obj = MyModel.objects.all()[0] >>> type(obj.info) <type 'NoneType'> >>> obj.info = {"test": [1, 2, 3]} >>> obj.save() **[Code at GitHub](http://github.com/bradjasper/django-jsonfield/tree/master)**
- models
- fields
- model
- json
- db
- field
- json-field
- jsonfield
iter_fetchmany
When executing custom sql, the temptation is to use fetchall or fetchone, since the API for fetchmany is a bit awkward. (fetchall makes all records resident in client memory at once; fetchone takes a network round-trip to the DB for each record.) This snippet, hoisted from django.db.models.sql.query.results_iter, presents a nice, simple iterator over multiple fetchmany calls which hits a sweet spot of minimizing memory and network usage.
- sql
- db
- db-api
- fetchmany
34 snippets posted so far.