Login

Tag "queryset"

39 snippets

Snippet List

Memory efficient Django Queryset Iterator

While checking up on some cronjobs at [YouTellMe](http://www.youtellme.nl/) we had some problems with large cronjobs that took way too much memory. Since Django normally loads all objects into it's memory when iterating over a queryset (even with .iterator, although in that case it's not Django holding it in it's memory, but your database client) I needed a solution that chunks the querysets so they're only keeping a small subset in memory. Example on how to use it: `my_queryset = queryset_iterator(MyItem.objects.all()) for item in my_queryset: item.do_something()` [More info on my blog](http://www.mellowmorning.com/2010/03/03/django-query-set-iterator-for-really-large-querysets/)

  • queryset
  • iterator
  • memory
  • gc
Read More

Custom managers with chainable filters

The Django docs show us how to give models a custom manager. Unfortunately, filter methods defined this way cannot be chained to each other or to standard queryset filters. Try it: class NewsManager(models.Manager): def live(self): return self.filter(state='published') def interesting(self): return self.filter(interesting=True) >>> NewsManager().live().interesting() AttributeError: '_QuerySet' object has no attribute 'interesting' So, instead of adding our new filters to the custom manager, we add them to a custom queryset. But we still want to be able to access them as methods of the manager. We could add stub methods on the manager for each new filter, calling the corresponding method on the queryset - but that would be a blatant DRY violation. A custom `__getattr__` method on the manager takes care of that problem. And now we can do: >>> NewsManager().live().interesting() [<NewsItem: ...>]

  • manager
  • queryset
Read More

Batch querysets

Most of the time when I need to iterate over Whatever.objects.all() in a shell script, my machine promptly reminds me that sometimes even 4GB isn't enough memory to prevent swapping like a mad man, and bringing my laptop to a crawl. I've written 10 bazillion versions of this code. Never again. **Caveats** Note that you'll want to order the queryset, as ordering is not guaranteed by the database and you might end up iterating over some items twice, and some not at all. Also, if your database is being written to in between the time you start and finish your script, you might miss some items or process them twice.

  • queryset
  • batch
Read More

ParentModel and ChildManager for Model Inheritance

This is the approach I've taken to access instances of child models from their parent. Functionally it's very similar to snippets [1031](http://www.djangosnippets.org/snippets/1031/) and [1034](http://www.djangosnippets.org/snippets/1034/), but without the use of `django.contrib.contenttypes`. Usage: class Post(ParentModel): title = models.CharField(max_length=50) objects = models.Manager() children = ChildManager() def __unicode__(self): return self.title def get_parent_model(self): return Post class Article(Post): text = models.TextField() class Photo(Post): image = models.ImageField(upload_to='photos/') class Link(Post): url = models.URLField() In this case, the `Post.children` manager will return a queryset containing instances of the appropriate child model, rather than instances of `Post`. >>> Post.objects.all() [<Post: Django>, <Post: Make a Tumblelog>, <Post: Self Portrait>] >>> Post.children.all() [<Link: Django>, <Article: Make a Tumblelog>, <Photo: Self Portrait>]

  • model
  • manager
  • queryset
  • inheritance
Read More

SearchableManager

A drop-in chainable manager for providing models with basic search features such as +/- modifiers, quoted exact phrases and ordering by relevance.

  • search
  • model
  • manager
  • queryset
Read More

Case-insensitive lookup by default

I wanted lookups on tags to be case insensitive by default, so that things like Tag.objects.get(name='Tag') would return any similar tags (ignoring case differences), i.e. `<Tag: tag>`. This snippet makes lookup on the 'name' field case-insensitive by default, although case-sensitive lookups can still be achieved with 'name__exact'. Methods like get_or_create will work as expected and be case-insensitive.

  • model
  • tags
  • tagging
  • manager
  • case-insensitive
  • iexact
  • queryset
Read More

Convert Q object to function

This is a function to take a Q object and construct a function which returns a boolean. This lets you use the exact same filter syntax that Django's managers use and apply it inside list comprehensions, or to non-persistent objects, or to objects of different types with the same attribute names.

  • q-objects
  • field
  • queryset
Read More

Easier chainability with custom QuerySets

Django allows you to specify your own ModelManager with custom methods. However, these methods are chainable. That is, if you have a method on your PersonManager caled men(), you can't do this: Person.objects.filter(birth_date__year=1978).men() Normally, this isn't a problem, however your app may be written to take advantage of the chainability of querysets. For example, you may have an API method which may return a filtered queryset. You would want to call with_counts() on an already filtered queryset. In order to overcome this, we want to override django's QuerySet class, and then make the Manager use this custom class. The only downside is that your functions will not be implemented on the manager itself, so you'd have to call `Person.objects.all().men()` instead of `Person.objects.men()`. To get around this you must also implement the methods on the Manager, which in turn call the custom QuerySet method.

  • model
  • manager
  • queryset
Read More

Generic model filter from request GET data

You just use standart django query terms, for example: <form> <input class="field_text filter_from" type="text" name="cost__gte" placeholder="from" value="{% if request.GET.cost__gte %}{{ request.GET.cost__gte }}{% endif %}"> <input class="field_text filter_to" type="text" name="cost__lte" placeholder="to" value="{% if request.GET.cost__lte %}{{ request.GET.cost__lte }}{% endif %}"> </form> model: class Object(models.Model): cost = models.IntergerField() objects = ObjectManager()

  • filter
  • forms
  • queryset
Read More

Multiple querysets

This is an upgrade of snippet [1103](http://www.djangosnippets.org/snippets/1103/). Exemplary usage: class Blog(models.Model): name = models.CharField(max_length=100) def __unicode__(self): return self.name class Post(models.Model): title = models.CharField(max_length=50) blog = models.ForeignKey(Blog) def __unicode__(self): return self.title class Meta: abstract=True class Article(Post): text = models.TextField() class Link(Post): url = models.URLField() blog = Blog(name="Exemplary blog") blog.save() Article(title="#1", text="Exemplary article 1", blog=blog).save() Article(title="#2", text="Exemplary article 2", blog=blog).save() Link(title="#3", url="http://exemplary.link.com/", blog=blog).save() qs1 = Article.objects.all() qs2 = Link.objects.all() qsseq = QuerySetSequence(qs1, qs2) # those all work also on IableSequence len(qsseq) len(QuerySetSequence(qs2, qs2)) qsseq[len(qs1)].title # this is QuerySetSequence specific qsseq.order_by('blog.name','-title') excluded_homo = qsseq.exclude(title__contains="3") # homogenic results - returns QuerySet type(excluded_homo) excluded_hetero = qsseq.exclude(title="#2") # heterogenic results - returns QuerySetSequence type(excluded_hetero) excluded_hetero.exists() You can implement more `QuerySet` API methods if needed. If full API is implemented it makes sense to also subclass the `QuerySet` class.

  • queryset
  • chain
  • iterable
  • indexable
Read More

User manager

If you have a model with foreign key to User, you can use this manager to show (i.e. in admin interface) only objects, that are related to currently logged-in user. Superuser sees all objects, not only his. Requires: [ThreadlocalsMiddleware](http://code.djangoproject.com/wiki/CookBookThreadlocalsAndUser)

  • filter
  • foreignkey
  • user
  • manager
  • queryset
  • owner
  • users
  • user-foreign-key
Read More

Queryset Foreach

Call a function for each element in a queryset (actually, any list). Features: * stable memory usage (thanks to Django paginators) * progress indicators * wraps batches in transactions * can take managers or even models (e.g., `Assertion.objects`) * warns about `DEBUG`. * handles failures of single items without dying in general. * stable even if items are added or removed during processing (gets a list of ids at the start) Returns a `Status` object, with the following interesting attributes * `total`: number of items in the queryset * `num_successful`: count of successful items * `failed_ids`: list of ids of items that failed

  • queryset
  • progress
  • callback
  • map
  • foreach
  • iterator
Read More