Login

3110 snippets

Snippet List

models.MultipleEmailField

Define validator `multiple_email_validator` that splits value by commas and calls `validate_email` validator for each element found. Then define MultipleEmailField with this default validator and augmented max_length. Then ... use it!

  • models
  • fields
  • validators
Read More

Effective content caching for mass-load site using redirect feature

Hi All, I would like to share my idea and experience in using http redirect feature to organize effective content caching and avoid site blocking on mass requests to not yet cached content. First of all, I should explain that I don't mean usual template-based pages, whose caching in django is enough in almost all cases. My speech is concerning to content such as dynamically created images, external files and other content which may require almost unpredictable long time period to be ready, up to about 10 seconds. Anyway, the content which is already in cache, can be returned immediately. The question is - what *should* happen if the requested content is *not yet present* in the cache? The HTML content is not a problem in this case - we can return intermediate page with meta information to ask a client to retry the content after some period of time. But non-HTML content, such as image, or json, is a problem. The usual *blocking* behavior of the django caching subsystem is not appropriate in a case when the content preparation process takes *more than several fractions* of a second. The whole period while the content is preparing (painting, getting from the external resource and so on) *hundreds* of users will *ask not yet cached* content until all available application server connections will be *occupied* by waiting requests and new user will wait for the content even if the content requested by him is actually ready to be returned. The celery package resolves this problem *partially*, allowing to schedule cache update before cache expiration in asynchronous manner (I am using threadpool package for this purpose instead), and avoiding blocking on content preparation procedure, but the problem with not yet available content remains unresolved. We might schedule periodic cache update to avoid such a situation, but this way leads to unnecessary resource load and may be inappropriate at all in some cases, f.e. for the service like CloudMade or Google "staticmap" painting map content basing on the request parameters. My solution is *using HTTP Redirect feature* . Ideal solution would be available if most of HTTP clients supported Retry-After header in the HTTP Redirect response. Unfortunately, no one of most popular clients (neither Mozilla, nor IE, nor Chrome) does it. Really they ask redirected URL immediately after receiving the redirection response, and ignore Retry-After header at all. Anyway, I hope that they *will* support this feature, but this is not enough right now. So, instead of delegating waiting job to the client side, we need to organize waiting on the server. For this purpose I have created small separate application server with it's *own connection pool*, and delegated waiting task to him (see the snippet). This server needs one URL (/retry_after in the example) looking to the view above, installed in the url.py, and additional DEFAULT_RETRY_TIMEOUT parameter in the settings.py. The view uses request GET parameters, so Django will not try to cache it's results. The only task for this server is to *suspend* redirected requests until requested time is *really arrived*, and then redirect the request to the original URL. To make possible to process many requests simultaneously, it breaks waiting and redirects too long time requests to the self basing on the default timeout. Using this server is simple. We will instantiate the 'retry' server on the separate URL as a *separate application server* and will redirect all requests required to be delayed to this server, passing original URL and the time moment when the requested content should be ready, as parameters. The main server which prepares and returns the content, is *never waiting*. It returns prepared content from the cache, if it is found there, or *starts* content preparation process in *asynchronous* manner using threads, celery package, or any other appropriate technique, and *immediately redirects* the request to the 'retry' server The following is a fragment of the code using 'retry' server: ... # Create your views here. def jams(request,z,x,y): jams_timeout = settings.JAMS_TIMEOUT retry_after_server = settings.RETRY_AFTER_SERVER try: jams_ts,jams_png = request_jams(z,x,y) except Http404,ex: return HttpResponseNotFound() if not jams_png: if settings.DEBUG: logger.debug("JAMS NOT YET READY FOR: %s" % str((z,x,y))) # Tell a client to retrieve jams later time_after = time.time() + settings.DEFAULT_RETRY_TIMEOUT if settings.DEBUG: logger.debug("SEND RETRY AFTER FOR: %s" % str((z,x,y))) loc = reverse(jams,args=(z,x,y)) if request.get_host(): loc = request.build_absolute_uri(loc) r = HttpResponseRedirect( retry_after_server + '?' + urlencode({'redirect': loc,'retry_after':int(time_after)}) ) r['Retry-After'] = settings.DEFAULT_RETRY_TIMEOUT return r # return a png got from the cache if settings.DEBUG: logger.debug("RETURNING JAMS FOR: %s" % str((z,x,y,http_date(jams_ts),http_date(time.time())))) r = HttpResponse(jams_png) r['Content-Type'] = "image/png" r['Content-Length'] = len(jams_png) r['Content-Location'] = request.path # extra data for cache negotiation # Common and ConditionalGet Django Middleware will do the rest r['ETag'] = '"%s"' % md5_constructor(jams_png).hexdigest() r['Last-Modified'] = http_date(jams_ts) r['Expires'] = http_date(jams_ts + jams_timeout) return r ... At the application level, the request_jams() call tries to get content from the cache, and starts content update process asynchronously if required. It returns content got from the cache, and timestamp when the content has been created last time, or pair of None,None values, if the requested content is not available immediately. The line starting from 'r = HttpResponseRedirect' redirects the request to the 'retry' server, if the content is not available immediately. I am using nginx as a 'front-end' proxy server which passes requests to two separate fastcgi daemons - 'retry' and main django application servers separating requests by prefix. The following is a fragment of the nginx.conf configuration file for the nginx server: server { ... location /jams { fastcgi_pass unix:/var/tmp/jams.sock; ... } location /retry_after { fastcgi_pass unix:/var/tmp/retry.sock; ... } ... } As you can see, two locations are passed to *different* application server instances in the production environment, so they *don't block each other*. Really they both are implemented inside one django project, and started from the same directory as fastcgi daemon instances with different starting parameters (sockets to listen, log files etc.). I've used such structure to configure and debug my server easy, either just starting one developer server for both locations, or to have a possibility to start production server in described 'split' mode on the other hand. You can see that the both servers (jams and retry) use the same DEFAULT_RETRY_TIMEOUT setting from the settings.py file. The DEFAULT_RETRY_TIMEOUT plays role of load balancing parameter for requests which wait finishing asynchronous job. After the retry timeout is passed, the control returns to the client by the redirection response (or, for smart clients, the client itself waits the retry timeout before requesting the resource again), so the server may cleanup resources temporary acquired by these requests. Additional bonus of using such technique is stability of server processor load. Even in prime time, the server load is stable and controlled: number of the application instances and number of threads in the thread pool (evaluating asynchronous jobs) for each instance is fixed by configuration, nginx load per request is minimal, and concurrent requests are balanced (between server, network, and clients) by the retry timeout and redirect responses. You can see the result of the work on the our site, f.e. [on this page](http://www.doroga.tv/nnov/apps/map/), where all (transparent version) jam tiles are painting such a manner. The only difference for the production server environment is that some top zooms of most popular regions are refreshed by the special daemon (also written using django as a 'management command') which forces tiles to be refreshed in the cache even no one client requests it right now — to decrease response time for the jams map after long 'sleep' period.

  • cache
  • load
  • redirect
  • nginx
  • content
  • mass
Read More

Newsletter email template

You can use this class for render and send html email message with the same logic and facility of website page creation. Just create an html template file with the same name of Class in lowercase.

  • template
  • email
  • newsletter
Read More

send_html_mail

There are many versions, this is the one I like. This is quite generic, can auto generate text version of the mail if required.

  • html-mail
Read More

Detailed Error Reporting by Email

The default traceback sent by email when an error occurs, usually gives too little information comparing it to the error page in the DEBUG mode. This snippet guerilla-patches error handling and sends by email the same information as you would see in DEBUG mode. To set it up, add the snippet to any `models.py` of an installed app. (I wonder why this hasn't been implemented in the core)

  • email
  • error
  • reporting
Read More

manage.py reboot

1) Simply create a "management/commands" folder in one of your INSTALLED_APPS folders. Then add a "reboot.py" file in your "management/commands" 2) In settings.py, you can OPTIONALLY add: PROJECT_NAME = 'Blah' PROJECT_DOMAIN = 'blah.com' These two settings will be used to create a "Site" object as well as a superuser. If you choose not to use these settings, the reboot command simply reverts to using your DATABASES[default] username as superuser. 3) Execute the command via "python manage.py reboot" Enjoy.

  • runserver
  • setup superuser
  • recreate database
  • install fixtures
Read More

Checkbox or radio iterator as template filter

This snippet is based on [Bill Freeman's MultiSelect checkbox iterator template filter](http://djangosnippets.org/snippets/2151/). Usage: See docstring

  • checkbox
  • choice
  • templatefilter
  • iterator
  • checkbox-input
  • form-choice
  • radio-button
  • radio-input
Read More

Dynamic tabular inlines with optional drag-n-drop sorting

This jQuery javascript enables dynamic add/delete of rows in tabular inlines. It adds a "+" icon at the bottom of the inline to allow addition of new rows, and replaces the default delete checkbox with a "x" icon for deletion, giving you the possibility to add/delete rows instantly without reloading the page. In addition, it gives you drag-n-drop ordering functionality with a named position model field using jQuery UI Sortable. **Usage (see below for example):** Just include the javascript on your admin page, together with jQuery, and it'll automatically affect all tabular inlines. Optionally, also include jQuery UI Sortable and an Integer field in your inline model named "position" (or whatever you set "position_field" to), which will automatically hide the position field and enable drag-n-drop sorting. **Developed for:** * jQuery 1.3.2 * jQuery UI 1.7.1 * Django trunk (tested in Django v1.0.2) * (Might work with other versions with or without adjustments, but not tested) **Settings (in top of javascript):** * "position_field" is the name of an integer model field that is used for ordering the inline model. If left empty or not found, the drag-n-drop functionality is dropped. Defaults to "position". * "add_link_html" for custom look of "add"-buttons. Defaults to Django's built-in "+" image icon. * "delete_link_html" for custom look of "delete"-buttons. Defaults to Django's built-in "x" image icon. **Use example: ** *admin.py:* class NameInline(admin.TabularInline): model = Name extra = 1 class PersonAdmin(admin.ModelAdmin): inlines = [NameInline] class Media: js = ['js/jquery-1.3.2.min.js', 'js/ui/ui.core.js', 'js/ui/ui.sortable.js', 'js/dynamic_inlines_with_sort.js',] css = { 'all' : ['css/dynamic_inlines_with_sort.css'], } admin.site.register(Person, PersonAdmin) *models.py:* class Person(models.Model): year_born = models.PositiveIntegerField(_('year born'), null=True, blank=True) class Name(models.Model): profile = models.ForeignKey(Profile, verbose_name=_('profile')) position = models.PositiveIntegerField(_('position'), default=0) name = models.CharField(_('name'), max_length=100) class Meta: ordering = ('position',) *dynamic_inlines_with_sort.css:* /* To make row height of saved items same as others */ .inline-group .tabular tr.has_original td { padding-top:0.5em; } .inline-group .tabular tr.has_original td.original p { display:none; } Please post bugs in comments.

  • javascript
  • dynamic
  • admin
  • sort
  • jquery
  • ordering
  • inlines
  • inline
  • tabular
  • sortable
Read More

Django Incremental Counter Tag

Counter tag. Can be used to output and increment a counter. For usage, see docstring in the code. This is the first complete tag that I've implemented, I hope that there are no bugs and that it's thread safe.

  • incremental
  • counter
  • increment
  • numbering
Read More

head inclusion middleware

Inspired by [http://www.djangosnippets.org/snippets/712/](YUI Loader as Django middleware) This loads in css and javascript files where you want them (usually in the head) - but allows you to put them anywhere in your code - i.e. in a TemplateTag. so your head code will look like this: ` <head> ... <!-- HEAD_init --> ... </head> ` then somewhere in your templates you can load in javascript or css files like so: ` <!-- HEAD_include myfile.js myotherfile.css --> ` It automatically checks if you've already included the files, and only puts them in once. It automatically figures out if its a javascript or css file by the file name - If you have an irregular filename (i.e. a google maps api script url) you can force it by using either of the following tags: ` <!-- HEAD_include_js [my javascript file] --> <!-- HEAD_include_css [my css file] --> ` Or you can write inline code to get rendered in the head: ` <!-- HEAD_render <script> someJavascriptCall(); </script> --> ` Todo: make it compress the js into one file...

  • middleware
  • javascript
  • css
Read More

Enhanced "avoid widows" template filters

Building on [jcroft's snippet](http://www.djangosnippets.org/snippets/17/), here's a slightly more advanced version which has two filters, one for basic text and the other for html snippets. Usage is like so: <h2>{{ blog_entry.headline|escape|widont }}</h2> {{ blog_entry.html|widont_html }} On top of Jeff's reasons for using these filters, they are important because they help keep one of [God's commandments](http://www.ebible.com/bible/NIV/Exodus+22%3A22). ;)

  • filter
  • widows
  • typography
  • widont
Read More