Upload progress handler using cache framework

Author:: ebartels
Posted:: April 2, 2008
Language:: Python
Version:: .96
Score:: 8 (after 9 ratings)

Download
Raw

Ticket #2070 allows you to create your own file upload handlers. Here's an example handler that tracks a file's upload so you might display a progress meter after form submission.

The snippet has two parts, the upload handler which tracks progress, and an upload_progress view used to report back to the browser.

The upload handler uses django's cache framework to store the data, which can then be retrieved by the view to send back to the browser.

Note: Your form's http post request must have a query parameter (X-Progress-ID) sent along with it, which should contain a unique key to identify the upload. That key will also be sent with your ajax requests to the upload_progress view to retrieve the progress data.

Setup: place the UploadProgressCachedHandler anywhere you like on the python path, and add to your settings.py:

from django.conf import global_settings 
FILE_UPLOAD_HANDLERS = ('path.to.UploadProgressCachedHandler', ) + \
    global_settings.FILE_UPLOAD_HANDLERS

Set up the upload_progress view in any of your apps along with a corresponding entry in your urlconf.

Here's some javascript example code to make the ajax requests and display the progress meter: http://www.djangosnippets.org/snippets/679/

from django.core.files.fileuploadhandler import FileUploadHandler
from django.core.cache import cache

class UploadProgressCachedHandler(FileUploadHandler):
    """
    Tracks progress for file uploads.
    The http post request must contain a header or query parameter, 'X-Progress-ID'
    which should contain a unique string to identify the upload to be tracked.
    """

    def __init__(self, request=None):
        super(UploadProgressCachedHandler, self).__init__(request)
        self.progress_id = None
        self.cache_key = None

    def handle_raw_input(self, input_data, META, content_length, boundary, encoding=None):
        self.content_length = content_length
        if 'X-Progress-ID' in self.request.GET :
            self.progress_id = self.request.GET['X-Progress-ID']
        elif 'X-Progress-ID' in self.request.META:
            self.progress_id = self.request.META['X-Progress-ID']
        if self.progress_id:
            self.cache_key = "%s_%s" % (self.request.META['REMOTE_ADDR'], self.progress_id )
            cache.set(self.cache_key, {
                'length': self.content_length,
                'uploaded' : 0
            })

    def new_file(self, field_name, file_name, content_type, content_length, charset=None):
        pass

    def receive_data_chunk(self, raw_data, start):
        if self.cache_key:
            data = cache.get(self.cache_key)
            data['uploaded'] += self.chunk_size
            cache.set(self.cache_key, data)
        return raw_data
    
    def file_complete(self, file_size):
        pass

    def upload_complete(self):
        if self.cache_key:
            cache.delete(self.cache_key)



# A view to report back on upload progress:

from django.core.cache import cache
from django.http import HttpResponse, HttpResponseServerError 

def upload_progress(request):
    """
    Return JSON object with information about the progress of an upload.
    """
    progress_id = ''
    if 'X-Progress-ID' in request.GET:
        progress_id = request.GET['X-Progress-ID']
    elif 'X-Progress-ID' in request.META:
        progress_id = request.META['X-Progress-ID']
    if progress_id:
        from django.utils import simplejson
        cache_key = "%s_%s" % (request.META['REMOTE_ADDR'], progress_id)
        data = cache.get(cache_key)
        return HttpResponse(simplejson.dumps(data))
    else:
        return HttpResponseServerError('Server Error: You must provide X-Progress-ID header or query param.')

Comments

kayamb (on May 2, 2008):

Hi,

Thanks for those 2 snippets, they look very promising!

However, I can't quite figure out how to use them. In particular what kind of view should the html form's action attribute point to, and how would that view handle the form? Should that view wait for the completion of the upload before returning anything?

Here follows my first shot at making that view. Is that a good start, and if so, what is missing?

def frontpage(request):
    if request.method == 'POST':
        # [????] I don't know what to put there!...
    else:
        form = UploadForm()
    return render_to_response('frontpage.html', { 'form': form }, context_instance=RequestContext(request))

Thanks a lot!

Julien

kayamb (on May 3, 2008):

Never mind the comment above, I think I hadn't applied the patch properly.

revolunet (on July 10, 2008):

Hello and thanx a lot for these great two samples. Is it compatible with actual trunk ?

Can you show an example of the view who 'starts' the upload please ? i cannot figure it :(

can i use request.upload_handlers.insert(0, UploadProgressCachedHandler()) in my view instead of modify the settings.py ?

revolunet (on July 17, 2008):

hello

ok after some hours i finally made it work with some changes :

im using apache2+mod_wsgi with latest trunk

i use session instead of cache middleware, with a unique sessionkey for each upload ID. i have to use self.request.session.save() in the receive_data_chunk to update the data for the progress view.

instead of tweaking settings.py, i add in my view :

if request.method == 'POST':
    request.upload_handlers.insert(0, UploadProgressSessionHandler(request))

Thanks !

revolunet (on July 17, 2008):

in my upload_file_handler, i had to add

os.remove(f.temporary_file_path())

so the temporary file in /tmp can be removed

tuttle (on July 26, 2008):

Thanks for this snippet!

Anyway, I think it is worth mentioning that this will

1) NOT currently work with Django devel webserver (cannot serve multiple requests at once),

2) NOT work with a cache not sharing data between processes such as locmem://.

Finally I made this to work on prefork Apache with memcached://.

t3mp0 (on August 29, 2010):

To get this work with Django 1.2, line 1 needs to be changed to:

from django.core.files.uploadhandler import FileUploadHandler

williamcai (on December 11, 2010):

its great. but as tuttle's comments, it didn't work with Django develop web-server, which took me some time to find this. It's better to put this notice somewhere remarkable.

jhl (on December 12, 2010):

With a couple of exceptions, header parameter names are typically changed by the server. For example, X-Progress-ID would become HTTP_X_PROGRESS_ID. I haven't tested this, but the conditional clause

elif 'X-Progress-ID' in self.request.META:
    self.progress_id = self.request.META['X-Progress-ID']

should probably be changed to

elif 'HTTP_X_PROGRESS_ID' in self.request.META:
    self.progress_id = self.request.META['HTTP_X_PROGRESS_ID']

aaronfay (on May 13, 2011):

I think it's also worth noting that the GET is important, I tried to put the X-Progress-ID in my form and then access it with 'self.request.POST' but the POST data isn't ready yet! Infinite recursion is the bomb...

Aaron

jachetto (on September 21, 2012):

I'm not able to make it working in Admin. Looks like something goes wrong with csrf. After submission i get a csrf error protection (i'm just tryng to add a progress bar in my standard django admin ui).

Any idea?

maciej.m (on October 28, 2012):

The size of uploaded data differ from length of the file:

{"uploaded": 3997696, "length": 3954569}

Please login first before commenting.

Upload progress handler using cache framework

More like this

Comments