Login

Log all interaction with user to the DB

Author:
inuwashi
Posted:
January 17, 2011
Language:
Python
Version:
1.1
Score:
1 (after 1 ratings)

Due to compliance requirements in the financials industry we needed to log every request a user made to our system, the action taken (view function) and response from the server.

I found a lot of other logging solution bit most revolved around debugging and DB query logging. I needed to be able to tell what a user did while being logged in as much detail as I could with out tracking the mouse pointer position on screen.

So I created this (, my first ,) middleware. Its very simple really. keeping track of a request, view_func and response object in a single model called Record (models.py file included in the code).

The fields I used are optimized to what I intend to show in the UI I am planning for this model. Depending on how you use the doc string of your views they can be tapped to explain to the user what each request/func/responce group in a session is meant to do.

There were a few gotcha's: 1. I only care about authenticated requests. So I added the 'quest.user.is_authenticated()' test. 2. I did not care about the favicon request so I skipped them. 2. The actual login request is not authenticated while the response is. This caused the process_response/view to look for a record that is not there. So I added the 'except ObjectDoesNotExist' to skip this case.

I added one bell: Logging a full HTML reply is wasteful and mostly useless. I added two values in the setting files. LOGALL_LOG_HTML_RESPONSE to toggle if we want to log them or not. And LOGALL_HTML_START to describe what a full HTML starts with. Personally I use the first few characters of my base.html template that all the rest of my templates expend. I simplified the code to the left for readability. 

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
#settings.py

#Should we log full HTML responses?
LOGALL_LOG_HTML_RESPONSE = True

# If we how do we recognized a full HTML response 
LOGALL_HTML_START = "<!DOCTYPE html"


#logAll.models.py
from django.db import models

from django.contrib.auth.models import User

class Record(models.Model):
    """
    Basic log record describing all user interaction with the UI.
    Will be propagated by a middle ware.
    This will be one BIG DB table!
    """
    created_at = models.DateTimeField(auto_now_add = True)
    sessionId = models.CharField(max_length=256)

    requestUser = models.ForeignKey(User)
    requestPath  = models.TextField()
    requestQueryString = models.TextField()
    requestVars = models.TextField()
    requestMethod = models.CharField(max_length=4)
    requestSecure = models.BooleanField(default=False)
    requestAjax = models.BooleanField(default=False)
    requestMETA = models.TextField(null=True, blank=True)
    requestAddress = models.IPAddressField()

    viewFunction = models.CharField(max_length=256)
    viewDocString = models.TextField(null=True, blank=True)
    viewArgs = models.TextField()

    responseCode = models.CharField(max_length=3)
    responseContent = models.TextField()




#logAll.middleware.py

import time
import simplejson as json

from django.core.exceptions import ObjectDoesNotExist

from logAll.models import Record
from settings import LOGALL_LOG_HTML_RESPONSE, LOGALL_HTML_START

class LogAllMiddleware(object):

    def process_request(self,request):
        # Only log requests of authinticate users                                                                                                                                                               
        try:
            if not request.user.is_authenticated():
                return None
        except AttributeError:
            return None

        # Skip favicon requests cause I do not care about them
        if request.path =="/favicon.ico":
            return None

        newRecord = Record(
            created_at = str(time.time()),
            sessionId = request.session.session_key,

            requestUser = request.user,
            requestPath  = request.path,
            requestQueryString = request.META["QUERY_STRING"],
            requestVars = json.dumps(request.REQUEST.__dict__),
            requestMethod = request.method,
            requestSecure = request.is_secure(),
            requestAjax = request.is_ajax(),
            requestMETA = request.META.__str__(),
            requestAddress = request.META["REMOTE_ADDR"],
            )

        newRecord.save()

        return None

    def process_view(self, request, view_func, view_args, view_kwargs):
        try:
            if not request.user.is_authenticated():
                return None
        except AttributeError:
            return None

        # Fix the issue with the authrization request                                                                                                                                                        
        try:
            theRecord  = Record.objects.get(
                sessionId = request.session.session_key,
                requestUser = request.user,
                requestPath  = request.path,
                requestMethod = request.method,
                requestSecure = request.is_secure(),
                requestAjax = request.is_ajax(),
                requestMETA = request.META.__str__()
                )
            theRecord.viewFunction = view_func.func_name
            theRecord.viewDocString = view_func.func_doc
            theRecord.viewArgs = json.dumps(view_kwargs)

            theRecord.save()
        except  ObjectDoesNotExist:
            pass

        return None


    def process_response(self, request, response):

        # Only log autherized requests
        try:
            if not request.user.is_authenticated():
                return response
        except AttributeError:
            return response

        # Skip favicon requests cause I do not care about them
        if request.path =="/favicon.ico":
            return response


        # Fix the issue with the authorization request                                                                                                                                                                          
        try:
            theRecord  = Record.objects.get(
                sessionId = request.session.session_key,
                requestUser = request.user,
                requestPath  = request.path,
                requestMethod = request.method,
                requestSecure = request.is_secure(),
                requestAjax = request.is_ajax(),
                requestMETA = request.META.__str__()
                )

            theRecord.responseCode = response.status_code

            # Decidce wether we want to log the a full html response
            # as this will probabaly will take a LOT of space.
            #
            # In my case most of the replies I want to catch happen
            # to be plain text ajax replies

            if LOGALL_LOG_HTML_RESPONSE:
                # IF set to true then log the respoce regardless                                                                                                                                                               
                theRecord.responseContent = response.content
            elif response.content.startswith(LOGALL_HTML_START):
                theRecord.responseContent = "FULL HTML RESPONSE"
            else:
                theRecord.responseContent = response.content

            theRecord.save()

        except  ObjectDoesNotExist:
            pass

        return response

More like this

  1. Template tag - list punctuation for a list of items by shapiromatron 11 months, 2 weeks ago
  2. JSONRequestMiddleware adds a .json() method to your HttpRequests by cdcarter 11 months, 3 weeks ago
  3. Serializer factory with Django Rest Framework by julio 1 year, 6 months ago
  4. Image compression before saving the new model / work with JPG, PNG by Schleidens 1 year, 7 months ago
  5. Help text hyperlinks by sa2812 1 year, 8 months ago

Comments

dudus (on January 19, 2011):

Wont you log request datetime?

#

code_shogan (on February 20, 2011):

line 150 to 156 with the code block that begins if LOGALL_LOG_HTML_RESPONSE:` looks a little strange, in fact it may as well look like this...

if response.content.startswith(LOGALL_HTML_START):
    theRecord.responseContent = "FULL HTML RESPONSE"
else:
    theRecord.responseContent = response.content

at the moment regardless of what LOGALL_LOG_HTML_RESPONSE is your code just like my example above calls theRecord.responseContent = response.content anyway.

I don't think this is what you wanted...

#

Please login first before commenting.