Login

SQLLoggerMidleware + infobar

Author:
robvdl
Posted:
January 19, 2008
Language:
Python
Version:
.96
Score:
2 (after 2 ratings)

This middleware will add a log of the SQL queries executed at the top of every page, in the form of an IE-like 'infobar'. It also includes a query count and total and per-query execution times.

This snippet is based on snippet #344 "SQL Log Middleware + duplicates" by guettli.

I did some adjustments to the code, to also add support for the application/xhtml+xml mime type, aswell as add the code for the infobar.

Need DEBUG on, aswell as DEBUG_SQL, should not be used on production sites.

It also expects a 16x16 png image called infobar_icon.png, which it is looking for in the /media/images folder (or /static/images, depending on your MEDIA_URL setting). I used a little light bulb icon from the Tango icon set, but am unable to attach the image. Any 16x16 png will do off course.

Update 0.1.1: In the initial version of this middleware, the path to /media/images was hard-coded, and you had to adjust this middleware accordingly. Now, it correctly uses the MEDIA_URL setting from your settings.py file, so you no longer have to edit this middleware.

0.1.2: Made some adjustments to the CSS to get rid of a padding bug when the bar was displayed on a Django error page. Also if a page contains no queries, it won't bother printing an 'empty' table, with just column headers.

0.1.3: More tweaks to the CSS, odd/even row shading on queries table.

0.1.4: More CSS tweaks again

0.1.5: Minor tweaks

0.1.6: Sorry, I just realised that after some time, this snippet broke in the latest SVN of Django with a Unicode error, as arthur78 mentioned.

I have managed to fix it, by wrapping line 258 and 259 in an str() function.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
"""
$Id: SQLLogMiddleware.py 306 2007-10-22 14:55:47Z tguettler $

This middleware
in settings.py you need to set

DEBUG=True
DEBUG_SQL=True

# Since you can't see the output if the page results in a redirect,
# you can log the result into a directory:
# DEBUG_SQL='/mypath/...'

MIDDLEWARE_CLASSES = (
	'YOURPATH.SQLLogMiddleware.SQLLogMiddleware',
	'django.middleware.transaction.TransactionMiddleware',
	...)

Slightly modified from original by Rob van der Linde <[email protected]>

	- Now supports XHTML content types
	- Now generates a nice dropdown window/banner at the top of the page.
	  Only works in Firefox (and other standards compliant browsers), but
	  then, no serious developers use IE anyway...

"""

import re, os, time, datetime
from django.conf import settings
from django.db import connection
from django.template import Template, Context

_HEAD_SECTION_RE = re.compile(r'(</head.*>)', re.DOTALL)
_BODY_SECTION_RE = re.compile(r'(<body.*?>)', re.DOTALL)

class SQLLogMiddleware:
	start = None

	def process_request(self, request):
		self.start = time.time()

	def process_response (self, request, response):
		# self.start is empty if an append slash redirect happened.
		debug_sql = getattr(settings, "DEBUG_SQL", False)
		if (not self.start) or not (settings.DEBUG and debug_sql):
			return response
		timesql = 0.0
		for q in connection.queries:
			timesql += float(q['time'])
			seen = {}
			duplicate = 0
		for q in connection.queries:
			sql = q["sql"]
			c = seen.get(sql, 0)
			if c:
				duplicate += 1
			q["seen"] = c
			seen[sql] = c + 1
		t = Template('''
			<div id="sqllog">
				<span><a href="javascript:activate_slider();"><strong>Django SQL log for this page: click to toggle</strong></a></span>
				<div>
					<p>
						<strong>request.path:</strong> {{ request.path|escape }}<br />
						<strong>Total query count:</strong> {{ queries|length }}<br />
						<strong>Total duplicate query count:</strong> {{ duplicate }}<br />
						<strong>Total SQL execution time:</strong> {{ timesql }}<br />
						<strong>Total Request execution time:</strong> {{ timerequest }}<br />
					</p>
					{% if queries %}
						<table border="0" cellspacing="0">
							<tr>
								<th>Time</th>
								<th>Frequency</th>
								<th>SQL</th>
							</tr>
							{% for sql in queries %}
								<tr class="{% cycle 'odd' 'even' %}">
									<td>{{ sql.time }}</td>
									<td class="alright">{{ sql.seen }}</td>
									<td>{{ sql.sql }}</td>
								</tr>
							{% endfor %}
						</table>
					{% else %}
						<p>No SQL queries for ths page.</p>
					{% endif %}
					{% if duplicate %}
						<p class="dupl">To avoid duplicates, read: <a href="http://www.djangoproject.com/documentation/db-api/#caching-and-querysets" target="_blank">Caching and Querysets</a>.</p>
					{% endif %}
					<button onclick="about_sql_debug()">About</button>
				</div>
			</div>
		''')
		c = Template(r'''
			<style type="text/css">
				/*<![CDATA[*/
					/* I use the !important keyword on just about any element
					   here, mainly as a precaution. Since we want this to work
					   with any web page (if possible), We should make sure other
					   CSS styles from the underlying page do not interfere with
					   the infobar. This should work on _most_ webpages, but I do
					   expect that in some cases, you may need to add some minor
					   corrections in your site's CSS file. This is, only if some
					   of your styles are interfering with the infobar's styles.
					   Also, this thing is only expected to work in Firefox, and
					   other CSS compliant browsers, Opera, Safari, Konqueror, etc.
					   Most web developers don't use IE anyway, which is who this
					   middleware is aimed at. It's not recommended to run on
					   production sites for security.
					*/

					html {
						padding-top: 24px !important;
					}

					#sqllog {
						font: normal 8px "MS Sans Serif", sans !important;
						color: black !important;
						background: #ffffe1 !important;
						position: fixed !important;
						top: 0 !important;
						left: 0 !important;
						width: 100% !important;
						height: 24px !important;
						border-bottom: 2px outset !important;
						z-index: 255 !important;
						overflow: hidden !important;
						padding: 0 !important;
						margin: 0 !important;
						line-height: normal !important;
					}

					#sqllog div {
						overflow: auto !important;
						height: 276px !important;
					}

					#sqllog.slideropen {
						height: 300px !important;
					}

					html.slideropen {
						padding-top: 300px !important;
					}

					#sqllog table, #sqllog tr, #sqllog td, #sqllog th {
						color: black !important;
						border: none !important;
					}

					#sqllog table {
						margin: 0 4px 12px 4px !important;
					}

					#sqllog td {
						padding: 2px 0 !important;
						vertical-align: top !important;
					}

					#sqllog th {
						background-color: #ffffe1 !important;
						padding: 2px 16px 2px 0 !important;
						font: bold 8px "MS Sans Serif", sans !important;
						text-align: left !important;
					}

					#sqllog .alright {
						padding-right: 20px !important;
						text-align: right !important;
					}

					#sqllog .odd {
						background-color: #ffffe1 !important;
					}

					#sqllog .even {
						background-color: #f4f4c8 !important;
					}

					#sqllog span a {
						color: black !important;
						display: block !important;
						padding: 5px 4px 0 26px !important;
						height: 19px;
						background-image: url("''' + settings.MEDIA_URL + 'images/infobar_icon.png") !important;' + r'''
						background-repeat: no-repeat !important;
						background-position: 4px 3px !important;
						cursor: default !important;
						text-decoration: none !important;
					}

					#sqllog span a:hover {
						text-decoration: none !imporant;
						color: HighlightText !important;
						background-color: Highlight !important;
					}

					#sqllog a {
						color: #5b80b2 !important;
						text-decoration: none !imporant;
					}

					#sqllog a:hover {
						color: #003366 !important;
						text-decoration: none !imporant;
					}

					#sqllog p {
						margin: 12px 4px 12px 4px !important;
						line-height: normal !important;
					}

					#sqllog p strong {
						display: block !important;
						float: left !important;
						width: 220px !important;
					}

					#sqllog p.dupl {
						margin: 0 4px 12px 4px !important;
					}

					#sqllog button {
						margin: 0 0 12px 4px !important;
					}
				/*]]>*/
			</style>
			<script type="text/javascript">
				/*<![CDATA[*/
					function activate_slider()
					{
						var sqllogClass = document.getElementById('sqllog').className
						var htmlClass = document.getElementsByTagName('html')[0].className;
						if (sqllogClass.search(/slideropen/) == -1)
						{
							document.getElementById('sqllog').className += ' slideropen';
							document.getElementsByTagName('html')[0].className += ' slideropen';
						}
						else
						{
							sqllogClass = sqllogClass.replace(/slideropen/, '');
							htmlClass = htmlClass.replace(/slideropen/, '');
							document.getElementById('sqllog').className = sqllogClass;
							document.getElementsByTagName('html')[0].className = htmlClass;
						}
					}

					function about_sql_debug()
					{
						alert('Django SQL Debugger 0.1.6\\n\\nA free middleware (filter), for use in any Django application. Shows the SQL queries generated by your web applications in realtime, using an \'IE style\' collapsable infobar at the top of every page. To get rid of this bar, the web developer should disable this middleware from the web application\\'s settings.py file.\\n\\nOriginal code "SQLLogMiddleware + duplicates", from Django Snippet #344, by "guettli".\\nModifications & Javascript + CSS implementation of the Infobar by Rob van der Linde.\\n\\nUnknown Licence, I would like to go with BSD, but that depends on the original author\'s Licence.');
					}
				/*]]>*/
			</script>
		''')
		timerequest = round(time.time() - self.start, 3)
		queries = connection.queries
		html = str(t.render(Context(locals())))
		css = str(c.render(Context(locals())))
		if debug_sql == True:
			if response.get("content-type", "").startswith("text/html") or response.get("content-type", "").startswith("application/xhtml+xml"):
				tag = _BODY_SECTION_RE.search(response.content)
				if tag:
					response.content = _BODY_SECTION_RE.sub(tag.group(0) + html, response.content)
					tag = _HEAD_SECTION_RE.search(response.content)
					if tag:
						response.content = _HEAD_SECTION_RE.sub(css + tag.group(0), response.content)
			return response
		assert os.path.isdir(debug_sql), debug_sql
		outfile = os.path.join(debug_sql, "%s.html" % datetime.datetime.now().isoformat())
		fd = open(outfile, "wt")
		fd.write('''<html><head><title>SQL Log %s</title></head><body>%s</body></html>''' % (request.path, html))
		fd.close()
		return response

More like this

  1. Template tag - list punctuation for a list of items by shapiromatron 10 months, 2 weeks ago
  2. JSONRequestMiddleware adds a .json() method to your HttpRequests by cdcarter 10 months, 3 weeks ago
  3. Serializer factory with Django Rest Framework by julio 1 year, 5 months ago
  4. Image compression before saving the new model / work with JPG, PNG by Schleidens 1 year, 6 months ago
  5. Help text hyperlinks by sa2812 1 year, 6 months ago

Comments

arthur78 (on March 13, 2008):

Hi, I am getting this error:

Traceback (most recent call last):

File "C:\Python25\Lib\site-packages\django\core\servers\basehttp.py", line 277, in run self.result = application(self.environ, self.start_response)

File "C:\Python25\Lib\site-packages\django\core\servers\basehttp.py", line 631, in call return self.application(environ, start_response)

File "C:\Python25\Lib\site-packages\django\core\handlers\wsgi.py", line 209, in call response = middleware_method(request, response)

File "d:\projects\karapuz..\karapuz\sql-logger.py", line 264, in process_response response.content = _BODY_SECTION_RE.sub(tag.group(0) + html, response.content)

UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 205: ordinal not in range(128)

#

robvdl (on June 9, 2008):

Sorry, I just realised that after some time, this snippet broke in the latest SVN of Django with a Unicode error, as arthur78 mentioned.

I have managed to fix it, by wrapping line 258 and 259 in an str() function.

#

Please login first before commenting.