Web crawler/bot detection and blocking middleware

Sets request.is_crawler Allow bot lockout from certain urls in urlconf ,add view parameter 'deny_crawlers' ex. url(r'^foo/$', 'views.foo',{'deny_crawlers' : True},name='foo') view param is removed after middleware pass.