Web crawler/bot detection and blocking middleware
Sets request.is_crawler Allow bot lockout from certain urls in urlconf ,add view parameter 'deny_crawlers' ex. url(r'^foo/$', 'views.foo',{'deny_crawlers' : True},name='foo') view param is removed after middleware pass.
- middleware
- bot
- yahoo
- crawler
- robot
- googlebot
- cuil
- slurp