Too many Invalid HTTP_HOST header exception errors
Have you deployed an application to production?
Are you getting too many, seemingly random Invalid HTTP_HOST header
exception errors?
Background
I have recently deployed a Django app:
- served via gunicorn/Nginx
- onto an AWS Lightsail instance
- with a CloudFlare certificate for https/SSL (followed this answer on StackOverflow for configuring it)
Following deployment I had a barrage of Invalid HTTP_HOST header
error emails. Every. Single. Day.
I looked at the Nginx configuration. But it looked identical to other configurations I have in production. Configurations that do not have this kind of error.
The Nginx server
block looks like:
upstream dbr_project {
server unix:/home/ubuntu/[..]/gunicorn.sock fail_timeout=0;
}
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name subdomain.example.com;
if ($http_x_forwarded_proto = "http") {
return 301 https://$server_name$request_uri;
}
...
}
On further inspection, these exceptions were being caused by bots. Example user agent strings:
HTTP_USER_AGENT = 'Mozilla/5.0 (compatible; Nimbostratus-Bot/v1.3.2; http://cloudsystemnetworks.com)'
HTTP_USER_AGENT = 'masscan/1.0 (https://github.com/robertdavidgraham/masscan)'
Even though /robots.txt
is set up to prevent robots visiting all URLs. This is its content:
User-agent: *
Disallow: /
So the cause looks external. And the “dirty” fix I resorted to is to stop reporting this class of error.
Fix
Update: I revised this fix following the comments made below. What I applied to finally fix this is an Nginx more than a Django fix. I describe the Nginx change I made in the section “Revised Solution” below.
I’ve followed this answer on StackOverflow to stop reporting, or “suppress”, this error.
My previous LOGGING
config (which I stripped down to a bare minimum for this post) looked like:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
|
To stop reporting this error:
- I’ve added a
null
handler; docs onLOGGING
handlers here - set the
handler
for loggingdjango.security.DisallowedHost
exception to use thisnull
handler
This is how the logging configuration looks like after the changes. The newly-added handler and logger are highlighted:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
|
Final Thoughts
I’m not sure this is the best “solution”. In fact I regard it a “dirty fix” more than a solution.
Because ideally I wouldn’t be suppressing any django.security
exception notifications.
But it follows Django’s standard machinery to suppress logging a specific exception. Which is a good thing.
Let me know if you handle this differently. Or in any way better!
P.S. Sentry or similar can group this exception into one “error” page. Which in effect contains thousands of this exception’s occurrence. That does not count as a better approach 😊
Revised Solution
Take a look at the comments thread below. Bots are effecting requests without passing the correct host
.
Solution is to enforce the domain subdomain.example.com
can using Nginx. This is the resulting Nginx config:
upstream subdomain_project {
# fail_timeout=0 means we always retry an upstream even if it failed
# to return a good HTTP response (in case the Unicorn master nukes a
# single worker for timing out).
server unix:/home/ubuntu/[..]/gunicorn.sock fail_timeout=0;
}
server {
listen 80 default_server;
listen [::]:80 default_server;
if ($host !~* ^(subdomain.example.com)$ ) {
return 444;
}
}
server {
server_name subdomain.example.com;
...
}
Notice how:
- the domain
subdomain.example.com
is being handled by the secondserver
block - unless the request’s
host
issubdomain.example.com
, then HTTP 444 is returned.
Comments !