A minimal Websockets setup with Django in production
A requirement came up. Obvious in hindsight.
Changes made to cells in one sheet should be reflected in the same sheet for other users. When the sheet is open in other browser tabs/windows.
This calls for “server side push” using web sockets. I.e. the server needs to push a notification to open “clients”.
Another way to do this would be to have client browsers Ajax-polling for changes. But that would be wasteful. Let’s only update the sheet when valid changes are saved!
What does minimal mean in this case?
This application is used by a team internally. Usage not exceeding ten concurrent users. The implications of this:
- One process to serve web socket requests is enough.
- No real performance testing was done.
- No consideration of alternatives to
daphnesuch as uvicorn or starlette. I picked up
daphnebecause it came up “first on the list of alternatives”. That’s it!
- No need to handle websocket interactions asynchronously in my case. You can read more about going sync vs async with websockets in Django channels here.
This configuration remains unchanged until problems crop up. Because “premature optimisation is the root of all evil”.
Blueprint: Before & After
http request protocol below refers both to
https. Same applies to
wss. Assume that for local development the protocol is unsecure, whilst secure in production.
Before introducing websockets, the web browser made an
http request to Nginx. At this point Nginx serves the request using
gunicorn, hitting Django1.
After adding websockets in the mix, Nginx still serves
http requests. But it’s now able to serve
ws requests by talking to
daphne. In this case you can replace
daphne with any other Websocket termination server:
The “new” item in the building blocks above is therefore daphne:
Daphne is a HTTP, HTTP2 and WebSocket protocol server for ASGI and ASGI-HTTP, developed to power Django Channels.
It supports automatic negotiation of protocols; there’s no need for URL prefixing to determine WebSocket endpoints versus HTTP endpoints.
daphne component can be replaced with alternatives as uvicorn or starlette.
The other item of note is that
ws:// connections are an “open” connection. “Data” travels in both directions along the same socket. As opposed to
daphne is part of the Django Channels effort:
Channels augments Django to bring WebSocket, long-poll HTTP, task offloading and other async support to your code, using familiar Django design patterns and a flexible underlying framework that lets you not only customize behaviours but also write support for your own protocols and needs.
I’ve followed the docs. The installation and excellent tutorial sections helped me get everything to work locally.
For completeness’ sake, these code changes are for an installation using these package versions:
channels==2.4.0 channels-redis==3.0.1 Django==3.0.8 redis==3.5.3
in a Python
The changes needed:
channelsto route websocket requests to the main channels entrypoint
- Routing code changes:
- main project-level routing entrypoint
- app level entrypoint(s), in this example using just one example
- The consumer that hosts all the event-handling and message sending logic our app needs to implement.
1. settings module changes
channels as the first app in my project’s list of
INSTALLED_APPS. Why first?
Please be wary of any other third-party apps that require an overloaded or replacement
runservercommand. Channels provides a separate
runservercommand and may conflict with it. An example of such a conflict is with
whitenoise.runserver_nostaticfrom whitenoise. In order to solve such issues, try moving channels to the top of your
INSTALLED_APPSor remove the offending app altogether.
Then added this new setting for
channels app to use:
1 2 3 4 5 6 7 8 9 10
Note that I already had
redis installed for caching and the application’s existing task queue with Huey.
2. Routing changes
I usually call the “default” app
proj. This makes it obvious that the app is a container for project-wide items. As is the case with this new
routing module. It contains the ProtocolTypeRouter that serves as main entry point for the ASGI application.
1 2 3 4 5 6 7 8 9 10 11 12
myapp is the test app used for this exmaple. The top-level router above contains a reference to a
myapp.routing module. This URLRouter routes
websocket type connections via their HTTP path.
myapp/routing.py contains the below:
1 2 3 4 5 6 7
channels allows us to structure our web socket URLs in the already familiar format we’re used for standard
3. The consumer
The final module that needs adding is the consumer. In
- Structure your code as a series of functions to be called whenever an event happens, rather than making you write an event loop.
- Allow you to write synchronous or async code and deals with handoffs and threading for you.
myapp/consumers.py implements a
SheetConsumer class which extends WebsocketConsumer:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53
The above is based on the Write your first consumer tutorial section. Instead of chat messages, the data is about a sheet’s cell updates. Updates that need to be applied for the same sheet open in other browser tabs/windows.
I’m passing the user ID of the authenticated user in
broadcaster_id. To be able to tell which user “triggered” the websocket message being “broadcasted”.
Give it a try locally
This is another great feature of
channels. Not even your usual
manage.py runserver workflow needs to change. Just note the new item in your default
runserver output when it starts:
$ ./manage.py runserver Watching for file changes with StatReloader Performing system checks... System check identified no issues (0 silenced). August 01, 2020 - 16:07:41 Django version 3.0.8, using settings 'proj.settings.local' Starting ASGI/Channels version 2.4.0 development server at http://127.0.0.1:8000/ Quit the server with CONTROL-C.
The socket handshakes are also shown in the output:
WebSocket HANDSHAKING /ws/sheet/sheet1/ [127.0.0.1:65181] WebSocket CONNECT /ws/sheet/sheet1/ [127.0.0.1:65181]
Awesome! Let’s deploy!
Not so fast 😊
I followed the
channels documentation here alongside Django’s own documentation on deploying ASGI applications here. But I applied ~~two~~ three tweaks that I’d rather explain.
daphne command tweak
I experienced the
CRITICAL Listen failure: [Errno 88] Socket operation on non-socket exception described here.
This was fixed by following the suggestion to remove the
-fd 0 switch. This switch is suggested by default in the channels docs.
In the current use case I do not need to use this switch. Because I do not need to bind multiple Daphne instances to the same port my production instance. In case I do, I will need to change my structure (see next section) to have
daphne called directly from supervisor. Rather than via bash script.
I had implemented
proj/asgi.py as described in the channels docs here. This works fine locally. But it led to this exception described here when executing web socket requests in production. I changed the
proj/asgi.py as described in this stackoverflow answer which makes it have this content:
1 2 3 4 5 6 7 8 9
This change replaces usage of
channels.routing.get_default_application. The stackoverflow answer above is supported as per
channels‘ docs here.
I do not know whether this fixes things properly. Should I should have done something else? It appears to be a small incompatibility between
channels and Django docs.
channels docs suggest creating
asgi.py from scratch. While Django 3.0.8 auto-created
If you have a better resolution to this please let me know (comment below).
Recall that my configuration is using channels_redis as backing store.
Since my production application runs on Ubuntu 18.04 LTS, default
apt-get redis version was
This resulted in this weird
BZPOPMIN - "ERR unknown command 'BZPOPMIN'" error. This is because redis version 5 or higher is needed.
Therefore please upgrade redis for your configuration. In my case I’ve followed the quickstart docs, especially the “Installing Redis more properly” section.
On to the production config files!
Deployment - resulting configuration
My configuration’s components:
- An executable bash script runs
daphne. I use this to be able to run and test
daphnedirectly in the Django project’s virtualenv.
- A supervisor
conffile to have this bash script process managed by supervisor.
- Nginx, of course.
start_daphne.bash contents. Remember to
chmod +x your bash script.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
File located at:
/etc/supervisor/conf.d/daphne.conf. Remember to create the log file directories.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
Nginx configuration references I used: channels deployment docs and this answer on stackoverflow. Follow those links to understand what I did.
Relevant Nginx config contents:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
Note the newly-added
This is not configuration as such. But as you can see the whole tutorial did not tackle
wss usage. One reason is that in this project’s case the SSL certificate part is not handled by Nginx. Since the project is using Cloudflare SSL, Cloudflare takes care of it even “before Nginx”.
wss logic I have is done at client-side level. This allows the same code to use the correct protocol locally and in production. The code sets up the connection depending on the current
http protocol in use:
1 2 3 4 5 6 7 8 9 10 11 12
Please let me know (in the comments below) whether anything I’ve done is wrong or I can improve it.
This was my first experience with Websockets and Django together. And it was pleasant one. The few “conflicting docs” issues described above, although blocking, were kinda expected.
Credits: Diagram above drawn using excalidraw.com.