Server Architecture
Introduction
Our game servers are hosted on a dedicated Ubuntu server in Falkenstein, Germany, rented from Hetzner as a custom AX42 machine. Alongside the standard 2x 512GB SSDs configured in RAID, we’ve added a 1TB SSD for non-Survival game servers, and a 22TB HDD for backups and cold storage.
To configure and manage our game servers and various community bots on this Ubuntu server, we rely on the Pterodactyl server management panel.
Game Servers
Pterodactyl runs all of our game servers in isolated Docker containers. We can then manage these game servers through the Pterodactyl UI panel, as well as the SFTP access it provides.
While all these servers effectively sit alongside each other, we use a BungeeCord server, referred to as our Proxy server, to connect several servers together for the Resource World feature. The following diagram represents the visual hierarchy, and our Proxy setup is explained further below.
Proxy Network
Our proxy network handles connections to and from the Survival and Resource that servers sit 'below' the Proxy in the hierarchy, and are collectively considered to be our "Main" server.
The Proxy server utilises a shared set of MySQL databases for a number of use cases:
- Whitelist database
- Writes to this database to whitelist new players.
- NB: Our whitelist applies to all servers within both proxy servers.
- Reads from this database to authenticate connections.
- AdvancedBan database
- Reads from this database to prevent banned members connecting.
- NB: Our banlist applies to all servers within both proxy servers.
- LuckPerms database
- Reads from this database to synchronise permissions across the Bungee network.
- PremiumVanish database
- Reads from this database to synchronise staff members vanished status across the Bungee network.
For all of our databases, phpMyAdmin provides a useful web interface for access and administration.
In addition to the Proxy Databases, we use more MySQL databases in the main network for the following use cases:
- Resource World Gatekeeper database
- Writes to this database to save entity data before transferring servers
- Reads from this database to load entity data upon transferring servers
- CoreProtect database
- Writes to this database to log player progression and history
- Reads to this database to restore previous server data
- Plan database
- Writes to this database to log player analytics
As noted in the diagram, the Resource server does not interact with the Survival server directly, and vice versa. Players are transferred from one server to another via the Proxy. The same is true of chatlogs too - note that messages are also not sent from the Resource Server to #ingamechat directly.
Bots
While detailed here in their own section, these bots are hosted and managed via Pterodactyl much like our game servers. In fact, for Pterodactyl and Docker there is almost no functional difference between a game server and a bot. Hosting these bots via Pterodactyl helps to keep most staff applications managed through a single functional UI.
We run bots on our dedicated server for the following use cases:
- Appbot
Sends server applications to an#applications
channel, and whitelist commands to our servers - Modbot
Shares IPs, posts FAQs, checks pings, converts timezones, restarts servers, sends reminders (and plenty more!) - Chester
Forwards any DM or completed Member Report that you send it to a private staff channel - Emoji Roles
Assigns roles when reacting to certain messages, as seen in our #welcome channel - Playerlistbot
Checks player count and player statuses across Survival and Resource servers when!playerlist
is used in#ingamechat
- Snackers
Posts Etho-themed welcome messages in #general, and assigns the 'Inactive' role to members if they do not speak in the Discord for three weeks - MusicBot
Plays YouTube and Soundcloud songs in our voice channels, utilising a publicly available bot - ErgoBot
Runs automation tasks on our Pterodactyl servers based on API responses from external services, for example Mojang's version manifest
Backups
Our backups work through a combination of Restic and Backblaze. Restic provides automated incremental backups solution for all our server data, while Backblaze provides us with insanely cheap external storage, to avoid the dedicated server being a single point of failure.
We currently keep between 28 and 42 days worth of incremental backups on Backblaze, to balance the amount of available backups with the costs of storing and pruning this data. Pruning to 28 days is automatically run by a cronjob on the 1st and 15th of each month.
Rendered with Diagrams by mingrammer