Warning: this information is outdated. A more up to date technical draft can be found here [PDF], but is still outdated. Sorry :(
When launching tmate, an ssh connection is established to tmate.io (or your own server) in the background through libssh. The server ssh key signatures are specified upfront and are verified during the DH exchange to prevent man in the middle attacks.
When a connection is established, a 150 bits session token is generated, then a tmux server is spawned in a jail with no file system, with its own PID namespace to isolate the server from other processes, and no user privileges. To allow this, all files required during the tmux server execution are opened before getting jailed. These measures are in place to limit the usefulness of possible exploits targeting the tmux server. The attacker would not be able to access other sessions, ensuring confidentiality.
When an ssh client connects to tmate.io (or your own server), the tmux unix socket is looked up on the file system. On lookup failures, a random sleep is performed to prevent timing attacks, otherwise a tmux client is spawned and connected to the remote tmux server.
The local and remote tmux servers communicate with a protocol on top of msgpack, which is gzipped over ssh for network bandwidth efficiency as vim scrolling can generate massive amounts of data.
In order to keep the remote tmux server in sync with the local tmux server, PTY window pane's raw outputs are streamed individually as opposed to synchronizing the entire tmux window. Furthermore, window layouts, status bar changes, and copy mode state are also replicated. Finally, most of the tmux commands (like bind-key) are replicated. This ensures that the key bindings are the same on both side.
The remote client's keystrokes are parsed and the outcome is sent to the local tmux server. This includes tmux commands such as split-window, window pane keystrokes, or window size information.
This project can take many interesting directions.
Here is what I have on the roadmap: