Saturday, 23 September 2017

Anet Part 1: Negotiating With the Clients

This Anet thing has been sitting around on my HDD for ages. There's two parts to the Anet server - negotiating with the clients, and then the service table subscription, distribution & keep alive pings. This covers the first part. The second part (tables and pings) to follow when I finally get around to writing things down...

Determining Addresses

So, the initial setup here is that we have an ANET server on the primary network, and a client machine running behind a switch. The client is the machine that will actually host games for other clients.
This setup looks like this:

 --------------         ---------------         ------------------------
|    Server    |       |     Switch    |       |      Game Machine      | 
|  10.82.129.5 |<----->| 10.82.129.114 |<----->|     192.168.0.120      |
|   Runs ANET  |       |               |       | Hosting and/or playing |
 --------------         ---------------         ------------------------

So, how does the server get to the client?

Although the game machine has an IP address, it's behind a switch. NAT means that the address the server will see packets coming from will not match the address that the game machine thinks it has. However the game knows the IP address of the server, and it also knows the port to start talking to ANET on (fixed at 21143).

It is actuallly up to the game to determine the IP address that the server should use to contact it, and to do that it has to obtain the public address of the switch that it is hidden behind. This is why the patched versions are required to work on a modern connection - the game is determining the address that the outside world must reply to in order to establish a connection, and only newer games know how.

The game determines this using uPnP and IGD. The network trace from the client side shows HTTP/SOAP access to the router to retrieve this information: This, very roughly, follows the sequence

  • SSDP to find the router
  • request to router "GET /gatedesc.xml HTTP/1.1 "
  • The response shows us how to issue the request to the gateway
  • request to router "POST /upnp/control/WANIPConn1," with GetStatusInfo
  • The response gives us the router status
  • request to router "POST /upnp/control/WANIPConn1" with GetExternalIP
  • The response gives us the external IP address (10.82.129.114 in this case). This is the address we give to anet.
  • request to router "POST /upnp/control/WANIPConn1" with AddPortMapping
    • This request has the description "I76Nitro"
    • We pass our (client) IP address - in this test it was 192.168.0.120, however this private address never reaches anet
    • We pass a port number to use (21143 in this case) - this is the same number we'll give anet
  • The response lets us know we're good to go
At this point the game is in a position to supply the server with an "external" IP address to talk back to, and is able to communicate on the target port - the router handles external to internal IP port mapping. When we're done then we issue
  • request to router "POST /upnp/control/WANIPConn1" with DeletePortMapping
  • The response lets us know we've torn down

As an aside; This is also why direct links don't work between ANET and the game when they're on the same LAN segment. This attempt to obtain the game machine "external" IP doesn't manage to get the correct switch properties when there isn't an external IP to get.

For a more "modern" approach to this problem of UDP around NAT then look up WebRTC's STUN/TURN/Relay behaviour.

So, how does the game talk to the server?

The game and the server talk to each other over UDP. The server sits listening on port 21157 for incoming data packets, and the game opens a (random) port to talk on.

The basic format of an Anet data packet on the wire is a UDP packet that starts the payload data with a 'd' (i.e. 0x64). This is followed by another character that tells us what kind of packet this is, and from there we can parse out the data and understand the message.

There are actually a handful of different types of packets, but since the top level packets can actually be made up of combinations of other packets then this gets involved fairly quickly, however in practice the kind of packets and the data that gets sent is fairly easy to follow.

At the top level you will see "Syn", "Ack", "Data", "Ping" and "Ping Response" packets. Other packet types will be contained inside Data Packets. Occasionally multiple packets will be sent in a single "Gather" packet, which combines a number of top level packets into a single transmission. In Ruby-speak we can filter the common types of packets with code like this:

 
 case s
 when 'dY'
     @type = PacketTypes::TYPE_SYN;
 when 'dU'
     @type = PacketTypes::TYPE_ACK
 when 'dB'
     @type = PacketTypes::TYPE_PING
 when 'dC'
     @type = PacketTypes::TYPE_PINGRESPONSE
 when 'dT'
     @type = PacketTypes::TYPE_DATA
 when 'dG'
     @type = PacketTypes::TYPE_GATHER
 when 'd('
     @type = PacketTypes::TYPE_TSERV
 when 'e1'
     @type = PacketTypes::TYPE_ADDCLIENT
 when 'd^'
     @type = PacketTypes::TYPE_SUBSCRIBE
 when 'd&'
     @type = PacketTypes::TYPE_UNSUBSCRIBE
 when 'd%'
     @type = PacketTypes::TYPE_SMALL
 else
 ...
 
Note the "AddClient" type, which starts with 'e'. Although we can look at most Anet packets as starting with 'd' the client add request starts with 'e'. However that's contained in a data packet (dT) so top level packets will always start with a 'd'. More on that later.

The top level packets that get sent (Syn, Ack and Data) have a two byte packet number associated with them, and the sender will expect a return with this identifier to indicate reception. The main exception to this are Ping and Ping Response packets, which have a single byte sequence number. More on this later

So, how does the game establish a connection to the server?

The game sends a SYN packet to the server to announce that it's ready to host a game. It expects the server to reply with an ACK to acknowledge receipt of this packet, and a SYN from the server.

So the sequence is:

  • Game sends a SYN to the server
  • Server sees the SYN and sends back a SYN to the game on the open port
  • The Server sends an ACK for the game SYN
  • The Server waits for the ACK from the game acknowledging receipt of the SYN

The Actual Server/Game Handshake in detail

Given this and the anet server code we can trace the activity as the game connects to the server. Dumping some Wireshark traces from the server end then we can see:

 
Internet Protocol Version 4, Src: 10.82.129.114, Dst: 10.82.129.5
User Datagram Protocol, Src Port: 21143, Dst Port: 21157
Data (26 bytes)
0000  64 59 11 78 15 05 06 0a 52 81 72 52 97 0a 52 81   dY.x....R.rR..R.
0010  05 52 a5 07 0a 52 81 72 52 97                     .R...R.rR.

This is the initial SYN from the game to the ANET server The SYN packet breaks down as:

  • 64 59: dY - a SYN packet
  • 11 78 : 16 bit packet number (0x7811)
  • 15 : Packet Length (21 bytes)
  • 05 : Version = 5
  • 06 : Address Size
  • 0a 52 81 72 52 97 : Src Address #1
  • 0a 52 81 05 52 a5 : Dest Address
  • 07 : Capabilities
  • 0a 52 81 72 52 97 : Src Address #2
The 6 byte address is the four byte IP plus two bytes for the port number. In this case it's:
  • Source Address 1&2: 0a 52 81 72 52 97 => 10.82.129.114:21143
  • Dest Address: 0a 52 81 05 52 a5 => 10.82.129.5:21157
The anet server uses this address to reply on. The capabilities are given by comm_driverInfo_t, and 0x07 means "is visible, knows the playerlist and you can send it a gamelist".

Now that the game has announced itself to the anet server, the server replies with a SYN of it's own to the game, and also acknowledges receipt of the game's SYN packet.

Internet Protocol Version 4, Src: 10.82.129.5, Dst: 10.82.129.114
User Datagram Protocol, Src Port: 21157, Dst Port: 21143
Data (26 bytes)
0000  64 59 72 f6 15 05 06 0a 52 81 05 52 a5 0a 52 81   dYr.....R..R..R.
0010  72 52 97 07 0a 52 81 05 52 a5                     rR...R..R.

This is a SYN that the server sends to the client that has just announced itself. This has the same format as the incoming SYN, with the following unique properties:

  • 72 f6 : 16 bit packet number (0xf672)
  • 0a 52 81 05 52 a5 : Src Address #1
  • 0a 52 81 72 52 97 : Dest Address
  • 0a 52 81 05 52 a5 : Src Address #2
In this case Src Address #1&2 are 10.82.129.5:21157 and Dest Address is 10.82.129.114:21143

These two SYN values show us attempting to establish a symmetric link. The server has chosen a unique packet ID, but in this case then the source and address are fairly simple.

Internet Protocol Version 4, Src: 10.82.129.5, Dst: 10.82.129.114
User Datagram Protocol, Src Port: 21157, Dst Port: 21143
Data (5 bytes)
0000  64 55 11 78 80                                    dU.x.

And this is the ACK from the server back to the client. The packet breaks down as:

  • 64 55: dU - an ACK packet
  • 11 78 : 16 bit packet number - this was the packet ID in the game SYN (0x7811)
  • 80 : packet number offset. The value of 0x80 means "ignore this field".

Next up we're waiting for the client to ACK our SYN. This takes the client several seconds to work through, so we actually wind up resending the SYN request a couple of times. This is a identical resend, and we attempt this every few seconds (roughly 3 or four seconds apart)

So we see these re-transmissions

 174161 64.286843956   10.82.129.5           10.82.129.114         UDP      68     21157 -> 21143 Len=26
 180170 66.814850322   10.82.129.5           10.82.129.114         UDP      68     21157 -> 21143 Len=26
 188610 70.958528631   10.82.129.5           10.82.129.114         UDP      68     21157 -> 21143 Len=26
 195822 75.101658184   10.82.129.5           10.82.129.114         UDP      68     21157 -> 21143 Len=26
 209281 79.245317336   10.82.129.5           10.82.129.114         UDP      68     21157 -> 21143 Len=26
 217715 83.389025871   10.82.129.5           10.82.129.114         UDP      68     21157 -> 21143 Len=26
 226179 87.542323140   10.82.129.5           10.82.129.114         UDP      68     21157 -> 21143 Len=26
 234626 91.686158758   10.82.129.5           10.82.129.114         UDP      68     21157 -> 21143 Len=26
 

At this point the ACK comes back from the client.

Internet Protocol Version 4, Src: 10.82.129.114, Dst: 10.82.129.5
User Datagram Protocol, Src Port: 21143, Dst Port: 21157
Data (5 bytes)
0000  64 55 72 f6 80                                    dUr..

This ACK from the client has the same format we used for ours. The packet breaks down as:

  • 64 55: dU - an ACK packet
  • 72 f6 : 16 bit packet number - this was the packet ID we sent in our SYN
  • 80 : packet number offset. Another "ignore this field".

Following this there's a cluster of ACK's, as the client catches up with the multiple SYN packets we sent out. Since we sent out identical SYN requests the corresponding ACK packets are all identical. Since we sent 8 duplicate SYNs then we also see 8 ACK returns. It looks like the ACK is only generated for outstanding SYNs when the game actually leaves the setup screen and the player on the host is on the game level.

 237028 92.782751239   10.82.129.114         10.82.129.5           UDP      60     21143 -> 21157 Len=5
 237029 92.782835138   10.82.129.114         10.82.129.5           UDP      60     21143 -> 21157 Len=5
 237030 92.782857474   10.82.129.114         10.82.129.5           UDP      60     21143 -> 21157 Len=5
 237031 92.782880754   10.82.129.114         10.82.129.5           UDP      60     21143 -> 21157 Len=5
 237032 92.782903744   10.82.129.114         10.82.129.5           UDP      60     21143 -> 21157 Len=5
 237033 92.782980110   10.82.129.114         10.82.129.5           UDP      60     21143 -> 21157 Len=5
 237034 92.783002020   10.82.129.114         10.82.129.5           UDP      60     21143 -> 21157 Len=5
 237035 92.783024780   10.82.129.114         10.82.129.5           UDP      60     21143 -> 21157 Len=5

So, how does the server communicate securely with the game?

Following this then we are in the information exchange phase. The ANET server sends out an initial challenge packet. This is sent as a data packet (type 'dT') which contains a Tserv packet (type 'd(')

This is Activision's "Trivial Challenge Authentication", used to secure password transfer. From the header it's used to perform:

response(challenge) = MD5(TDESn(MD5(password), challenge))

However since we don't use any of the user account stuff in Nitro, then we really don't care: We'll see the server send out the TSERV packet and the game will ACK the packet.

Next we'll look at the tables and how the available games are advertised.