Saturday, 23 September 2017

Anet Part 1: Negotiating With the Clients

This Anet thing has been sitting around on my HDD for ages. There's two parts to the Anet server - negotiating with the clients, and then the service table subscription, distribution & keep alive pings. This covers the first part. The second part (tables and pings) to follow when I finally get around to writing things down...

Determining Addresses

So, the initial setup here is that we have an ANET server on the primary network, and a client machine running behind a switch. The client is the machine that will actually host games for other clients.
This setup looks like this:

 --------------         ---------------         ------------------------
|    Server    |       |     Switch    |       |      Game Machine      | 
|  10.82.129.5 |<----->| 10.82.129.114 |<----->|     192.168.0.120      |
|   Runs ANET  |       |               |       | Hosting and/or playing |
 --------------         ---------------         ------------------------

So, how does the server get to the client?

Although the game machine has an IP address, it's behind a switch. NAT means that the address the server will see packets coming from will not match the address that the game machine thinks it has. However the game knows the IP address of the server, and it also knows the port to start talking to ANET on (fixed at 21143).

It is actuallly up to the game to determine the IP address that the server should use to contact it, and to do that it has to obtain the public address of the switch that it is hidden behind. This is why the patched versions are required to work on a modern connection - the game is determining the address that the outside world must reply to in order to establish a connection, and only newer games know how.

The game determines this using uPnP and IGD. The network trace from the client side shows HTTP/SOAP access to the router to retrieve this information: This, very roughly, follows the sequence

  • SSDP to find the router
  • request to router "GET /gatedesc.xml HTTP/1.1 "
  • The response shows us how to issue the request to the gateway
  • request to router "POST /upnp/control/WANIPConn1," with GetStatusInfo
  • The response gives us the router status
  • request to router "POST /upnp/control/WANIPConn1" with GetExternalIP
  • The response gives us the external IP address (10.82.129.114 in this case). This is the address we give to anet.
  • request to router "POST /upnp/control/WANIPConn1" with AddPortMapping
    • This request has the description "I76Nitro"
    • We pass our (client) IP address - in this test it was 192.168.0.120, however this private address never reaches anet
    • We pass a port number to use (21143 in this case) - this is the same number we'll give anet
  • The response lets us know we're good to go
At this point the game is in a position to supply the server with an "external" IP address to talk back to, and is able to communicate on the target port - the router handles external to internal IP port mapping. When we're done then we issue
  • request to router "POST /upnp/control/WANIPConn1" with DeletePortMapping
  • The response lets us know we've torn down

As an aside; This is also why direct links don't work between ANET and the game when they're on the same LAN segment. This attempt to obtain the game machine "external" IP doesn't manage to get the correct switch properties when there isn't an external IP to get.

For a more "modern" approach to this problem of UDP around NAT then look up WebRTC's STUN/TURN/Relay behaviour.

So, how does the game talk to the server?

The game and the server talk to each other over UDP. The server sits listening on port 21157 for incoming data packets, and the game opens a (random) port to talk on.

The basic format of an Anet data packet on the wire is a UDP packet that starts the payload data with a 'd' (i.e. 0x64). This is followed by another character that tells us what kind of packet this is, and from there we can parse out the data and understand the message.

There are actually a handful of different types of packets, but since the top level packets can actually be made up of combinations of other packets then this gets involved fairly quickly, however in practice the kind of packets and the data that gets sent is fairly easy to follow.

At the top level you will see "Syn", "Ack", "Data", "Ping" and "Ping Response" packets. Other packet types will be contained inside Data Packets. Occasionally multiple packets will be sent in a single "Gather" packet, which combines a number of top level packets into a single transmission. In Ruby-speak we can filter the common types of packets with code like this:

 
 case s
 when 'dY'
     @type = PacketTypes::TYPE_SYN;
 when 'dU'
     @type = PacketTypes::TYPE_ACK
 when 'dB'
     @type = PacketTypes::TYPE_PING
 when 'dC'
     @type = PacketTypes::TYPE_PINGRESPONSE
 when 'dT'
     @type = PacketTypes::TYPE_DATA
 when 'dG'
     @type = PacketTypes::TYPE_GATHER
 when 'd('
     @type = PacketTypes::TYPE_TSERV
 when 'e1'
     @type = PacketTypes::TYPE_ADDCLIENT
 when 'd^'
     @type = PacketTypes::TYPE_SUBSCRIBE
 when 'd&'
     @type = PacketTypes::TYPE_UNSUBSCRIBE
 when 'd%'
     @type = PacketTypes::TYPE_SMALL
 else
 ...
 
Note the "AddClient" type, which starts with 'e'. Although we can look at most Anet packets as starting with 'd' the client add request starts with 'e'. However that's contained in a data packet (dT) so top level packets will always start with a 'd'. More on that later.

The top level packets that get sent (Syn, Ack and Data) have a two byte packet number associated with them, and the sender will expect a return with this identifier to indicate reception. The main exception to this are Ping and Ping Response packets, which have a single byte sequence number. More on this later

So, how does the game establish a connection to the server?

The game sends a SYN packet to the server to announce that it's ready to host a game. It expects the server to reply with an ACK to acknowledge receipt of this packet, and a SYN from the server.

So the sequence is:

  • Game sends a SYN to the server
  • Server sees the SYN and sends back a SYN to the game on the open port
  • The Server sends an ACK for the game SYN
  • The Server waits for the ACK from the game acknowledging receipt of the SYN

The Actual Server/Game Handshake in detail

Given this and the anet server code we can trace the activity as the game connects to the server. Dumping some Wireshark traces from the server end then we can see:

 
Internet Protocol Version 4, Src: 10.82.129.114, Dst: 10.82.129.5
User Datagram Protocol, Src Port: 21143, Dst Port: 21157
Data (26 bytes)
0000  64 59 11 78 15 05 06 0a 52 81 72 52 97 0a 52 81   dY.x....R.rR..R.
0010  05 52 a5 07 0a 52 81 72 52 97                     .R...R.rR.

This is the initial SYN from the game to the ANET server The SYN packet breaks down as:

  • 64 59: dY - a SYN packet
  • 11 78 : 16 bit packet number (0x7811)
  • 15 : Packet Length (21 bytes)
  • 05 : Version = 5
  • 06 : Address Size
  • 0a 52 81 72 52 97 : Src Address #1
  • 0a 52 81 05 52 a5 : Dest Address
  • 07 : Capabilities
  • 0a 52 81 72 52 97 : Src Address #2
The 6 byte address is the four byte IP plus two bytes for the port number. In this case it's:
  • Source Address 1&2: 0a 52 81 72 52 97 => 10.82.129.114:21143
  • Dest Address: 0a 52 81 05 52 a5 => 10.82.129.5:21157
The anet server uses this address to reply on. The capabilities are given by comm_driverInfo_t, and 0x07 means "is visible, knows the playerlist and you can send it a gamelist".

Now that the game has announced itself to the anet server, the server replies with a SYN of it's own to the game, and also acknowledges receipt of the game's SYN packet.

Internet Protocol Version 4, Src: 10.82.129.5, Dst: 10.82.129.114
User Datagram Protocol, Src Port: 21157, Dst Port: 21143
Data (26 bytes)
0000  64 59 72 f6 15 05 06 0a 52 81 05 52 a5 0a 52 81   dYr.....R..R..R.
0010  72 52 97 07 0a 52 81 05 52 a5                     rR...R..R.

This is a SYN that the server sends to the client that has just announced itself. This has the same format as the incoming SYN, with the following unique properties:

  • 72 f6 : 16 bit packet number (0xf672)
  • 0a 52 81 05 52 a5 : Src Address #1
  • 0a 52 81 72 52 97 : Dest Address
  • 0a 52 81 05 52 a5 : Src Address #2
In this case Src Address #1&2 are 10.82.129.5:21157 and Dest Address is 10.82.129.114:21143

These two SYN values show us attempting to establish a symmetric link. The server has chosen a unique packet ID, but in this case then the source and address are fairly simple.

Internet Protocol Version 4, Src: 10.82.129.5, Dst: 10.82.129.114
User Datagram Protocol, Src Port: 21157, Dst Port: 21143
Data (5 bytes)
0000  64 55 11 78 80                                    dU.x.

And this is the ACK from the server back to the client. The packet breaks down as:

  • 64 55: dU - an ACK packet
  • 11 78 : 16 bit packet number - this was the packet ID in the game SYN (0x7811)
  • 80 : packet number offset. The value of 0x80 means "ignore this field".

Next up we're waiting for the client to ACK our SYN. This takes the client several seconds to work through, so we actually wind up resending the SYN request a couple of times. This is a identical resend, and we attempt this every few seconds (roughly 3 or four seconds apart)

So we see these re-transmissions

 174161 64.286843956   10.82.129.5           10.82.129.114         UDP      68     21157 -> 21143 Len=26
 180170 66.814850322   10.82.129.5           10.82.129.114         UDP      68     21157 -> 21143 Len=26
 188610 70.958528631   10.82.129.5           10.82.129.114         UDP      68     21157 -> 21143 Len=26
 195822 75.101658184   10.82.129.5           10.82.129.114         UDP      68     21157 -> 21143 Len=26
 209281 79.245317336   10.82.129.5           10.82.129.114         UDP      68     21157 -> 21143 Len=26
 217715 83.389025871   10.82.129.5           10.82.129.114         UDP      68     21157 -> 21143 Len=26
 226179 87.542323140   10.82.129.5           10.82.129.114         UDP      68     21157 -> 21143 Len=26
 234626 91.686158758   10.82.129.5           10.82.129.114         UDP      68     21157 -> 21143 Len=26
 

At this point the ACK comes back from the client.

Internet Protocol Version 4, Src: 10.82.129.114, Dst: 10.82.129.5
User Datagram Protocol, Src Port: 21143, Dst Port: 21157
Data (5 bytes)
0000  64 55 72 f6 80                                    dUr..

This ACK from the client has the same format we used for ours. The packet breaks down as:

  • 64 55: dU - an ACK packet
  • 72 f6 : 16 bit packet number - this was the packet ID we sent in our SYN
  • 80 : packet number offset. Another "ignore this field".

Following this there's a cluster of ACK's, as the client catches up with the multiple SYN packets we sent out. Since we sent out identical SYN requests the corresponding ACK packets are all identical. Since we sent 8 duplicate SYNs then we also see 8 ACK returns. It looks like the ACK is only generated for outstanding SYNs when the game actually leaves the setup screen and the player on the host is on the game level.

 237028 92.782751239   10.82.129.114         10.82.129.5           UDP      60     21143 -> 21157 Len=5
 237029 92.782835138   10.82.129.114         10.82.129.5           UDP      60     21143 -> 21157 Len=5
 237030 92.782857474   10.82.129.114         10.82.129.5           UDP      60     21143 -> 21157 Len=5
 237031 92.782880754   10.82.129.114         10.82.129.5           UDP      60     21143 -> 21157 Len=5
 237032 92.782903744   10.82.129.114         10.82.129.5           UDP      60     21143 -> 21157 Len=5
 237033 92.782980110   10.82.129.114         10.82.129.5           UDP      60     21143 -> 21157 Len=5
 237034 92.783002020   10.82.129.114         10.82.129.5           UDP      60     21143 -> 21157 Len=5
 237035 92.783024780   10.82.129.114         10.82.129.5           UDP      60     21143 -> 21157 Len=5

So, how does the server communicate securely with the game?

Following this then we are in the information exchange phase. The ANET server sends out an initial challenge packet. This is sent as a data packet (type 'dT') which contains a Tserv packet (type 'd(')

This is Activision's "Trivial Challenge Authentication", used to secure password transfer. From the header it's used to perform:

response(challenge) = MD5(TDESn(MD5(password), challenge))

However since we don't use any of the user account stuff in Nitro, then we really don't care: We'll see the server send out the TSERV packet and the game will ACK the packet.

Next we'll look at the tables and how the available games are advertised.

Sunday, 30 July 2017

Interstate 76 File Format Summary

I've got a bunch of stuff on ANET to come up, but it's taking a bit longer than I expected to get a working dummy implementation, so as an interim posting, here is summary of the file formats for I76 and Nitro.

This should break down all the format information so far, with my current thinking on what goes where. It should be easier to follow than tracking through the blog postings.

Saturday, 15 July 2017

Manually Repacking I76/Nitro Levels

Nitro Map Reprocessor

This is something that dropped out of a chat with one of the I76 forum posters about fixing up old maps.

There are a number of I76 & Nitro mission files that have issues, like misplaced objects or are crash prone, and could do with fixing. However without the sources this is a difficult process involving a hex editor and a lot of trial and error.

So, attached to this is a set of Ruby scripts which simplify some of the process, by decompiling missions into a text form (JSON), and then recompiling them back into runnable files.

The code is in This Tarball Here, and is in three scripts; a decompiler (main_ar.rb), a recompiler (main_raar.rb) and a utility library that handles the actual format data(objects.rb). It's a little "my first Ruby program" and needs cleaning up (we could use a smarter approach/library than pack() for a start), but it's functional enough to use.

This script extracts the mission information and this data is converted into JSON, with the decompiler taking care of some obvious conversions, such as to & from float, and also grouping data where we know what it is (position, rotation, etc).

This format is essentially the same described in earlier posts, with a few fixes - notably Erik's comment that the flag field we thought followed the object name is really a float (giving us a 3x3 transform matrix), and it also handles LDEF sections, which support linked arrays of objects and additional fields in the WDEF.

Although we could unpack the zone definitions and game scripting there's little that could be manually edited here, so they're left as "opaque" sections and the data in the JSON is just a byte array. The same is true of a couple of other sections (revision data, etc).

To Run Them

Unpack a File

To unpack a file try ruby -w ./main_ar.rb gumball.rac

Where gumball.rac is a level file. This produces a set of json files, one per section. These files have the original name with the segment name and ".json" attached to the end, so "gumball.racODEF.json" is the object section, "gumball.racWDEF.json" is the world section, etc.
It also produces an index file, ".idx", which contains the list of sections. In this case gumball.rac.idx

Repack a File

To repack use the main_raar script, and either give it an index file, or a list of the unpacked JSON sections, e.g. ruby -w ./main_raar.rb gumball.rac.idx

Which produces the file "tmp.dat" which is a mission file that can be placed in the game directory and run.

So, as an example, running ruby -w ./main_ar.rb gumball.cbt Then the output file gumball.cbtODEF.json contains the following definition (around line 400):
{
    "name": "bddonut1",
    "identifier_tag": 0,
    "rotation": [
      -4.371138828673793e-08,
      0.0,
      1.0,
      0.0,
      1.0,
      0.0,
      -1.0,
      0.0,
      -4.371138828673793e-08
    ],
    "position": [
      995.0,
      0.699999988079071,
      50300.0
    ],
    "trailing": [
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      0
    ],
    "class_id": 2,
    "basicstop": 0
},
This is the Dount Stand: Viewing it from the front and overhead view then originally it's:
We can change the position by editing the position data, so for example:
    "position": [
      995.0,
      0.699999988079071,
      50300.0
    ],
to
  "position": [
      965.0,
      0.699999988079071,
      50300.0
    ],
And if we run the output processor, and copy the file back into the missions directory with
ruby -w ./main_raar.rb gumball.cbt.idx  
cp tmp.dat  ~/.wine32/drive_c/Program\ Files/Activision/I76Nitro/miss8/gumball.cbt
Then when we reload the map we've moved the Dount Stand forward.

Similarly we can copy the entire Donut stand section, and by creating 3 more copies and adding 30 to the X position each time we get....

which is the in car view at the top of this post

Known Bugs

Although it should produce the same output from a given input there are a couple of files where label strings with trailing spaces are cropped during string cleanup (so from " Foo " to " Foo"). It also doesn't do much in the way of syntax validation or checking.

updated 16/7: the file was opened in text mode, not binary. This is fine on Linux, but means that on Windows the read of mission files came up short. Fixed and uploaded V2 of the tarball.

Saturday, 10 June 2017

Anet and I76 Nitro Multiplayer

I76 uses a host server running a program called Anet to co-ordinate online multiplayer.

Activision released the code for the server, compatible with the Nitro riders version of I76, and we can bring this up on a modern machine with a little bit of work.

What's going on behind the scenes

Fundamentally the job of the Anet server is to track lists of available machines. Clients can publish details of games they wish to host, and can also query the Anet server for the details of other machines currently hosting games. They can also request a connection to a client that is hosting a game via the Anet server.

The logical concept in Anet used to track all this is the session: A session is defined (in dp2.c) as "a group of machines" which are associated together: a single online game is a session, as is a chat lobby or any other shared service. Each session has a type number, called a species, which determines what it is (so a Nitro game is "604") and for each session then one of the machines in the session is designated as a host or "master".

So, all client machines connect to the Anet server, and dptab_addPeer() is used to track connected machines. If a machine wants to host a game then it will (can?) send a dp_session_t packet, which describes a session - in this case a game it is running. The Anet server has a list of all the active sessions and the associated host/master and client machines.

Other players which connect to the Anet server can see the available session(s), and can issue join requests. On join requests then the details of the connecting client are passed to the machine hosting the game. At this point the hosting machine and connecting client can then handshake, so the players can communicate directly and run the actual game. Importantly the Anet server is not involved in the actual game data transfer.

There's quite a few details being glossed over there - each machine has it's own list of sessions, which are shared with the Anet server on connecting, and there's a lot of additional message types, group management, scoring and voting logic as well as keepalive pings, host transfers, persistent status (via freeze/thaw), message queuing and html status generation.

Building and Installing on a modern distribution

Build Machine

You'll want a Linux distribution, and my recommendation would be a 32 bit distribution. The modified code should build and run on a 64 bit host, but you will need 32 bit versions of the development libraries anyway. Just using the 32 bit version is simpler.

My Test Setup

Out of the box then (at least for me) Anet doesn't seem to like the case where the server and clients are all sharing a single network segment. The address resolution doesn't cope well and fails to issue the correct connection information to the clients. I believe the SYN handling in dpio_SYN_PACKET_ID is tripping up, but my test gameplaying "machines" are all VirtualBox hosts running Wine, and with bridged ethernet links so things could be hosed on that front.

For me this breaks the simple test case on a private network - at some point I mean to fix this, but for now I just work around it with my network configuration.

My test setup for a basic run is:



Internet |--| Switch |---| Anet Server |
                     |
                     |---| Router |----| Client #1 |
                                           |
                                           |----| VM#1 |
                                           |----| VM#2 |
                                           |-... etc

So the router performs simple DHCP and NAT for the two clients, and the Anet server has a resolvable hostname that the clients can find.

Initially get the source from Dan Kegel's master tarball. This requires a fairly old GCC (2.9 era) to build as is. Something like CentOS or RHEL version 3 would support this and will run in a VM.

Otherwise we need to make a couple of code changes to get this to build, and to reduce the number of warnings

  • Some simple syntax fixes
  • Structure packing conventions have changed: update the the Pack statements to cover the structure rather than individual members
  • print format specifiers updated (consistent use of longs vs ints)
  • thaw buffer is undersized (this trips GCC stack protection)
  • Modify the make files to remove -Werror and specify -m32
  • Fix the library paths for libm and libbsd (/usr/lib/i386-linux-gnu/ on a modern box)
There are still a few mismatches, but that'll do to reduce the noise in the core build.

Install follows the same basic steps as the older version, in terms of setting up an alink user and how to expose the HTML information pages. However we need to

  • Change the machine shell from dash to bash - some setup scripts rely on bash-isms ("dpkg-reconfigure dash" and say no on debian)
  • Switch over the location of information files from the (defunct) activision server to a live server (play.interstate76.com is the one I chose) and use wget rather than a custom perl script.

Actually Building up the Sources

You'll want to grab Dan Kegel's source tarball: http://www.kegel.com/anet/anet-0.10.tar.gz

And there's a patch to this code with my current fixes: tp-anet-changes.patch

Unpack the tarball and patch the source with a line like:


tar -xf anet-0.10.tar.gz
cd anet-0.10
# Permissions fix for a couple of R-only files
chmod -R +rw .
patch -p1 < ../tp-anet-changes.patch
 

Then just type in "make" and things should build and run the autotests. If the build machine doesn't have a good network connection then the build tests will stall: You'll see the tests start with

dptabt1 regression test...+ ./normal/dptabt1
followed by a few more lines and finally
id 2: Sent 3'th variable to h:2; table ndbug, subkey ndbug; len 4
at which point everything will just hang. If that happens then just hit Ctrl-C; you will have to build on a machine with a live network connection to run the tests.

Monday, 1 May 2017

The New AngelScript Raycast Vehicle Model

This is a quick note on how to use the Urho3D vehicle model. It's based around the latest git repository, which has AngelScript interfaces for the Bullet Raycast Vehicle. It's written for the state of the Urho3D repo trunk on Apr 29.

The Vehicle Model

The AngelScript object representing the vehicle is RaycastVehicle . This is a wrapper around the underlying Bullet btRaycastVehicle.

To use this we have the basic workflow of:

  • Create a Node for the Vehicle as a child of the Scene, then under this node
    • Create a StaticModel component for the vehicle model
    • Create a RigidBody component for basic physics properties
    • Create a CollisionShape component for the physics engine to handle collisions
    • Create a RaycastVehicle component. This is the new component and we should also
      • Call Init() on this component to set it up
      • Create a node per wheel, and link the wheels to the vehicle
      • Set up the suspension and tyre properties per wheel
      • Call ResetWheels() when set up
At this point we can use the raycastVehicle.SetSteeringValue() and raycastVehicle.SetEngineForce() to set the direction and driving force through the wheels.

The Suspension and Wheel Parameters

We essentially create a node per wheel, then connect the wheels to the vehicle using AddWheel(). There are a handful of basic properties that need setting, both as part of the initial call, and also as additional parameter settings.

The wheel node has a model attached (we can use the cylinder as last time), however in the case of the Raycast vehicle we don't need to provide a collision box or any other information with this model- it's decorative with the vehicle model updating position according to the associated wheel & suspension parameters we supply (for example, if we have scale the wheel model down then the car will simply appear to float above the ground).

The parameters that we need to determine are:

  • Acceleration and Braking
    • Car Mass: Typically in Kg. Applied to the Chassis.
    • Braking Force: Higher values for sharper braking, Supplied to SetBrake().
    • Engine : Higher values for more force. Supplied to SetEngineForce().
  • Suspension Details
    • Suspension Rest Length: Maximum Length of the (unloaded) suspension (in M)
    • Suspension Travel: Maximum Suspension travel (in cm).
    • Suspension Stiffness: Higher values for a stiffer suspension.
    • Suspension Damping during Relaxation: Lower values for longer ringing suspension.
    • Suspension Damping during Compression: Lower values for longer ringing suspension.
  • Wheel Properties
    • Wheel Radius:
    • Wheel Width:
    • Wheel Friction Slip: Tire friction to ground. Limits the maximum impulse pushed through the wheel.
    • Roll Influence: A side impulse to the chassis. High values will cause the car to flip on turning.
Most of these are self explanatory, and are either supplied to the wheel constructor or as separate parameters. Also there's a mix of units and ranges here (so Rest length of 0.3M but the travel limit in cm would be 300 for full travel). We also supply some basic parameters to determine how the wheel is positioned on the car (is it a front wheel, the direction of travel and the axle axis). A simple setup looks something like this:
    raycastVehicle.AddWheel(wheelNode, wheelDirection, wheelAxle, suspensionRestLength, wheelRadius, isFrontWheel);
    raycastVehicle.SetWheelSuspensionStiffness(id, suspensionStiffness);
    raycastVehicle.SetWheelDampingRelaxation(id, suspensionDamping);
    raycastVehicle.SetWheelDampingCompression(id, suspensionCompression);
    raycastVehicle.SetWheelFrictionSlip(id, wheelFriction);
    raycastVehicle.SetWheelRollInfluence(id, rollInfluence);
    raycastVehicle.SetMaxSuspensionTravel(id, suspensionTravel);

Limits

As well as obvious missing features in the model (gearing, upper speed limits, wheel parameters, terrain traction, etc) there's a couple of limitations here - the model does not seem to support sideslip under high speed turning: The tyres either have traction or the car rolls, but nothing in between (but this may be a reflection of the settings we have). The car also needs tweaks to the collision box, and also the speed reading is almost certainly off.

The sideslip we can probably emulate using side forces and playing with tyre friction and checking the speed and turning circle, but that's more complex than this example. Further testing required.

Sunday, 30 April 2017

Adding Simple Physics

Next up we can add some simple physics handling to the basic map and model loads under Urho3D, and this lets us handle simple objects and push cars round the map.

Introducing Physics

Urho uses the Bullet Physics library to provide a physics engine that supports collision detection and rigid body simulation.

Urho embeds a complete copy of Bullet under the Source/ThirdParty/Bullet/ directory, and there is a subset of the functionality exposed through AngelScript. However the AngelScript API doesn't support the complete feature set of Bullet that the C++ code can access, which will get annoying (as we see later).

Adding A Simple Colliding Entity

So initially we have to:

Enable Physics

This is a single line in the AngelScript

 thescene.CreateComponent("PhysicsWorld");

Turn on Collisions With the Scene

This is fairly simple we have the Node that the terrain Component was created in, and we want to tell the Physics engine that objects should collide with this terrain. To create this behaviour the engine wants us to do the following under the Terrain node:

  • Create a Rigid Body: A solid body, which doesn't deform, and has basic physics properties (such as mass and friction).
  • Add a Collision Shape: To support collisions, marked as Terrain in this case.

So, first we make a default RigidBody associated with the Terrain Node as a fixed object (the default, for a zero mass item).

RigidBody@ body = terrainNode.CreateComponent("RigidBody");
  body.collisionLayer = 2;
  

The collisionLayer parameter provides a bitmask that the Bullet engine uses to determine if objects should collide. The tutorials use the mask value of 2 throughout for terrain, and we'll keep that convention here.

There are actually two parameters here: collisionLayer and collisionMask. This is a detail we won't get into since it's not something that affects this demo, but when Objects 1 & 2 intersect then we evaluate:

  • the object 1 mask and object 2 collision layer
  • the object 2 mask and object 1 collision layer
If either of these operations are non-zero then we have a collision, otherwise they pass through each other.

Next we associate a collision shape - this is created alongside the static Rigid Body under the terrain node. The shape of this collider is marked as Terrain.

CollisionShape@ shape = terrainNode.CreateComponent("CollisionShape");
  shape.SetTerrain();
And that's it.

Add Shadows

Not necessary, but the relation of objects in the world is much clearer with shadows - otherwise entities have a tendency to look painted over, rather than on, the terrain. So we have to enable light shadows when we declare our light:

light.castShadows = true;
And when we create an object we'll also set the model castShadows property to true.

And The Object

We can add a simple colliding sphere (we'll call it a Marble), by creating a node with a Static Model based on "Sphere.mdl", and then adding the RigidBody and CollisionShape declaration.

For this example we use SetSphere() to set the collision shape to match the (spherical) model, and we give the RigidBody a mass and rolling friction and watch it go. Here's a complete working class, based on the code in 11_Physics.as:

class Marble {
Node@ marbleNode;
StaticModel@ mdlObj;
RigidBody@ body;
CollisionShape@ shape;

bool doPrint = false;

  Marble(Scene@ scene, float sz, float speed, float mass) {
    Create(scene, sz, speed, mass);
  }

  void Create(Scene@ scene, float sz, float speed, float mass) {
    marbleNode = scene.CreateChild();
    marbleNode.position = camera.cameraNode.position;
    marbleNode.rotation = camera.cameraNode.rotation;
    marbleNode.SetScale(sz);

    mdlObj = marbleNode.CreateComponent("StaticModel");
    mdlObj.model = cache.GetResource("Model", "Models/Sphere.mdl");
    mdlObj.material = cache.GetResource("Material", "Materials/Terrain.xml");
    mdlObj.castShadows = true;

    body = marbleNode.CreateComponent("RigidBody");
    body.mass = mass;
    body.rollingFriction = 0.9f;

    shape = marbleNode.CreateComponent("CollisionShape");
    shape.SetSphere(1.0f);

    body.linearVelocity = camera.cameraNode.rotation * Vector3(0.0f, 0.0f, 1.0f) * speed;

    SubscribeToEvent(marbleNode, "NodeCollision", "HandleNodeCollision");
  }

  void HandleNodeCollision(StringHash eventType, VariantMap& eventData)  {
    if (doPrint) {
    RigidBody@ otherBody = eventData["OtherBody"].GetPtr();
    RigidBody@ hitBody = eventData["Body"].GetPtr();
      Print("Ding "+ hitBody.id +" against "+ otherBody.id + " !");
    }
  }
}
This provides the basic object and also an optional debugging report on collisions.

We can play around with this code and check the behaviour of the physics simulation. There are some problems when we fire very small and very fast marbles which have a tendency to fall through the terrain or when we combine very heavy and very light objects - this is expected and the Bullet and Urho docs mention the need to handle these cases carefully.

Adding A Car

There's an example of a simple vehicle provided by the Urho file ./bin/Data/Scripts/19_VehicleDemo.as. It's worth taking a moment to look at how this works.

The car consists of a main hull body and four wheels. The main body is a RigidBody/CollisionShape and only has mass and air resistance, and all the power and steering is done by manipulating the wheels directly.

Each wheel is a cylinder model with an associated RigidBody and a spherical CollisionShape connected to the main body via a Hinge Constraint.

The hinge constraint is used to link the body to the wheel, and the wheel spins around this constraint. The hinge constraint limits are set from -180 to +180 degrees which allow the wheel to spin completely.

The constraint connecting the wheel to the body is also used for steering: by rotating the angle of the connection between the body and the wheel, the code deflects the front two wheels in response to steering commands.

Although the hinge is restricting motion around other axis there are some damped-spring style reactions to forces which can be seen at higher mass main body values, although I suspect this may be a bug involving the mass ratio of a light wheel to a heavy body. It's actually almost a usable hack for suspension style behaviour.

By default the reference code subclasses ScriptObject, which allows it to serialise the object to a file, and to handle control events directly. We can actually fire the event handler explicitly and break this connection if we want to simplify the implementation and don't care about serialising.

When accelerating the ApplyTorque() function is used to change the angular velocity of the wheel. In addition there's a directional component introduced to the front wheel (when steering is turned). Actual acceleration results from the interaction between the rotating wheels and terrain.

Lastly there's a downforce component which keeps the car connected to the terrain, by applying a downward force based on the car velocity.

Updating The Car

Although the code works with a basic setup, it's difficult to control and tune, and doesn't work when exposed to the kind of terrains we see in I76.

Firstly we need lower the downforce component. This works well to glue the car to the rolling terrain, but it tends to let the car drive down cliff faces on the I76 maps.

Next we modify the loaded model to be one of the one piece I76 car geometries, and tweak the wheel positions and collision bounds accordingly. We also move the body mass up to a more realistic (real world) value of around 1500 (assuming 1 unit = 1kg).

To simplify the control of acceleration we replace the "Torque" model of the demo with a simple ApplyImpulse() version. This doesn't rely on the torque for acceleration, but pushing the wheels directly. We put a simple check in to ensure that the wheel is in collision with the terrain before applying force through it, but that's all. This gives a simpler more controllable vehicle, although it's a bit like driving on glass.

How This Needs To Improve

Right now there's no suspension, or notion of vehicle simulation such as engine or tyre behaviour, weight transfer, etc. which would make the driving behaviour more like the original game.
We could add some of this in AngelScript, but that has an obvious overhead in terms of processing load and even a simplified model is a very complex problem. So are there any other options?

The good news is that the bullet physics library provides a complete abstraction - the btRayCastVehicle model. This supports a single solid body type which behaves like a vehicle, has a tunable suspension and wheel model, and takes care of all the details for us. There's a great example of it in action in this GitHub repository by the Urho user Lumak.

However the downside of the Bullet vehicle is that this is currently a C++ only thing - it's not exposed to the AngelScript interfaces. This is an annoyance, since using AngelScript to keep things cross platform is a definite advantage.

So we've hit a break where I need to figure out where to go next - Is there a plausible I76-like car behaviour without too much scripting code using a few approximations, or is native code for the car the way to go, or some mix and custom scripting interfaces down to the library? At this point experimentation will be required though, so that'll do for now...

Update - having just pulled the latest from the Urho3D Repo it turns out that RayCast vehicle integration & scripting hooks have been merged into the main release as of 20-something hours ago. So this decision is probably simpler than I thought.

Sunday, 23 April 2017

Urho I76 Model Viewer

Loading Up Models

The next thing to do under Urho3D is to load up the models for I76 assets.

The Hard Way

The hard way to do this is to manually generate an Urho VertexElement array and attach the model vertex and normal data, linking a VertexBuffer and IndexBuffer through a Geometry element.

This approach works, and there's an example of it in the sample application 34_DynamicGeometry.as, but it's fiddly and my implementation is error prone. So let's not do that.

The Easier Way

The simplest way to do this is to use the AssetImporter tool from the Urho3D distribution. This can take an OBJ format file, which we can generate easily, and produce output files that Urho3D can load directly. It adds an extra step, but simplifies the process.

So we generate the OBJ file format files as we do in the existing C++ code, and then use the fileSystem.SystemRun() call in AngelScript to shell out to the AssetImporter binary. There's code in the default Urho EditorImport.as application that we can use more or less directly for a prototype, that handles assembling the command line and also tacks on the .exe extension for windows. We'll replace this at some point, but for now let's just use it as is for the demo.

Otherwise there really isn't much clever here - we reimplement the ZFS unpacker in Angelscript to extract the resources, then we can view either individual models from GEO files, or complete vehicle assemblies from the VDF file. Extracting from the ZFS takes several minutes, and so the app should be run windowed, and the terminal will report the progress of the unpack. We could implement a background loader, but it's not worth the effort for a run-once process like this.

The only other annoyance is that the floating point output from AngelScript threw up errors with the AssetImporter by default - small values will be represented by scientific format (e.g. 3.3021e-05) and the importer doesn't like some of these, and there didn't seem to be a way to format the output more precisely. So we spot small values and treat these as exactly zero when we write them out.

Here's a tarball attached with:

  • AssetLoader: assetLoad.as - Load a ZFS file, and unpack the ZFS itself, embedded PAK files and do the basic conversion for GEO to objects and VQM to texture images.
  • ObjLoader: ObjLoad.as - View individual object files. Does on the fly conversion.
  • CarLoad: CarLoad.as - Load VDF files - this has some issues (the tendency to mess up interior geometry fragments, and some normals are iffy), but it's functional enough to pull out some basic geometry for the composite models.

The conversion process just applies a default image file for testing at present (whichever image you copy into Textures/test.png), and requires a defaultmat.xml Material file. It's lacking basic error checking, but as a proof of concept it holds together.

Also this will not work on the Nitro ZFS, since AngelScript doesn't know how to decompress the LZO compressed files without the platform specific libraries.

Otherwise, it mostly sorta works...

AngelScript: Some Thoughts

Having used AngelScript for a few days then the thoughts that keep cropping up are

The Good

  • Fast Prototyping.
  • Straightforward UI & 3D implementation.
  • Feature rich primitives.
  • Syntax: maps neatly to C++ versions.

The Bad

  • No debugger: This is a real PITA. The code has wound up with debug Print calls scattered around.
  • Silent Failures: Some setup errors leave things silently non functional, rather than producing more obvious error reports.
  • Slow: Although the heavy lifting is mostly done by the core engine having scripts generate things like texture images and bitmap resources can result in major slowdowns.

The Ugly

  • Ownership: The Urho Angelscript version uses reference counting for memory management, but without a debugger it's easy to lose track of what references are valid at times and leak resources (such as open file handles).
  • Not Quite Vanilla AngelScript: Enough custom types that the default AS docs don't always seem to apply directly or help understand Urho's use of the language.
  • Documentation: It's there, but not always useful, and the examples take some parsing to understand: it could be better.
  • No Printing Format control? I couldn't figure out how to persuade floats to output in a specific format, hence the low value rounding utility.