The "Getting Bolo to work over the internet" challenge

[superjer2000]

To make sure this is the case, I just re-ran this test in a more controlled environment.  That environment was: two Basilisk II instances each using ethernet-over-UDP (so I didn't have to muck about with actual hardware).  Bolo running on both.  Machine 1 IP address 10.0.0.1, machine 2 10.0.0.2.  Wireshark running on the host of machine 1.

Here is one of the packets involved in the connection process.  Please disregard the extra framing: Wireshark doesn't know how to dissect the BII Ethernet Tunneling gubbins.  The Bolo packet payload starts, appropriately, with the word 'Bolo' (outlined in both the hex and the ASCII).

View attachment 40556

Below this, at $0060, you can see $0a $00 $00 $01 (outlined in green), which in decimal is 10 0 0 1.  This is the local IP address of the machine embedded in the payload of the packet and is why NAT breaks this game.

That's awesome cheesestraws!!  

I wonder exactly how that is used?  I.e. When bolo wants to communicate with another machine I guess it takes the IP address from the packet instead of whatever header or other network information would typically be used?  If a bolo packet does make it back to another computer, I wonder if the client looks at the packet and confirms the IP address in the packet against the client as well?

For clarity: Would it be feasible to do a find/replace on these packets?  So in your case above, when System 1 sends out a bolo packet that has 10.0.0.1 in it, that gets replaced with the external IP.  So now System 2 sees the correct external IP and can get it's packets back to System 1's internal network.  But that packet still has the external IP address in it and so even though the router 'routes' it to Server 1 on the internal network, will Bolo ignore it because the IP in the packet (external) doesn't match System 1's internal IP?

That would require monitoring of packets going out and coming back.  

I also have no clue if it's as easy as just doing a find replace, or are there checksums that need to be recalculated as well?  Time to do a little bit more reading on this stuff.  I have very fond memories of all night Bolo LAN parties and would love to play again.

 

[cheesestraws]

Would it be feasible to do a find/replace on these packets?

This is essentially what an ALG (Application Layer Gateway) does.  SIP, which is a popular protocol for VoIP, does something kind of similar, where the IP address of the SIP speaker is embedded into the data stream.  So a lot of especially home/office routers have SIP ALGs, which essentially replace the IP addresses on the way through.  My intuition is that the same approach would work here, though note that you'd have to do the same in each direction, so you'd have to replace the IP with the internal IP on the way in as well, because...

will Bolo ignore it because the IP in the packet (external) doesn't match System 1's internal IP?

Probably!  So you need to translate in both directions, same as you would in the packet header.

That said, that is only an intuition.  I haven't tried it.  But—Bolo does the same thing in AppleTalk, it embeds the DDP address in the payload.  bbraun's AppleTalk VPN thingy does precisely this, rewrites the DDP addresses in the packet payload to match the header when it does its kind of AppleTalk NAT trick, and apparently that works well enough to play Bolo over his VPN thing.  So my bet is that that strategy would work with IP as well.

 

[superjer2000]

There is a Linux program called Mallory that at first glance seems to have be ability to modify UDP packets on the fly with a find replace type function. 
 

I haven’t looked too far into this but it might be an option to route Mac traffic to a Pi and change the relevant parts of the UDP packets. 
 

Is this along the lines of what @anthon was considering?

 

[olePigeon]

@cheesestraws  Would everyone need to do this?  Or only the person hosting?

 

[cheesestraws]

Would everyone need to do this?  Or only the person hosting?

There's no such thing as "hosting", because Bolo is weird.  There's no client-server here.  Instead:

The players in a game are arranged in a logical ring, around which messages are passed.  Kind of like a bus network; a message is received from one neighbour, you act on it, and forward it to your next neighbour.  I assume that if your own packet returns to you you just drop it on the floor.

When you join a game by giving the IP address of a node playing, every other player in the ring measures their latency to you and you get inserted into the ring somewhere where you will have the lowest latency to your neighbours.  So every player node needs to be able to talk to any other player node, because as the ring grows and shrinks as people join, you might find yourself needing to communicate with totally different computers.

So everyone needs to be emitting packets onto the Internet with the right IP addresses in, which means that everyone needs to be doing the packet rewriting.  Alternatively, you could have a central proxy that would create a "pretend" ring and act like a switching point in the overlay network.

If you think this sounds mad, it sounds mad to me too.

 

[Scott Squires]

Sorry, I bounced my idea off cheesestraws for sanity checking but didn't post the details here. My idea was for all the Bolo players to join a virtual private network. Then there would be no NAT and Bolo would work. The Raspberry Pi would act as the VPN gateway (since a retro Mac has no hope of running VPN software).

But based on some information from the nubolo develolper on the rec.games.bolo usenet group, it sounded like there might be functionality for a tracker to tell Bolo what IP address it should use in its packets. Since that would be an easy development task compared to a server, and only require operation of a central tracker not a network appliance on everyone's premises, I did some investigating into that. It turns out that nubolo and Bolo use entirely different tracker protocols. Apparently the bolo.usu.edu tracker server supported both.

During that investigation, I ended up with enough info that I thought it might not be too much effort to implement a proxy server. So currently a friend and I are working on reversing the network protocol. Everyone would connect to the proxy server as the "host" machine, and it would forward all the packets to the individual players, re-writing ip addresses as needed.

cheesestraws helped me devise a network setup for testing with Basilisk II. But since Basilisk II's networking is so kludgy, I wasn't 100% sure I wouldn't spend days getting it working, so instead I set up a physical test environment. Turns out I don't actually own a network hub anymore (now I have ordered one from ebay), so in order to snoop traffic I had to set up shop outside my network closet with all the bolo computers connected directly to my core switch. That way I can use the switch's monitoring port to capture traffic.

Bolo protocol reversing in progress:

 

[LaPorta]

Wow, this is a nice development. Many thanks to you guys for this!

 

[olePigeon]

Yaaay!

 

[cheesestraws]

Very nice progress!

 

[olePigeon]

@cheesestraws  Oops.  I was aware of the token ring style networking, I meant that if only one person needed to do it for when everyone else puts in an IP to join.  But if everyone needs a forward-facing IP, then that answers my question.

 

[cheesestraws]

But if everyone needs a forward-facing IP, then that answers my question.

Yup - as I understand it, the place you get put in the ring has basically nothing to do with what IP you join to, if that makes sense.  So all the other players need to know your public IP, and you need to know theirs, because you could end up with any of them as neighbours.

 

[superjer2000]

What kills me is that if the source code for bolo was public it would likely be pretty trivial to add a text box to enter your external IP address and embed that into the packets being sent rather than your macs internal IP address.  The code would then also look for that external IP address rather than the internal one on received packets. 
 

the nubolo guys said they built there game on The bolo object code. I’m not sure exactly what they mean by that - ie they disassembled it or they had access to the source code?

 

[cheesestraws]

object code. I’m not sure exactly what they mean by that - ie they disassembled it or they had access to the source code?

'object code' usually means the output of compilation (e.g. the traditional .o suffix on binary files compiled from C).  So I'm guessing they mean the binaries here.

 

[superjer2000]

'object code' usually means the output of compilation (e.g. the traditional .o suffix on binary files compiled from C).  So I'm guessing they mean the binaries here.

Agreed but how would that help them to port the game to a totally different processor?

 

[Scott Squires]

Nubolo claims they examined the original Bolo code to figure out game mechanics: tank acceleration, firing rate, pillbox firing rate based on health, various timers, etc. In Winbolo, those various mechanics were just made-up and did not match Bolo.

 

[LaPorta]

Correct me if I am wrong (I am not a programmer of any sort), but it seems like from what you are saying is that there isn't really a reliable way to centrally do this. Each and every machine has to send out modified packets?

 

[Scott Squires]

The problem with doing "search and replace" on the UDP packets is that the 4 bytes that represent a player's IP address can also completely legitimately be game data. Blindly replacing that data will cause infrequent but infuriating bugs.

The simplest thing to deal with would have been packet types that contain data in a fixed-position format. Unfortunately it's rather more annoying than that, but we're making good progress on it. Aside from bus order negotiation and exchanging game map and status data upon joining, all the events in the game are transmitted using Bolo packet id 2. That packet is a sequence of "opcodes" and associated data for that opcode: opcode, data, opcode, data, opcode, data, [...]. For any number or combination of opcodes. Each opcode's data has a different structure. One of those opcodes is the list of player IP addresses. So in order to correctly replace the IP addresses, the proxy has to know, at minimum, the length of the data for each opcode in the game. For example, if the address list is the 4th opcode in the packet, we have to at least know the length of the data associated with the first 3 opcodes, otherwise we can't even distinguish the "player IP address list" opcode from data.

 

[cheesestraws]

Correct me if I am wrong (I am not a programmer of any sort), but it seems like from what you are saying is that there isn't really a reliable way to centrally do this. Each and every machine has to send out modified packets?

Well, you can centrally proxy it.  In that case, the proxy stands in for every player, and replaces addresses in the packets as they come in and passes them back out again; so essentially each player sees itself in a ring with the proxy on both sides of it, and the central proxy relays and rewrites as necessary.  It's a bit of a mindbender, but you can kind of think of it as turning the topology where every player has a proxy 'inside-out', so that the ring exists largely within the proxy.  I think this is what @anthon is up to, roughly

 

[Scott Squires]

Right. The proxy essentially converts the protocol into a client-server model. Every Bolo instance sends and receives its packets to/from the proxy. This will have repercussions in the form of increased latency. But it has a side-benefit of possibly not requiring port forwarding.

Here's a diagram comparing normal Bolo networking and Bolo with a proxy:

 

[LaPorta]

Where would said proxy live?