Jump to content

The "Getting Bolo to work over the internet" challenge


Recommended Posts

  • 68kMLA Supporter

Yeah, there were several times where we learned something new and it was a "oh, this complicates things" moment.

 

On 2/11/2021 at 8:17 AM, anthon said:

I thought it might not be too much effort

 

Programmer's famous last words.

Link to post
Share on other sites
  • Replies 170
  • Created
  • Last Reply

Top Posters In This Topic

  • 68kMLA Supporter

Use the invite link to join the Astrospark server: https://discord.gg/Wb62CeHQJq

 

On the very left side, there's a column of servers you're a member of. Astrospark is the yellow spark icon. Make sure that is selected, and then in just right of that (still on the left), there is a list of channels for that server. That's where you find the #bolo channel.

Link to post
Share on other sites
  • 68kMLA Supporter

Thanks to @LaPorta, @superjer2000, and @olePigeon for participating in testing the (probably) first internet bolo game in ~20 years. We had up to 5 players at a time, and it seemed pretty solid. There was some occasional lag, but it mostly cleared up. The full ring latency for the 5 player game was over 500 ms. Six players is probably about the max I expect to work well. But it wouldn't hurt to try more to find out. olePigeon streamed the game on twitch: https://www.twitch.tv/wreckcentersw/video/949436681

Link to post
Share on other sites
  • 68kMLA Supporter

To my recollection, 6 was about the max that ever really worked over the net, anyhow. The theoretical maximum of 18 was likely only doable on a speedy local ethernet network.

 

This also gives me a great excuse to make some new maps...

Link to post
Share on other sites
  • 68kMLA Supporter

That was awesome!  The network worked incredibly well - the minimal lag was better than I remembered playing locally with LocalTalk. 

 

So many projects like this get announced but don't end up being launched so incredible to see this happen.  Thanks @anthon and @techbeach - what a polished solution!

 

How does the server work longer-term? Will it always be running or is it something that will be active just when a game is planned?  I'm looking forward to the next BoloFest!

Link to post
Share on other sites
  • 68kMLA Supporter

Thanks superjer! As they say, the second ninety percent of a project is the hard part.

 

The server is up and running long-term. People can use it any time. Let me know if it's ever not working.

Link to post
Share on other sites

I'm uploading a YouTube video of most of the session.  I forgot to hit record and didn't notice until about 20 minutes in.  (Video isn't available as of this posting, still uploading.  It's big.)

 

https://www.youtube.com/watch?v=Arp3G7hnso4

 

Also, back in the day when I used to play in Bolo tournaments, I made a bunch of maps.  I went by Chuckles back then.  They're small, fast maps, intended for quick games.

 

Chuckfest.sit.hqx

Edited by olePigeon
Link to post
Share on other sites
19 hours ago, anthon said:

I'll make an announcement thread soon. Somewhere that more people will see it. (The Lounge?) It could subsequently be used for organizing games.

 

Yes, please!  We need to hold tournaments. :D  68k MLA versus LEM versus ThinkClassic versus MacNN? :cool:

Link to post
Share on other sites
  • 68kMLA Supporter

The server is running and you can play now! You need to download Bolo 0.99.8 (this is a custom modified version for this project). There are instructions in the post linked below. An added note, if you have more than one version of Bolo present, you may need to edit the Internet Bolo Buddy preferences to launch "a particular copy" and choose the 0.99.8 version.

 

 

 

Link to post
Share on other sites
  • 68kMLA Supporter
Posted (edited)

There was a lot of work involved and it's difficult to describe succinctly in a way that would make sense to someone who hasn't just spent a month buried in the details of the Bolo network protocol, internet protocol routing, UDP, 68k assembly, etc. Some of the main points:

  • Understand the Bolo network protocol enough to re-write packets to proxy them.
  • Design a new proxy communication scheme that is compatible with the original Bolo protocol but tricks it into using the proxy as an intermediary.
  • Write the proxy server
  • Figure out a way to trick the Bolo application into helping us do NAT traversal
  • Make the needed modifications to the Bolo machine code to do NAT traversal
  • Fix bugs in the Bolo application

Here's a general overview of each category. I have omitted various details for brevity or because I didn't think of them while writing this down.

 

Bolo Network Protocol

 

This involved both network packet inspection and analyzing disassembly of the Bolo application.

 

The protocol has 11 different types of packets. Two of them are for trackers to request and receive game info. Five of them deal with joining a game and negotiating the ring order. Two of them manage game state (each action or event that occurs in-game) and three of them are involved in error recovery.

 

There end up being 6 packets that need to be parsed and modified to change the embedded IP addresses. Of those, all but one are fixed-format packets where we can just write the IP address at a fixed offset and be good. The last packet is actually where the 99% of the game occurs, the game state packet. It has a variable structure. While it only occasionally contains IP addresses, when it does it is when the ring order is changing and without it the game would break any time a player left the game.

 

The game state packet is passed around the communication ring. It contains a block of state for each player. A player receives the packet, updates their game with everyone else's data, removes their old state block from the packet, adds a new block with anything new from their perspective, and passes the packet to the next player.

 

Each player state block contains a list of "opcodes". The opcode is the type of action or event. Like a tank moving, firing, refueling, etc. Each opcode has its own data format. So in order to correctly parse the protocol, we needed to at least know how to calculate the length of each opcode. Otherwise we wouldn't be able to find the specific opcode ("player disconnect") that we need to modify to make the game work. Most of the opcodes have a fixed format, but there are several that have a variable length that depends on the contents. A lot of effort went into understanding the disassembly here to know how to parse these opcodes.

 

Player state blocks inside game state packets include a CRC checksum. This was probably a basic anti-cheat mechanism. So in order to change these packets, it was required to reverse engineer the CRC algorithm so the proxy server could stamp the modified packet with the correct CRC.

 

Proxy Server

 

When a player starts a game, they need to notify the proxy server so that it can start listening for game packets. This is done by configuring the Bolo tracker setting to notify the proxy server. When the proxy server gets the game info packet for a game id it doesn't recognize, it knows a new game is starting.

 

Each player is allocated a dedicated port number. This is how the proxy server tells different players apart. When the second player joins the game, they talk to the proxy on the port assigned to the first player. The proxy forwards packets between the two players, replacing any IP addresses inside with its own IP address and the correct port number for the player who sent the packet.

 

The proxy server keeps track of all the games in progress. It listens to the game info packets on the tracker interface, keeps the game info up to date, and sends it to Internet Bolo Buddy when requested. It snoops the game packets and extracts the player names so that it can list who is playing in each game.

 

It maintains a NAT mapping with players on the tracker port by pinging the game regularly for game info packets. This allows it to send NAT traversal assist packets when the ring topology changes (players joining or leaving).

 

NAT Traversal

 

In order to eliminate the need for people to configure port forwarding, we wanted to implement NAT traversal. When a computer ["server"] wants to talk to another that is behind NAT ["client"], it needs the client computer to talk to it first. That causes the router doing the NAT to recognize that the client is talking to the server, and that when incoming packets arrive from the server, they may be forwarded to the client. If the server tried to talk to the client first, then the router would just ignore it.

 

This is necessary because in the Bolo protocol, players can talk to other players without first having been spoken to.

 

Bolo already has a packet that results in the game communicating to another host. It is used during ring latency negotiation, where it essentially pings different players to find the fastest route. Version 0.99.7 will only accept this ping instruction from its upstream neighbor, but our modification makes it accept the ping from anyone. This allows the server to pretend to be a player that is already speaking with the game send a packet that instructs the game to send a fake ping to the new player who the game hasn't spoken to before. That establishes the mapping on the router so that future packets from the new player will be routed successfully.

 

Bug Fix

 

Bolo 0.99.7 contains an assumption that there would never be a Time Manager with version greater than 3. So Bolo DNS resolution doesn't work when used on MacOS 9, which has Time Manager v4. (This caused Bolo not to work at all on MacOS 9 because it has to do a DNS lookup even on an IP address to know that it wasn't a domain name.) A binary modification was made in Bolo 0.99.8 to fix this bug so that Bolo now works in MacOS 9.

 

Myths Busted

 

It would not have been a simple matter for the game to use the network header address instead of one embedded inside a packet. The address communicated are third party addresses that aren't in the headers. The protocol was designed for local serial ring networks and then later ported directly to UDP. If we waited for Stuart to implement a new protocol from the ground up without those limitations, we probably would have never gotten internet Bolo in the first place.

 

Another thought was that Bolo servers used to exist which mediated NAT traffic. We know enough about how Bolo works now to say with confidence that no such servers existed. It wouldn't have been possible to implement using the Bolo 0.99.7 network protocol, which we have studied at the assembly level.

 

What Happens Next

 

I'll get the code pushed up to GitHub. TechBeach is preparing the documentation of the binary modifications. This includes documentation of what changed from 0.99.7 to 0.99.7bv (the Carl Osterwald mod) and 0.99.7bv to 0.99.8. Soon we will put a getting started guide on the website. And create a thread in the Lounge to announce the server and for coordinating games.

Edited by anthon
Link to post
Share on other sites
  • 68kMLA Supporter

That's a really good writeup, thanks for the info!

 

On 3/16/2021 at 5:30 AM, anthon said:

This allows the server to pretend to be a player that is already speaking with the game send a packet that instructs the game to send a fake ping to the new player who the game hasn't spoken to before. That establishes the mapping on the router so that future packets from the new player will be routed successfully.

 

That's a clever way of doing that.  Nice.

 

On 3/16/2021 at 5:30 AM, anthon said:

It would not have been a simple matter for the game to use the network header address instead of one embedded inside a packet

 

Ah, I never got far enough into the reverse-engineering to work that one out.

 

On 3/16/2021 at 5:30 AM, anthon said:

The protocol was designed for local serial ring networks and then later ported directly to UDP.

 

The protocol was originally designed for Econet, IIRC, which is extremely LocalTalk-like in terms of model.  So it's still a bit eccentric, but not anywhere near as odd as it appears over UDP.

Link to post
Share on other sites
  • 68kMLA Supporter
Posted (edited)

Since I have a complete lack of knowledge of this stuff, what does one use to build all this? I must need something installed to enable Terminal to do this.

 

Edit: Ok cool, so I downloaded Go and built the thing. I created config.txt, but now for some reason it can't open the config.txt file that I created in the same directory as the built command line application.

Edited by LaPorta
Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...