BatteryShark

Reversing an ASIC-Assisted Arcade Game

A decade-spanning arcade reverse-engineering story involving Linux, protection hardware, timing, and one very stubborn game.

February 1, 2023 arcade, reverse-engineering, game-hacking, emulation, low-level-systems

Background

This one has been in the works for over a decade.

Had I known the amount of work that this game would require, I’m not sure that I would have ever started working on it.

However, the skills both used and learned while putting this together have been one of the most valuable reversing projects I have ever worked on.

To best illustrate everything that happened, I should probably start from the beginning…

First Attempt: ~2012

While working on other projects, I stumbled across a hard disk image for “Percussion Master” - an arcade game from IGS circa 2004 that was one of those knockoffs of drum-based music games such as Taiko or Drummania.

I decided to dig into the image and see what it was about.

HDD Image

The HDD image itself had a few partitions:

It was originally rumored to not be protected, although there was what appeared to be a kernel module alongside the game data.

I decided to dig into the game executable to see more.

The Game

The game executable was unstripped and based on SDL - a set of libraries that act as a framework to develop games and other ‘media-based’ content.

Normally, stuff like this doesn’t have function names available, so it was even more interesting to me as I could see all of the various functions and sort of piece together what they did:

I also noticed some relatively rudimentary “security” functions and figured that might be all that the game was doing from a protection standpoint…

Testing the Game

At this point, running the game in a VM, I was greeted with this:

At the time, I didn’t have much experience beyond trying to patch over functions, but I could skip the pccard checks and a few others. However, I spent a ton of time staring at this screen:

I suspected maybe the pccard was to blame, so I looked at the unstripped .o kernel module:

But I couldn’t really make heads or tails of what it was doing other than opening a character device at /dev/pccard0, using seek as a read/write counter, and reading/writing a static amount of memory to/from the game and card every frame.

Also, this game was silent because it was based on OSS - and the developers statically compiled the SDL Mixer library into the game (gg) - so replacing it was an impossibility at this time. Eventually, I discovered Alsa-OSS and got the default IGS startup sound, but was still stuck at that black screen.

Note From the Future: Later on, I would start using libpulsedsp and eventually osspd which is easier to use.

Giving up… for now

After a while, interest fizzled out - I was busy with grad school and had plenty of other projects to work on. I knew that it needed to connect with /dev/pccard0 and wasn’t too sure what it was using that for.

Second Attempt: ~2016

Game Engine Revisited

After working on several other games and getting used to figuring out custom IO and hotpatching functions, I decided to take another crack at the game.

This time, lending from my previous work with JVS-based games, I decided to try and emulate the pccard0 device based on understanding what the game was asking for. I had gotten away from the rudimentary patching shit out so things kind of work methodology that I think a lot of us start out with while reversing software and wanted to make something more true to the real thing.

Instead of patching out functions like the protection function above, I started writing code to handle the requests:

I even started a repository with my IO code and separated the logic from the what I now understood was an ASIC called the IGS027A or A27 for short.

I also figured out how to get into the test menu:

From here, I could map out the I/O controls and figure out how to hijack the game IO functions to get keybaord controls working.

I could now get into the game menus by bypassing the attract opening, but I figured the A27 was being used for playing the song and music, as it was then stuck on the song gameplay screen with no sound playing.

This was an era where I’d document things on video so…

Video:

VIDEO

I figured it had something to do with the SongProcess and SongProcessWrite functions, that call out to the A27 and take input:

From here, things were looking grim - I figured that the A27 had special data stored on it that the game used, or that it was telling the game in some special way to start the song. The A27 also used a read/write buffer and expected various “system_mode” values to dictate what kind of request was being sent:

Alongside what responses were supported:

And while some of these requests/responses made sense, others were far more vague:

Giving Up - again…

I figured that I needed someone with a real machine and A27 to capture request/response pairs, but this game was already super obscure and availability was so limited, that it was unlikely. I then shelved all my work again.

Third time’s a charm? - 2020

Getting the Hardware

So… during the COVID-19 Pandemic in 2020, I stumbled across several eBay listings for arcade machines and one jumped out at me:

I had it shipped to me and yup - early 2000s PC

I immediately tried to turn it on, but it appeared to have a bad graphics card and not boot:

There were many variables - I wasn’t sure if it had something to do with my converter:

I didn’t have a VGA-native display anymore… I figured that maybe the X11 configuration understood a CRT monitor with 4:3 modes or something incompatibile with every TV I tried.

Also, the HDD was clicking when starting up, so I had to copy it several times to get a working image to restore:

So much for: just replaced the Hard Drive and Video Card…

Cracking it Open

I also opened the PC to take a look at the inside and get a better look at this A27 that had given me so many problems:

The card itself was built with a Texas Instruments PCI1410 CardBUS controller, some other chips, a Varta battery, and a few LEDs that lit up and blink when the system boots.

Putting it on Pause

Seeing as how it wasn’t booting, I then shoved it into my closet with a promise that eventually, I would get around to finally running logging on the hardware, then maybe it would work!

Due to other reversing projects, it would be another two years until I’d end up pulling the machine down and trying to get it going again.

Okay… Maybe Fourth time? - 2022

Troubleshooting the PC

Pulling this PC back out, I wasn’t sure of where to go next… I had a few options:

  1. Figure out what was wrong with why the PC won’t boot, fix it, then run the logger I had written years prior.
  2. Build my own machine, plug the PCI card in, install Red Hat 7.3, and use the kernel module to connect with the A27 and hope the game works.
  3. Get the real PC to run another more-compatible OS, build my own kernel module, and ssh from my VM to the real pc and pass requests from the game to the card or run the game as normal.

Each item had their own benefits and drawbacks:

Oh also, no keyboard I had tried worked - so it was also possible that keybord support was not compiled into the kernel.

After a couple of attempts, the PC no longer booted (powered up, but no POST). I decided to finally go for broke and reimplement the kernel module - expecting that I might need to replace harware in the PC or build a new one altogether…

Reimplementing the Kernel Module

Reimplementing the kernel module consisted of reconstructing each function in the pccard.o file that was on the hdd. After several attempts, I had built what looked like a replacement - things were looking up!

However, I continually got this message:

Recovering files from the hdd - I also saw additional kernel modules that appeared to be for other Linux 2.4 versions, and it still bugged me that the kernel module wasn’t insmodding on a real Red Hat 7.3 install. As a result, I chalked this one up to a dead end - there were too many variables that might make the card not work, and figuring that out wasn’t even my goal - I just wanted to have the PCI card talk to the real game on the real OS.

When I started digging into the files on the OS partition for answers, however, things got more clear…

Analyzing the OS Partition

The developers had left several artifacts from their build on the HDD - no source code for anything we cared about, unfortunately, but a fully built Linux source tree otherwise!

Looking into the files, it was apparent that pccard.o was the object file from this compilation, and not a kernel module!

Taking the vmlinux image out of the src tree and throwing it into IDA confirmed that, yes, they built the pccard driver into the kernel!

Looking back at the original startup as well, it was apparent that the real OS startup script was insmodding a file that wasn’t a kernel module, erroring, and continuing as normal. It’s unclear as to why they put this file in there. Perhaps the file not existing was causing an error with the script and they wanted to fix some edge case? Seeing as previous games that used this OS configuration used kernel modules, perhaps it was a design choice made for this game to not include a separate module, but it’s hard to say.

At this point, I knew I had to fix the PC and get it booting - having the PC boot to the OS would give me the best chance. Even if I was still back at square one with X11 not starting, I could always write a tool that would talk to the A27 and do what I needed…

Fixing the PC

I took a wild guess at the no POST symptom and bought a replacement power supply. I decided to get the exact PSU that was already in the case for compatibility.

Right away, the PC booted - I was back in business!!! Except the X11 issue remained. I still couldn’t boot into the game, anything graphical, nor could I get keyboard working.

I decided that getting the game working to ensure the A27 still worked correctly was the best initial goal. After all, anything could have happened before the machine got to me as it wasn’t a guarantee that this thing even worked beforehand.

I decided to get a new graphics card (an MX440) to see if the artifacting card was the issue.

At the same time, I also bought a Dell Monitor from eBay to ensure I had something that would be compatible:

Aaand the damn thing still didn’t boot!

Getting Keyboard Working

I decided that I wasn’t getting anywhere until I figured out how I could get keyboard working or network so I could SSH into the machine and poke around - changing files and rebooting while putting everything in scripts would be way too tedious.

I had also built a test program in C that would talk to the card like the game would to at least try one packet which worked!

I knew that the built-in pccard driver was working! The next step would be getting the real game working…

Getting the Game Working

Looking at the log messages from trying to insmod the NVIDIA kernel module, it appeared that the card was unsupported, but how!?

It turns out that I had recevied a P118 (FX5200), when I had really wanted a P70 (MX400) or P73 (MX440).

To be fair to the seller, they do look similar:

As this kernel module wouldn’t be able to support this card, and I really didn’t want to compile a newer (older) version of the kernel module and stick it in the OS, I bought a replacement card - actually, I bought two just in case:

I installed the new card aaaaaaand…

no POST…

I put the FX5200 back in - no POST

I put the original MX400 back in… also no POST.

Had I killed the motherboard this time? The Ports? Did the PSU fry again?

I bought another PSU - figured maybe I had a short somewhere killing this cheap piece of crap…

no POST…

Fixing the PC (Again)

At this point, I had sunk so much into this, I had to continue, but getting the motherboard would be a challenge. ECS isn’t typical like your average Gigabyte or MSI motherboard. In addition, I’d be looking for models from a specific era - ideally the same one. Fortunately, there was an identical motherboard online:

and a sister-model that was a little newer but had a mostly similar chipset that also should have worked:

I bought them both to have a backup that might reasonably work, and noticed that the original board was not fully mounted on standoffs to the case. In addition, I had read that the AGP slots on these models were notorious for shorting out the board.

Anyway, replacing the board, the PC finally booted again!

(Almost) Back in Business

I started up my test program aaaand it no longer worked - every time I tried to talk to the A27, I would get a kernel panic!

Had I killed the ASIC this time? It was possible - changing the motherboard, multiple shorts due to shoddy original installation.

After reinstalling all of the hardware (again), I decided to try something - knowing that the replacement motherboard came with its own RAM, and 1.5GB to boot, along with the fact that PCI card memory maps its IO, I decided to pop out the extra memory and stick to the original 512MB.

After that adjustment, the test was once again working. My only guess is something BAR related screws up when memory is mappable after a certain amount… clearly this was a bug from the driver.

Also, the game fully booted!

VIDEO

VIDEO

Finally - I could start really working on getting this game emulated… but the fun was just beginning…

Focusing on the Game

At this point, I had decided to write a hook on reads/writes for the A27 in order to understand what was happening. While I could see the original structures in the game executable, I wasn’t sure what all of the structures actually did.

Working through each screen of the setup menu, I recorded the various requests and responses to document each and understand their context.

I eventually documented the full request and response packet sent to the A27!

A27 Request Packet

The request packet consists of the following structure:

#define PCCARD_DATA_BUFFER_SIZE 0x4000

typedef struct _TRACKBALL_DATA{
	unsigned short player_index;
    unsigned short align;
	unsigned short vx;
	unsigned short vy;
	unsigned char direction;
	unsigned char press;
	unsigned char pulse;
    unsigned char align2;
}TrackballData,*PTrackballData;

typedef struct _A27_WRITE_HEADER{
	unsigned int data_size;
	unsigned int system_mode;
	unsigned int key_input;
	TrackballData trackball_data;
    unsigned char checksum_1;
    unsigned char checksum_2;
	unsigned char light_disable;
	unsigned char key_sensitivity_value;
	unsigned char light_state[4];
	unsigned char light_pattern[4];
}A27WriteHeader,*PA27WriteHeader;

typedef struct _A27_WRITE_MESSAGE{
    A27WriteHeader header;
    unsigned char data[PCCARD_DATA_BUFFER_SIZE];
}A27WriteMessage,*PA27WriteMessage;

A27 Response Packet

The response packet has a lot more going on - also, reading from the A27 responds with 0xF1 if successful:

typedef struct _A27_READ_HEADER{
    unsigned int data_size;
    unsigned int system_mode;
    unsigned char coin_inserted;
    unsigned char asic_iserror;
    unsigned short asic_errnum;
    unsigned int button_io[6];
    unsigned short num_io_channels;
    unsigned char protection_value;
    unsigned char protection_offset;
    unsigned short game_region;
    unsigned short align_1;
    char in_rom_version_name[8];
    char ext_rom_version_name[8];
    unsigned short inet_password_data;
    unsigned short a27_has_message;
    unsigned char is_light_io_reset;
    unsigned char pci_card_version;
    unsigned char checksum_1;
    unsigned char checksum_2;
    unsigned char a27_message[0x40];
}A27ReadHeader,*PA27ReadHeader;

typedef struct _A27_READ_MESSAGE{
    A27ReadHeader header;
    unsigned char data[PCCARD_DATA_BUFFER_SIZE];
}A27ReadMessage,*PA27ReadMessage;

A27 System Mode Codes

In addition, the A27 programs were also figured out at this point (for the most part):

enum A27_Program{
    A27_MODE_HEADER_UPDATE,
	A27_MODE_READWRITE_TEST = 1,
	A27_MODE_KEY_TEST = 2,
	A27_MODE_LIGHT_TEST = 3,
	A27_MODE_COUNTER_TEST = 4,
	A27_MODE_TRACKBALL = 5,
	A27_MODE_SCREEN_COIN = 11,
	A27_MODE_SCREEN_MODE = 13,
	A27_MODE_SCREEN_SONG = 14,
	A27_MODE_SONG = 15,
	A27_MODE_SCREEN_RANKING = 20,
	A27_MODE_CCD_TEST = 25,
	A27_MODE_RESET = 27
};

It’s believed that the A27 gets its name from 27 programmable subroutines that it has available. It should be noted that other games (e.g. Rock Fever 4, Speed Driver, etc.) likely reorder programs, may not use the same programs, or any combination of the two.

Now that I had a better understanding of what the game wanted at each screen, it was time to see if I could get gameplay working on the real machine with keyboard support!

The I/O

While the ‘button_io’ component of the response packet appeared to be read by the game for the menu, I couldn’t actually play any of the songs!

The reason behind this is that the game evaluates menu controls within the game executable, but the note ‘hits’ and ‘judgement’ are all happening on the I/O… from the serial/gameport on the A27 card from the original drums… which I didn’t have.

Speculative Note: From an architectural standpoint, having the ASIC perform the judgement would allow for much more granular timing and accuracy than what would be guaranteed on a single-threaded program from the early 2000s. By not having judgement happen every frame, and offloading that logic to the ASIC, an extended draw call or anything else that would screw up timing wouldn’t affect input. Also, it’s a handy way to break the game if you don’t have the ASIC!

However, as we’ve covered, the developers were kind enough to include the key_input on the request packet, which the A27 happily accepts as an alternate input… I could finally play songs!

The Patch… and Other Game Weirdness

While collecting packets from the ASIC data, I started writing a patch in an attempt to reproduce the game state presented on the real PC. Short of dumping the ASIC logic, I would be stuck with behavioral analysis in an attempt to approximate what the ASIC should be doing at any given point.

Digging into the game binary and being able to test on the real thing opened more interesting artifacts:

Also a character select menu which isn’t present.

The game also appears to have a Drum King or Percussion Master mode that isn’t in use, but utilizes both player drum controls at once.

Along with these options, I also added a windowed mode and some additional quality-of-life fixes (such as changing hardcoded paths to run the game from any path) and put them all under envars in the patch here.

It all started coming together… until I started working on the songs.

At this point, the intro, how to play, staff screen, demo loops, and the actual songs were still not running… and it’s all due to Program 15 - the ‘Song’ state.

Figuring out the SongState

Starting with the opening as a simple example, I wrote a script to parse all of the packet data to understand how this command works, and what is happening for each song.

As it turns out, the ‘Song’ command itself is actually broken down into several subcommands:

enum A27_Song_Subcommand{
 A27_SONGMODE_PLAYBACK_HEADER,
 A27_SONGMODE_PLAYBACK_BODY,
 A27_SONGMODE_MAINGAME_SETTING=3,
 A27_SONGMODE_MAINGAME_WAITSTART,
 A27_SONGMODE_MAINGAME_START,
 A27_SONGMODE_MAINGAME_PROCESS,
 A27_SONGMODE_RESULT=9,
 A27_SONGMODE_RESULTDATA_SET=11,
 A27_SONGMODE_RESULTDATA_COMPLETE=12,
};

I started songs and fuzzed the various bytes to determine which had an effect. To determine which were relevant, I looked at the SongProcess and SongProcessWrite functions in the game again. While most of these commands are fairly straightforward, aren’t necessary to figure out at this point, or are unused by the card, the ‘SongUpdate’ (6) command appeared to be next target.

Knowing most of the packet data structures, it was now possible to at least get something akin to notes showing up:

However, there were still several questions:

Dumping the Charts

Constructing some note charts from early dumps, it appeared that things were starting to come together, albeit not accurate by any stretch.

As no note data was being sent to the A27, it was appared pretty early on that the A27’s UVEPROM contained the note data for each song and difficulty. In addition, the song test menu had an option to record your own charts and play them back:

However, one other source of notes happen to be the game executable! I had never noticed before, but the development code for key recording included a copy of every note chart. I would assume the idea would be to build a new chart based on existing notes.

And yet another source of notes would technically be the packet dump I performed of every song and every chart thanks to the song test menu.

Seeing as I had three sources to reproduce the notes, I had a few options:

  1. Dump the UVEPROM, hope it’s not encrypted or something more of a pain, and figure out that format to play the real charts.
  2. Assume the charts in the executable were accurate, and dump them with a script.
  3. Read my packet dumps and approximate when a cursor appeared to reconstruct a chart. 4.

At this point, converting the built in note charts to a format that could be replayed seemed to be the best path forward. As a result, I wrote a script to extract the note charts from the game executable, and convert them into the very ‘rec’ format that the debug code used.

Now I had somewhat reasonable charts, yet didn’t have actual gameplay to use them with…

Rebuilding the Engine

At this point, it was pretty apparent that all of the stuff in SongUpdate required the ASIC to provide logic:

While recreating the whole engine was an option, the purpose of this was to faithfully recreate the ASIC logic as a best effort short of dumping the logic itself. As a result, I measured the values between dumps (frames) to make some determinations such as :

To support this kind of logic, I’d need quite a bit of code.

To pare down what had to be implemented:

All of our gameplay logic at this point relied on the current 32nd note beat - established from elapsed time in ms based on when the song started.

To calculate this, we have a 2 byte value in each note chart that represents the tempo/bpm (e.g. 13900 or 8755). These are binary coded decimal (BCD) and need to be converted to a float first (e.g. 139.00 or 87.55):

song_event->tempo = Short2Float(rec_file->header.hbpm);

Next, since we can’t make a thread that loops on beats natively, we have to convert it to something we CAN control - milliseconds.

We need the ms_per_beat for the song. To calculate this:

ms_per_beat = (60 * 1000) / bpm;

We’ll also need the ms_per_measure to determine where our measure bars are rendered - along with stretching the ‘fever’ notes the appropriate amount of pixels:

ms_per_measure = ms_per_beat * 4

Lastly, we’ll need the ms_per_32nd_note to determine what beat we’re on for each iteration based on the elapsed time of the song. Instead of having numbers in variable names because that’s ugly, we’ll call it ms_per_ebeat:

ms_per_ebeat = ms_per_beat / 8;

While we could deep dive into the logic that relies on this this code, at a high level:

At this point, everything appears to be working as expected:

VIDEO

VIDEO

Wrapping Up

I suppose the lesson here is - short of decapping or cloning the custom logic on the ASIC, handling the functionality essentially involves monitoring the behavior somehow and recreating part of the runtime - sometimes quite a bit of it.

It’s worth noting that PercussionMaster is similar to the previous game (Rock Fever 4) with the exception that RF4 contains a rearranged set of ‘commands’, it’s missing a few, and contains more ‘lanes’ than the given 6 per player (up to 36). Realistically, one might be able to somewhat easily convert pm_patch to work with that game as well once the notes are dumped from the game executable, and the inconsistencies from the A27 are addressed.

It’s also worth noting that the A27 contains two roms - an ‘external’ program rom and an ‘internal’ program rom inside of the ASIC package. While it’s possible to dump the external program rom, it’s likely encrypted with a key from the internal rom or something similar. But a truely accurate emulator will one day need to dump these along with recreating the ARM7 code or lifting it into llvm and then translating or something else.

Ideally, with a dump of the ROMs and logic from the ASIC along with some kind of lifter, a truly accurate solution can be made. Until then, we’ll just have to do things the hard way.

Cheers!

February 2023