gamearchive.js

Copyright 2010-2021 Adam Nielsen <[email protected]>

This is a Javascript library that can read and write archive files used by a number of MS-DOS games from the 1990s. Archive files are like .zip files, except many games used their own custom file formats. This library is an attempt to provide a unified interface for reading and writing many of these formats.

Supported file formats

| Game | Files | Format | Code | |---------------------------|------------|---------------------------|---------------------| | Alien Carnage [!C] | .-0 | Alien Carnage Data Bank | arc-bnk-carnage | | Blood | .rff | Monolith Resource File Format v2.0/3.0/3.1 | arc-rff-blood- | | Bio Menace | gamemaps. | id Software Gamemaps (RLEW) | arc-gamemaps-id | | Blake Stone | gamemaps.* | id Software Gamemaps (RLEW) | arc-gamemaps-id | | Catacomb 3-D | gamemaps.* | id Software Gamemaps (RLEW + Carmack) | arc-gamemaps-id-carmack | | Catacomb Abyss | gamemaps.* | id Software Gamemaps (RLEW + Carmack) | arc-gamemaps-id-carmack | | Catacomb Armageddon | gamemaps.* | id Software Gamemaps (RLEW + Carmack) | arc-gamemaps-id-carmack | | Catacomb Apocalypse | gamemaps.* | id Software Gamemaps (RLEW + Carmack) | arc-gamemaps-id-carmack | | Corridor 7 | gamemaps.* | id Software Gamemaps (RLEW) | arc-gamemaps-id | | Commander Keen 4-6 | .exe | Keen 4/5/6 .exe | arc-exe-keen[456] | | Commander Keen 4-6 | gamemaps. | id Software Gamemaps (RLEW + Carmack) | arc-gamemaps-id-carmack | | Cosmo's Cosmic Adventures | *.vol, *.stn | Cosmo Data Volume | arc-vol-cosmo | | Dangerous Dave | dave.exe | Dangerous Dave .exe | arc-exe-ddave | | Death Rally | *.bpa | Death Rally BPA File | arc-bpa-drally | | Descent | *.hog | Descent HOG File | arc-hog-descent | | Doofus | *.exe | Doofus .exe | arc-exe-doofus | | Doofus | *.g-d | Doofus Game Data | arc-gd-doofus | | Doom | *.wad | Where's All the Data File | arc-wad-doom | | Duke Nukem 3D | *.grp | BUILD Group File | arc-grp-build | | Duke Nukem II | *.cmp | Cosmo Data Volume | arc-vol-cosmo | | Gateworld | *.gw[123] | HomeBrew File Folder | arc-gw-homebrew | | God of Thunder | *.dat | God of Thunder Data File | arc-dat-got | | Grand Prix Legends | *.dat | Papyrus Data File (V2) | arc-dat-papyrus-v2 | | Halloween Harry | .bnk | SubZero Data Bank | arc-bnk-harry | | Hocus Pocus | hocus.exe | Hocus Pocus .exe | arc-exe-hocus- | | Hocus Pocus | *.dat | Hocus Pocus Data File | arc-dat-hocus | | IndyCar Racing | *.dat | Papyrus Data File (V2) | arc-dat-papyrus-v2 | | IndyCar Racing II | *.dat | Papyrus Data File (V2) | arc-dat-papyrus-v2 | | J.R.R. Tolkien's Riders of Rohan | *.dat | Papyrus Data File (V1) | arc-dat-papyrus-v1 | | Lion King, The | *.dat | East Point File Storage | arc-epf-eastpoint | | Lost Vikings, The | *.dat | Lost Vikings Data File | arc-dat-lostvikings | | Major Stryker | *.ms[123] | Cosmo Data Volume | arc-vol-cosmo | | Monster Bash | *.dat | F.A.S.T. Data File | arc-dat-fast | | NASCAR Racing | *.dat | Papyrus Data File (V2) | arc-dat-papyrus-v2 | | NASCAR Racing 2 | *.dat | Papyrus Data File (V2) | arc-dat-papyrus-v2 | | NASCAR Racing 3 | *.dat | Papyrus Data File (V2) | arc-dat-papyrus-v2 | | Nomad | .dat | Papyrus Data File (V1) | arc-dat-papyrus-v1 | | Operation Body Count | gamemaps. | id Software Gamemaps (RLEW) | arc-gamemaps-id | | Prehistorik | *.cur | Prehistorik Data File | arc-cur-prehistorik | | Raptor | .glb | Raptor Game Library | arc-glb-raptor | | Rescue Rover 2 | gamemaps. | id Software Gamemaps (RLEW + Carmack) | arc-gamemaps-id-carmack | | Redneck Rampage | *.grp | BUILD Group File | arc-grp-build | | Scubaventure | *.dat | F.A.S.T. Data File | arc-dat-fast | | Shadow Warrior | .grp | BUILD Group File | arc-grp-build | | Spear of Destiny | gamemaps. | id Software Gamemaps (RLEW + Carmack) | arc-gamemaps-id-carmack | | Stargunner | .dlt | Stargunner DLT Archive | arc-dlt-stargunner | | Super 3-D Noah's Ark | gamemaps. | id Software Gamemaps (RLEW + Carmack) | arc-gamemaps-id-carmack | | Terminal Velocity | *.pod | Terminal Reality POD File | arc-pod-tv | | Vinyl Goddess From Mars | .lbr | Vinyl Library File | arc-lbr-vinyl | | Wacky Wheels | .dat | Wacky Wheels Data File | arc-dat-wacky | | Wolfenstein 3-D v1.0 | gamemaps. | id Software Gamemaps (RLEW) | arc-gamemaps-id | | Wolfenstein 3-D v1.1+ | gamemaps. | id Software Gamemaps (RLEW + Carmack) | arc-gamemaps-id-carmack | | ''N/A'' | *.exe | Generic MS-DOS Executable | arc-exe-generic |

• [!C] Compression/decompression is not yet implemented.

Installation as an end-user

If you wish to use the command-line gamearch utility to work with game archives directly, you can install the CLI globally on your system:

npm install -g @camoto/gamearchive-cli

For Arch Linux users the AUR package gamearchive-cli is also available.

Command line interface

The gamearch utility can be used to manipulate archive files. Commands are specified one after the other as parameters. Use the --help option to get a list of all the available commands. Some quick examples:

# Copy a file out of an existing archive
gamearch open duke3d.grp extract stalker.mid

# Create a new archive
gamearch add stalker.mid save -t arc-grp-build music.grp

# Convert an archive into another format
gamearch open duke3d.grp save -t arc-rff-blood-v301 duke3d.rff

# Extract a file by offset from an archive that has no filenames
gamearch open data.dat extract -n output.filename @100

# Extract all files from an archive
gamearch open duke3d.grp extract '*'

# Extract all files from an archive that has no filenames
gamearch open data.dat extract '@*'

To get a list of supported file formats, run:

gamearch --formats

Installation as a dependency

If you wish to make use of the library in your own project, install it in the usual way:

npm install @camoto/gamearchive

See cli/index.js for example use. The quick start is:

import { arc_grp_build } from '@camoto/gamearchive';

// Read an archive into memory
const content = {
    main: fs.readFileSync('duke3d.grp'),
    // Some formats need additional files here, see handler.supps()
};
let archive = arc_grp_build.parse(content);

// List the files in the archive
for (const file of archive.files) {
    console.log(file.name);
}

// Extract a file, decompressing it if necessary.
const data = archive.files[0].getContent();
fs.writeFileSync(archive.files[0].name, data);

// Rename a file
archive.files[0].name = 'RENAMED.BIN';

// Remove a file
archive.files.splice(i, 1);

// Replace a file
let f = archive.files[1];
f.diskSize = f.nativeSize = fs.statSync('new.bin').size;
f.getContent = () => fs.readFileSync('new.bin');

// Write the archive back to disk with the modifications
const outBuffer = arc_grp_build.generate(archive);
fs.writeFileSync('new.grp', outBuffer.main);

Installation as a contributor

If you would like to help add more file formats to the library, great! Clone the repo, and to get started:

npm install

Run the tests to make sure everything worked:

npm test

You're ready to go! To add a new file format:

1. Create a new file in the formats/ folder for your format. Copying an existing file that covers a similar format will help considerably. If you're not sure, arc-grp-build.js is a good starting point as it is fairly simple.

2. Edit formats/index.js and add an import statement for your new file.

3. Make a folder in test/ for your new format and populate it with files similar to the other formats. The tests work by creating a standard archive file with some preset files in it, and comparing the result to what is inside this folder.

   You can either create these archives by hand, with another utility, or if you are confident that your code is correct, from the code itself. This is done by setting an environment variable when running the tests, which will cause the archive file produced by your code to be saved to a temporary file in the current directory:

   # Prepare the location for the test files.
   mkdir test/arc-myformat/
   touch test/arc-myformat/default.bin   # Repeat for all needed files
       
   # Run the tests and save the output.
   SAVE_FAILED_TEST=1 npm test
       
   # Check the failed output and if it's correct, overwrite the expected
   # output with the test result.
   mv test/arc-myformat/default.bin.failed_test_output test/arc-myformat/default.bin

   It is helpful however, to create these files first before implementing your new format, as then you only need to keep running the tests and tweaking your code until all the tests pass.

4. Create a file in test/ for any extra tests your new format needs. Typically all formats will at least have tests that confirm the identify() function is correctly rejecting files, but you can also add additional tests here if your format needs it. See test-arc-pod-tv.js for a minimal example that only has tests for the identify() function, or test-arc-dat-fast.js for an example with a number of extra tests, in this case testing that filenames are correctly converted into code numbers, something that is unique to that file format.

5. Update the README.md with details of your new format and supported games.

If your file format has any sort of compression or encryption, these algorithms should go into the gamecomp.js project instead. This is to make it easier to reuse the algorithms, as many of them (particularly the compression ones) are used amongst many unrelated file formats. All the gamecomp.js algorithms are available to be used by any format in this library.

During development you can test your code like this:

# Read a sample archive and list the files, with debug messages on
$ DEBUG='gamearchive:*' ./bin/gamearch.js open -t arc-myformat example.dat list

# Make sure the format is autodetected correctly or if not why not
$ DEBUG='gamearchive:*' ./bin/gamearch.js identify example.dat

# Run only unit tests for the new format, with debugging on
$ DEBUG='gamearchive:*' npm test -- -g arc-myformat

If you use debug() rather than console.log() in your code then these messages can be left in for future diagnosis as they will only appear when the DEBUG environment variable is set correctly.

Development tips

This is a list of some common issues and how they have been solved by some of the format handlers:

Archive doesn't store filenames

• dat-hocus has a list of fake filenames that are assigned based on the index, with unknown.123 used for any extra files.
• lbr-vinyl stores an integer hash instead of a filename, so a list of known filenames is used to convert the hash back into a name, with unknown-123 used for any unmatched files.

Archive has duplicate filenames

• res-stellar7 appends the order number to each filename, so two files called SNG become SNG.0 and SNG.1.

Archive has folders

• dat-fast and pod-tv have the folder stored as part of the filename with a backslash as separator, so they are returned as-is (e.g. digi\sound.voc).
• res-stellar7 has subfolders stored as separate FAT blocks so from these, filenames are generated that include the folder, e.g. SSM.0/SNG.1.
• wad-doom has empty files used for starting and ending a group of files, so to ensure these files are kept together they are converted into a virtual folder. So files P_STRT, EXAMPLE, and P_END come out as P/EXAMPLE.