The game that corrupted its own help files: fixing Civilization's 30-year-old CHANGE freeze

If you have played the original Civilization (MicroProse, 1991) in DOSBox, you may have hit this one: you open a city, press CHANGE to pick something new to build, and the game freezes. No error, no crash, just a hung screen. The READ.ME that ships with version 474.03 knows about it (item 23) and suggests a workaround: turn off Instant Advice.

That avoids the symptom, but not the cause. I wanted to know what was actually happening, so I took it apart. The short version: the game corrupts its own data files, a little more every time it hits a specific code path, until a lookup loop has no way out.

TL;DR

The freeze is an infinite loop in the game's indexed text lookup. Three defects combine:

1. Incomplete shipped indexes. Each indexed .TXT file omits its last entry from its hash index. In PRODUCE.TXT that entry is *Caravan.
2. Off-by-one write-back. When a lookup falls back to a sequential scan, the game "helpfully" writes the found offset back into the index. But the offset comes from ftell() on a text-mode stream and is usually 1 byte late, so it never matches. Every later lookup repeats the scan and burns another empty bucket.
3. Unbounded probing. The lookup probes buckets until it finds a match or an empty bucket. Once every bucket is full, a key that can't be matched loops forever.

The fix: rebuild the indexes of seven .TXT files (text unchanged), plus an optional 1-byte patch to CIV.EXE that disables the write-back so they can never fill up again.

The symptom

• Only happens with Instant Advice on.
• The last thing on screen is the Military Advisor recommendation. The Domestic Advisor box, normally shown next, never appears.
• Intermittent: some cities, some turns. Later saves of a game are affected while earlier ones are not. Forum folklore blames unit stacking in or near the city.
• Reproduced in DOSBox 0.74-3; READ.ME says it happens on real hardware too.
• A useful clue: PRODUCE.TXT and ERROR.TXT had modification dates later than the install, even though nobody had edited them.

That last point turned out to be the whole story.

Setup and approach

The investigation went from data, to code, and back to the running game:

1. Save-file comparison. I parsed the .SVE format (28-byte city records, 12-byte unit records including the per-tile stack link) and compared an affected city with the same city in an unaffected save. Every stack link was well formed, so stacking corruption was ruled out as a direct cause.
2. Unpacking the executable. I wrote an EXEPACK decompressor. The stream unpacked cleanly to 201,520 bytes with its relocation table; the data segment base sits at linear 0x2A800, found via the C runtime copyright string.
3. Locating the code. References to the strings Military Advisor: and produce.txt lead to the advisor routine and to a generic text lookup at 1ECB:01B3, disassembled with ndisasm.
4. Format analysis and simulation. I decoded the index at the top of the .TXT files, reimplemented the lookup in Python and ran it on all 73 unit, improvement and wonder names.
5. Reproduction. Load a save, open the city, press CHANGE, check for a freeze. Repeated with original and corrected files, diffing the files before and after to catch writes by the game.

Diagnosis

The advisor flow

With Instant Advice on, CHANGE computes a military and a domestic recommendation. For each one the game builds a key of * plus the item name (e.g. *Caravan) and looks it up in PRODUCE.TXT to get the advice text ("We should build a …"). The military lookup succeeds and its box is drawn. The domestic lookup never returns.

In every frozen case, the domestic recommendation was Caravan. Unit stacking only matters indirectly: the defenders in and around a city change what the advisors recommend.

The indexed text file format

PRODUCE.TXT, ERROR.TXT, KING.TXT, HELP.TXT and BLURB0–BLURB4.TXT all start with a 528-byte hash index: 256 little-endian 16-bit buckets, written as 8 lines of 32 buckets, each line ending in CR LF. Entries start at offset 0x212 (530) as lines beginning with *, and the file ends with *END.

A bucket holds the file offset of an entry's *. The value 0x3030 (ASCII 00) means empty. The hash of a key of n bytes is:

h = sum(i * c[i] for i in 0..n-1)  mod 256

Collisions are resolved by linear probing. Bucket b lives at file offset 2 * (b + b/32) (the b/32 accounts for the CR LF after every 32 buckets).

The lookup routine (1ECB:01B3)

1. Compute h. Open the file in binary mode, read the 16-bit value at 2 * (h + h/32), close it, reopen it in text mode ("rt").
2. If the value is <= 0x1FF or equals 0x3030, the bucket is empty: go to the sequential scan.
3. Otherwise seek to the value, read one line (%[^\n]\n) and compare it with the key. Match → found.
4. No match → h = (h + 1) & 0xFF (1ECB:0398) and go back to step 1. There is no probe counter and no other exit.
5. Sequential scan: seek to 0x200; for each line remember ftell(), read the line, compare. Stop on a match or at EOF.
6. Write-back (1ECB:03DC): if the scan found the key, reopen the file as "r+b" and write the remembered ftell() value into the empty bucket where probing stopped.

Step 6 is a self-healing cache: the index repairs itself the first time a missing key is looked up. In theory.

           ┌──────────────────────────────────────────────┐
           ▼                                              │
  read bucket[h] ──empty?──yes──► linear scan ──found──► write ftell() into bucket[h]
           │                          │                   (off by one byte!)
           no                      not found
           ▼                          ▼
   line at offset == key? ──yes──► return found      return not found
           │
           no
           ▼
    h = (h+1) & 0xFF ───────────────────────────────────────┘
    (no counter: if no bucket is ever empty, this never ends)

The three defects, together

Incomplete shipped index. The last entry of every indexed file is missing from its index. *Caravan is the last entry of PRODUCE.TXT. So the first Caravan lookup misses the hash, scans, finds it, and writes back.

Off-by-one write-back. The offset from ftell() on a text-mode stream is usually 1 byte past the *: 2099 instead of 2098 for *Caravan (and in one file, 1 byte early). From offset 2099 the game reads Caravan without the *, which never matches *Caravan. So the next lookup probes past that bucket, reaches the next empty one, scans again, and writes 2099 again. Each Caravan recommendation consumes one empty bucket.

Unbounded probing. After roughly 205 Caravan recommendations no empty bucket remains, and every one of them holds 0x0833 (2099). Now any key whose probe chain never matches spins forever, opening and closing the file on every step. In the fully corrupted file this happens for 23 of the 73 keys: Caravan, Palace, and all 21 Wonders.

That explains everything that looked random: it depends on how many times the game has recommended a Caravan since install, and on whether this particular city's advisor recommends one of the 23 doomed keys. It also explains why the problem seems to get worse the longer you play (the damage accumulates in PRODUCE.TXT itself, not in the save), and why the file's modification date changed.

Confirmation

Reproduction

Watching the corruption happen

I started from a correct index with only the *Caravan bucket emptied, then pressed CHANGE three times on an affected city. After each press the game had written 2099 (0x0833) into one more bucket: 117, then 118, then 119. That's the corruption, step by step, live.

Audit of all indexed files

Every bucket was checked against the real entry offsets. Unindexed = entry missing from the index. Bad buckets = values not pointing at an entry's *, all off by one byte.

The KING and BLURB traces were probably already in the distributed archive.

Fix #1: rebuild the indexes

Replace the 528-byte index of the seven damaged files with one rebuilt from their actual entries. The text is untouched; only bytes 0–527 change.

# Rebuild the hash index at the top of a CIV .TXT file (256 buckets, CRLF every 32)
import struct, sys, re

P = bytearray(open(sys.argv[1], 'rb').read())
keys = [(m.start() + 1, m.group(1)) for m in re.finditer(rb'\n(\*[^\r\n]+)', P)]
tab = [0x3030] * 256
for off, k in keys:
    if k == b'*END':
        continue
    h = sum(c * i for i, c in enumerate(k)) & 0xff
    while tab[h] != 0x3030:
        h = (h + 1) & 0xff
    tab[h] = off
hdr = b''.join(struct.pack('<32H', *tab[i*32:i*32+32]) + b'\r\n' for i in range(8))
assert len(hdr) == 528
P[:528] = hdr
open(sys.argv[2], 'wb').write(P)
print(len(keys) - 1, 'keys indexed')

• Fidelity check. The rebuilt indexes match every correct bucket of the originals exactly (50 in PRODUCE, 9 in ERROR, 58 in KING, and 67/89/55/41 in BLURB0/1/2/4). That confirms the hash and probing order are the game's own. The only differences are the bad buckets (fixed or cleared) and the missing entries (added).
• Why it works. Every key in the file is now found through the hash and never reaches the write-back. Keys absent from the file, like Palace and the Wonders, reach an empty bucket, scan to EOF, return "not found", and write nothing. The probe loop always terminates.
• Simulation. 23 of 73 keys loop forever with the original PRODUCE.TXT; zero with the rebuilt one.
• Emulator. All saves, Instant Advice on: no freeze, and the Domestic Advisor shows its Caravan advice. Across six CHANGE presses the files stayed byte-identical: the game no longer writes to them.

Fix #2: a 1-byte patch to CIV.EXE

Rebuilt indexes stop the freeze, but the self-healing write-back is still a loaded gun. So CIVFIX.EXE is CIV.EXE with one byte changed so the lookup never writes back.

The guard at 1ECB:03C7 skips the write-back when the hash found the key (jnz 0x0440) or when the scan didn't (jz 0x0440). Turning the second jump into an unconditional jmp skips it every time. The return value at 0x0440 is unaffected, so callers still get found/not-found.

Patching inside a compressed executable

CIV.EXE is EXEPACK-compressed, which normally means you can't just poke a byte. But EXEPACK is a run-length scheme that stores non-repeating bytes as literal copy blocks. Mapping each unpacked byte back to its source showed that unpacked 0x1F083 is stored literally at file offset 0x1F072. So it can be changed in place: the compressed stream, relocation table, header and overlays all stay the same, and EXEPACK has no checksum. Unpacking the patched file yields an image that differs from the original at exactly that one byte.

Why not just fix the offset?

The written offset comes from ftell() on a text-mode stream. It was 1 byte late in most cases and 1 byte early in one. Writing ftell() - 1 would fix most entries and break the rest. Skipping the write-back is simpler and always safe: unindexed entries are found by the sequential scan on each lookup, about 2 KB of reading, which is not noticeable.

Tests

• Starting from an index with *Caravan removed, CHANGE pressed three times on an affected city: advice shown, no freeze, file unchanged.
• Run together with the rebuilt files: game starts, loads a save, shows advice, writes nothing.

How to apply it

1. Back up your game folder (at least CIV.EXE and the seven .TXT files).
2. Rebuild the index of PRODUCE.TXT, ERROR.TXT, KING.TXT, BLURB0.TXT, BLURB1.TXT, BLURB2.TXT and BLURB4.TXT with the script above.
3. Optionally, apply the 1-byte patch: check that offset 0x1F072 holds 74 6B, change 74 to EB. Keep the original as CIV_ORIG.EXE and keep the patched file named CIV.EXE (the runtime refers to its own image as civ.exe).

The patch prevents new damage but doesn't repair an index that is already full, so use it with the rebuilt files, not instead of them. After that you can turn Instant Advice back on.

Limitations

• I didn't trace exactly why the C runtime's text-mode ftell() is off by ±1. The fix doesn't depend on it.
• The probe loop is still unbounded in the patched executable. It's unreachable as long as indexes keep their empty buckets, but an index damaged by hand or another tool could still trigger it.
• Tested in DOSBox 0.74-3 only. The behaviour lives in the game's own code and data, so real hardware should behave the same.
• Version 474.03 only. Other releases may place the routine elsewhere: check for 74 6B at the offset before patching.

Key addresses

Unpacked image; data segment base at linear 0x2A800.

A cache that writes back a slightly wrong key, a hash table with no probe limit, and an index that ships one entry short. Each one is harmless on its own; together they make a game slowly eat its own help files until it hangs. A nice reminder that "self-healing" code needs a test for what happens when the healing is wrong.