A Python Server That Suddenly Stopped Working After Years — The Most Unusual .pyc Corruption I've Seen

on

After more than ten years of maintaining Linux servers and Python applications, I encountered a failure I had never seen before.

The application had been running for years without any changes. One morning it was simply down after a restart. Nothing had been deployed, no code had been modified, and there had been no package upgrades.

Initial Symptoms

The first error looked like an issue with Sentry (Raven):

ImportError: No module named eventlet

After temporarily removing the Raven middleware from the WSGI configuration, a new error appeared:

RuntimeError: populate() isn't reentrant

At first glance, this looked like a Django application registry problem. However, this error was only a secondary symptom. Django had already failed during application initialization, and every subsequent startup attempt produced the misleading populate() isn't reentrant exception.

Finding the Real Cause

Running Django directly from the command line finally exposed the actual problem:

ValueError: bad marshal data (string ref out of range)

The traceback pointed to:

pkg_resources
→ packaging
→ pyparsing

This error is extremely uncommon. It usually indicates that Python is trying to load a corrupted compiled bytecode file (.pyc).

The source code itself was perfectly fine.

The Fix

Removing the cached bytecode files immediately solved the problem.

sudo find /usr/lib/python2.7/dist-packages/pkg_resources \
    -type f \( -name "*.pyc" -o -name "*.pyo" \) -delete

After that:

python2.7 -c "import pkg_resources; print('OK')"

returned:

OK

The Django application started normally, Apache loaded the WSGI application without errors, and the website came back online.

Why Can This Happen?

Python stores compiled bytecode in .pyc files to speed up imports.

If one of these files becomes corrupted—for example because of:

  • unexpected power loss,

  • disk or filesystem issues,

  • interrupted writes,

  • storage corruption,

  • RAID inconsistencies,

Python may fail while loading the cached bytecode even though the original .py source files remain completely intact.

Instead of recompiling automatically, Python can abort with errors such as:

ValueError: bad marshal data (string ref out of range)

or similar marshal-related exceptions.

Deleting the damaged .pyc files forces Python to regenerate them from the original source code.

Lessons Learned

Several misleading errors appeared before the real cause was identified:

ImportError: No module named eventlet
RuntimeError: populate() isn't reentrant
ValueError: bad marshal data (string ref out of range)

Only the last one actually pointed to the underlying problem.

This case is a good reminder that secondary exceptions during framework startup often hide the original failure. Running the application directly from the command line instead of relying solely on Apache or mod_wsgi logs can significantly reduce troubleshooting time.

In my case, deleting the corrupted .pyc files restored the entire production system within minutes.

After more than a decade working with Python and Linux servers, this was the first time I had encountered bytecode corruption causing a complete production outage.

Keywords

Python, Python 2.7, Django, Apache, mod_wsgi, WSGI, pkg_resources, pyparsing, bad marshal data, string ref out of range, RuntimeError populate isn't reentrant, ImportError eventlet, Raven, Sentry, corrupted pyc, Python bytecode corruption, Linux server, production outage, troubleshooting, Django deployment.

Comments

Post a Comment