Evolution Best Practice: reboot middle tier monthly

iSystems and I have been researching an issue for some time, one surrounding this error message:

Cannot initialize Report Writer
Communication file
"C:\EvoTemp\EvRequestBroker\46F4197BFB96416FA1C2A8D370313616.tmp" is missing

Though there are several things that can cause it, in most cases I've seen it's a key kernel area — the global atom table — running out of space, and the only fix is to reboot the entire computer (restarting Evo services is not enough).

Recommendation: Reboot middle-tier servers, especially the one with the request broker, once a month to clear out this table before it fills up.

This reboot can be part of routine system maintenance, such as while applying Windows updates, and even every other month is probably OK.

Technical Details

Windows maintains something known as a "global atom table", and it holds key information to allow processes to communicate with each other, and this table holds up to 16k items. Though one kind of item (atoms) can be removed when no longer needed, the other kind of item (registered window messages) cannot be, and when this table fills up, we're stuck, with only a reboot available to clear it.

Normally, 16k items is plenty, but there's a bug in the Delphi libraries that Evolution uses, where it registers a new Window message in the form ControlOfs00400000000043C4, which is composed of the module's load address plus the thread ID. Since this is generally (but not universally) different each time an Evo program loads, these messages keep getting added (with no way to remove them), the table growing slowly until it fills up.

I had been closing in on this issue for months, but it was an iSystems dev who found the exact issue: a tip o' the geek hat to Dmitry for his good sleuthing.

We've observed that systems running Server 2008 or later have the problem a bit more often than those running Server 2003, and I believe this is because ASLR (Address Space Layout Randomization) changes the module's load address, meaning more unique window message strings. We're not 100% sure this is the reason, but the correlation is quite strong.

I've written some monitoring tools that can keep track of the size of the global atom table, sending an alert when the table starts getting too large, giving us time to reboot before the table gets out of hand. Logging output like:

Session 0; Service-0x2-86a999fb$; 26 atoms; 16294 wmsgs; 16031 Delphi; 16320 total

... letting me keep an eye on things - the above message shows a system that's failing with 16k items in the table.

Unfortunately for this customer I hadn't yet deployed the monitoring on their gear (sorry Bob!), so I captured the above right before rebooting. It's very interruptive when it happens during the day.

If you're my customer, I'll be setting this up on your middle tier servers so I can watch it, as well as scheduling reboots as needed.

The fix

Since Evolution is not using these registered Window messages for interprocess communications, they're able to modify the Delphi control library to not generate this multitude of messages, sticking with a single fixed string. This string will be re-registered over and over, taking just one slot in the table, avoiding overflow.

And though the fix is straightforward from a source-code perspective, it concerns all of us that it might have surprising side effects, so it's going to take a while to test very, very thoroughly.

In addition, the devs are focused on getting the Quechee release out the door, and since we have a workaround — reboot periodically — it reduces the pressure on the real fix.