Why is moving the Request Broker so painful?

The normal cycle of Evolution server retirement means that machines come and go, and for most of this the cycling is done without too much impact on the users, at least in the interruptive sense because the changes can be made quietly in the off hours.

Even a DB server can be swapped out overnight without the users noticing (save for speed improvements).

But not the Request Broker: moving the RB is the most interruptive process in Evolution server administration, and IT staff often wants to hold off on this as long as possible due to the interruption involved.

This post raises issues that must best dealt with during an RB migration. Note that it's not attempting to be a how-to, only to convey the efforts required.

Work done in advance

  • Install the Evo Deployment Manager and (using the Evo Mgmt Console) bring the new machine into the Evo fold. Add a Request Processor for sure.
  • Request a new license key from iSystems.
  • Define all VMR printers on the new RB machine with exactly the same names as found on the old RB machine. You can try to use the same drivers too, but in practice they're never the same enough to avoid printer issues.
  • Configure the usual shortcuts (to the Evo MC, to iSystems downloads, etc.) in the web browser.
  • Run test prints to each printer on the new machine, and check the test pages carefully to be sure that each one comes out on the proper printer: this is really easy to get wrong.
  • Be sure that the new machine is allowed to send email through the local mail relay machine (perhaps an Exchange server).
  • Install and configure the other Evo services that will go on the RB machine; this includes the Request Processor, Remote Relay, and API adapters.
  • Update the firewall to route inbound Remote Relay and API from the internet to the new machine. This will interrupt any connections in progress, so this should be done in the off hours.
  • Remove RR and API services from the old machine. If the old machine is to be completely decomissioned, remove the Request Processor too.
  • If the central Evolution Updates folder is on the old machine, copy it to the new, share it, and update all the shortcuts on all middle-tier machines to refer to the new location of the Evo Updates area. Un-share it on the old machine, then (after a time) delete it entirely.
  • Migrate the Evo Management Console to the new machine, which for many customers also requires an internal DNS update for the name emclive. Note that if the RB and MC are not on the same machine, it changes the way routine updates are done.
  • Since the RB is usually (always?) the one sending email, the mail relay server must be configured to allow connections from the new machine.
  • Create the necessary RB hierarchy on the new machine. This usually means a set of directories: EvoQueue, EvoVMR, EvoACH, and probably others. Survey carefully the old machine to make sure you've gotten them all. Prefer the D: drive for these directories if possible to free up space on the system C: drive, but it's not always possible to do this.
  • Insure that sharing of folders on the new machine mirrors the sharing on the old, including names and permissions. ACH permissions, in particular, are pretty important to get right to limit who can put/take a file there.
  • Using the standard Windows tool robocopy, make an initial copy of the full set of directories from the old to the new; this often includes a huge bulk of data, so getting a first pass done up front means less to do at the time of the update:
    robocopy \\evo1\D$\EvoVMR D:\EvoVMR /mir
    robocopy \\evo1\D$\EvoQueue D:\EvoQueue /mir
    robocopy \\evo1\D$\EvoACH D:\EvoACH /mir
    ...
    The /mir option says to "mirror" the source, and I typically put these copy commands in a script so I can run it again later.
  • Survey all other application that run on the RB machine, such as Evo Exchange or (commonly) WebApps middleware. These have to be installed on the new machine, but not actually enabled until it's time to do the actual cutover.
  • Check the task scheduler on the old machine to see what's running; configure similar tasks to run on the new machine (but you'll typically make them inactive until cutover).
  • Install whatever of my utilities that normally live on the RB machine; this almost always includes Clickrestore, and I'm always adding more to the mix.

On cutover

Though the above can all be done in the off hours and scheduled more or less randomly, the actual cutover has to be done all at one time. Figure an hour of downtime to get it all sorted out, assuming you've done your prep work properly.

  • In the Evo MC, stop all Evo services
  • In the Evo MC, uninstall the RB from the old machine, then install it on the new machine
  • In the Evo MC, start the RB (only!) and enter the license information— serial number and license key— and make sure it starts correctly. If the key is no good, go back to the RB on the old machine.
  • Populate the rest of the RB configuration, possibly using the Evolution.cfg file from the old server.
  • Stop the RB.
  • If you use the evolive hostname to access the server from the inside of the office, update internal DNS to reflect the new machine's IP address. Do this early in the process (once the RB license is confirmed) so that the old one has time to age of DNS caches.
  • Re-run the script containing the ROBOCOPY tool to make sure the new-machine copy of all the key files (Queue, VMR, ACH, etc.) exactly mirror the files on the old. Do not restart Evo services until this process completes.
  • Inform users to use the new share names instead of the old share names.
  • Un-share pretty much everything from the old machine; if users remain connected to the wrong machine, we want it to break rather than silently use the wrong data.
  • With the Evo MC, start the Request Broker and Request Processors (only!), and have an administrator login to Evo and check out the VMR printer configuration. Run sample jobs through the queue (as opposed to running through the workstation) to make sure that jobs print where they belong, that trays feed properly, and so on.If the printer configuration on the middle tier is changed (updating a tray from letter to legal, for instance), it usually requires stopping, then starting the Request Broker, then re-configuring VMR printers from within Evo.Only the customer can do this; I can't do it remotely.This is typically the most troubling aspect of RB migration.
  • Enable Windows scheduled tasks on the new machine; disable scheduled tasks on the old machine. Don't delete the old tasks yet though.
  • If any Evo scheduled tasks save files to the RB machine, check the paths to make sure they all point to the right place. Where the migration kept like for like (e.g., D:\EvoACH on both machines) there may be nothing to do, but if the new machine has no D: drive but the old one did, you'll be changing a lot of D: to C:The customer has to do this.
  • Periodically go into the Evo Task Queue and check for failed jobs which might be associated with the migration; there might still be paths to adjust.

After the fact

Once the new RB has been working for a while, it's safe to remove all the remnants of the RB from the old machine, and perhaps to decommission it entirely.

  • In the Evo MC, remove the Deployment Manager associated from the old machine
  • On the old machine, use Add/Remove Programs to uninstall the Deployment Manager and Management Console (as appropriate).
  • Delete the old Evo directories, including EvoVMR, EvoQueue, EvoACH, and any others that were associated with the RB.
  • Outright delete any Windows scheduled tasks that had been previously disabled.
  • Power the old machine down for a week or so to see if anything breaks