Using multiple internet providers

Customers ask from time to time if it's possible to get two separate internet circuits and perform load balancing / failover, so that if one goes down the other will keep going. Many firewalls now come with multiple WAN (Wide Area Network) interfaces, so this is a reasonable question.

Unfortunately, it probably doesn't do what you hope it does, which is to make circuit failure mostly invisible.

What works

Firewalls with two (or more!) internet connections do a really good job at load balancing traffic from inside to outside, servicing requests from your staff to the rest of the internet.

Since the firewall can tell when any given circuit has failed, it can send the traffic out the other circuit, and this often works so well that staff inside your office are unaware when failover occurs.

And many firewalls support Policy Based Routing, that can give preference to some kinds of traffic to one of the circuits, such that — for instance — outbound ADR transfers to the iSystems hotsite can be preferenced to the "lesser" connection so it won't swamp the same bandwidth used by Evo Remote customers.

Caveat: some outside services or websites make note of your source IP address, and if your IP address bops back and forth as the firewall uses both circuits for your traffic, this might confuse things. In fact, this used to be a problem with Evolution when Evo Remote and its associated Report Writer SSL connections didn't come from the same IP address.

In practice, the few sites that have this problem can usually be addressed by routing policies in the firewall.

What doesn't work

Inbound connections will not load balance; we'd love it if inbound Evo Remote would automatically be redirected to the other internet circuit so customers wouldn't be impacted during failure, but this is simply not generally possible.

Each connection has its own IP addresses assigned by the provider (Verizon, Time Warner, etc.), and there's generally no technical way for (say) Verizon to accept traffic that ought to be headed to Cox Cable, and there's no mechanism for the firewall to do this or ask the provider to do this.

It's a reasonable request, but it just doesn't work.

Instead, many service providers can define multiple DNS hostnames representing the different circuits (say, evo1.stevepay.com for the first circuit and evo2.stevepay.com for the second), and ask customers to use the "other" hostname when one goes down.

There are complications here, especially for customers who must whitelist your service bureau's IP address in their border firewall, but this just means going through the process twice rather than once.

NOTE: I do not recommend updating DNS with an alternate IP address in the event that a circuit is down unless the circuit will be down for some time. DNS resolvers typically cache the name-to-IP mapping, so they won't see changes immediately, and going to the backup and from the backup will not be a smooth transition.

Using separate names avoids this issue even though it does require customer interaction.

The BGP option

Technically, there is a way to have seamless inbound redundancy, but it requires expensive gear and even more expensive internet circuits: I'm only mentioning it here for completeness.

To do this, you need a router that supports the Border Gateway Protocol, a class C (or larger) IP address block assigned to you, and another ISP who's willing to BGP route the other guy's block to you.

In this way, the same IP block can be serviced by either circuit, and if one circuit goes down, BGP drops the bad route, allowing traffic to reach the working circuit.

Though it works very well — the entire internet operates with BGP — it's complicated and very expensive, and I believe it to be costly overkill for the Evolution service bureau.