One of the issues with any complex eCommerce site is the issue of single points of failure.
This cropped up yesterday with a Payment Gateway issue, but we were able to quickly bypass it… saving customer orders in the process.
What’s Redundancy? What’s Redundancy?
When you’re paying for Web hosting with a company, you’re paying for multiple redundant power connections. You’re paying for multiple Internet connections. You’re paying for multiple cooling systems. And you’re paying for a backup diesel generator on the roof, so that when that construction guy accidentally cuts a power line, your site is still up and running.
Your Website probably runs on multiple hard drives in a RAID-array, so that if one drive fails (and they all do at some point in their lives), the other one can keep going and you don’t lose functionality.
Our phone system relies on a couple of systems to work properly. An extended power outage, for example, can wreak havoc on the phones. But the failover is that we can use a Web portal to instantly re-route all inbound calls to a cellphone or alternative backup location. (We had to do this last week, when a construction guy cut through a main power line outside the building.)
Some things, though, like Payment Gateways, don’t have as much failover. It’s a dreaded single point of failure.
Last night, we received a whole bunch of calls from clients using Authorize.net as a payment gateway on their Magento stores.
What’s a payment gateway? When a person goes to a site to make a purchase, they often pay via credit card. When the person enters in their credit card onto the payment page, the shopping cart system encrypts the credit card number, and then securely sends it to a payment gateway. The payment gateway responds back, saying that it’s a good card / and it’s authorized for the amount of the purchase. Or it might respond back and say that the card is invalid, over it’s limit, or the billing address doesn’t match the order, and the order is rejected.
But yesterday, the payment gateway for authorize.net started sending back an error message for everyone trying to make a purchase, on hundreds of thousands of sites.
Now Authorize.net is a really robust, solid solution, and downtime is rare. But when it happens, you’re out of business. While some customers will call, others are just going to go away, and be frustrated that your site doesn’t work.
The good news is that the outage was short… no more than an hour or so.
So what to do if this happens again? What should you do?
We have a new client that called up, just a few minutes shy of the end of the day, that was affected. He only found out, though, because one of his customers called.
After logging into his Magento store, I was able to quickly by-pass the Authorize.net payment method, and instead use the saved or stored credit card method built-in to Magento.
Now this isn’t an ideal long-term solution, because you are storing credit card information on your site. It’s not PCI compliant. But it is an emergency way that you can save customer orders, process the card in a virtual terminal later, and then delete the credit card information from the system.
My advice for anyone running a Magento eCommerce store: plan for failure of key systems, and know what you can do quickly as emergency stop-gap measures to not lose sales. This is one example of knowing what to do, and being able to do this quickly, paid off.