Paychoice continues to have a bad time of it: they had a second flurry of attention this week when a customer of one of their licensees noticed a bogus payroll with four unknown employees with checks of around $8,000 each. The company noticed this before any money was transferred.
This is the first evidence Paychoice has seen of the bad guys actually using an Online Employer credential, though of course with logs having zillions of records, it's gotta be hard to really know for sure. It does seem like a stretch to imagine that the bad guy has had these credentials all along and not done anything with them, but I have no idea about the particulars.
Paychoice provided information on these four bogus employees (names plus partial bank account numbers) to its licensees, scanned their databases for these names and found at least one other use. This was caught before money changed hands as well. They've also forced another password change for all their users.
From what I can tell, PAI ("Payroll Associates, Inc.", the Paychoice people) jumped on this right away with substantial and detailed notifications to its customers, plus a lot of actions behind the scenes. Though this didn't so much touch on the areas I've been working on, I have the impression their security response is far better this time: they're learning, making less of the amateur mistakes seen in the past.
Payroll Associates has understandably been somewhat vague about the technical details of what happened, but the pretty clear impression I get is that they had never entirely nailed down the precise mechanism of the original breach late last month, but something about this new event led them to the "smoking gun". I don't know what it was, but assume it was SQL Injection. I could be wrong.
This suggests that the bad guy has had ongoing access to limited data on the OE site (full name, email address, username, password) since the beginning, and it's only now that they finally got the last piece of the puzzle. This suggests that this latest incident is part of the initial ongoing breach, not a second breach. I could be wrong on this too.
They've taken down portions of their Online Employer related to user and password management, so for now everybody has to call the Paychoice support line to get these functions taken care of. I understand they've geared up support staff a bit to handle the extra load, and they're working like mad over the weekend to recode the features for security.
At this point I am engaging in rank speculation, but if the code requires this much work to fix, it was probably not coded properly with security in the first place. An architecture that anticipated the common web threats might have had just a chink in the armor — requiring a narrow patch — but wholesale re-architecting suggests more fundamental flaws (such as passwords in cleartext, perhaps?).
Application programmers are commonly poorly versed in security coding techniques (especially web developers), and I've been previously called in for emergency mitigation of poor website software. It's never a matter of fixing just one bug, but making wholesale changes to the entire application to avoid the shoddy coding altogether.
My Tech Tip on SQL Injection has been consistently #2 or #3 on Google or Bing on the subject: it's a great place to learn about this area of secure web development. Other areas such as least-privilege use on the database — really fine-grained permissions — are also good lessons.
Taking a turn in the story, about a week and a half ago I got frustrated watching a botnet that nobody at Paychoice seemed to care about: they were advising their customers about a different malware threat, and completely ignoring the stolen passwords (A/V cleanup is not enough).
I finally decided to email Paychoice directly, laying out my case, and giving them information they could verify on their own without taking my word for it. This is not about me, but about customers, and if there is a threat they're not addressing, they (and their security consultants) need to know.
They had been telling their licensees that they were powerless to do anything about a botnet, and this was either outright ignorance, or baldface BS. My spidey sense puts this on the heads of the security consultants, not Paychoice. I'm just guessing.
I got a polite thank-you, and soon after that had received some queries from law enforcement asking for more information: of course I provided everything I had, and have been providing ongoing updates as I have uncovered additional technical details (all of them relatively minor).
Sadly, I've not seen any sign of revised guidance about the malware, so either they don't believe me, they think it's not their problem, or they're hoping the problem just goes away.
I am contacted by Paychoice licensees or their customers daily, and am giving pretty much the same advice throughout:
- Though this has been a serious matter, it's not nearly as bad as a whole raft of easy-to-imagine alternative nightmare scenarios could have been. I have no evidence that the bad guy got actual personal information, and there is no reason to have everybody change bank accounts or purchase credit protection services for their customers.
Paychoice licensees should encourage their customers to review payrolls for unknown employees or payrolls that seem larger than usual. I believe that some payroll systems can red-flag payrolls that are some % larger than usual, but I don't know if Paychoice is one of them.
Note: — if a bogus payroll is found, contact your service bureau or Paychoice immediatey!; I'm sure they have mechanisms to research it.
It's been suggested that requiring pre-notification for all ACH transactions is a good idea, and my limited knowledge of payroll supports this. A prenote is a special zero-dollar direct-deposit transaction that tests the path from the payroll company to the employee's bank account: if it bounces, they can fix it (account name, account number, routing number) before it affects a real payroll.
The prenote period is something like 10 days, so the first payroll or two for a new employee has to be done with a physical check: it's not so much the prenote process itself that provides security, but the built-in delay that gives time for somebody — the service bureau, the customer — to notice it.
Some service bureaus do in fact require prenote for everything, though I'm not sure that it's strictly for security reasons. Others have customers who prefer to be 100% direct deposit even for first payrolls to avoid having to deal with any paper checks.
Paychoice has helpfully offered some assistance in performing mass enablement of prenote for customers of licensees who ask (to save the service bureaus from having to do each one by hand), but I don't believe they've taken a position on whether this should be done.
If anybody actually opened that plugin_setup.exe badware referenced by the initial bogus emails, antivirus cleanup is not enough: the badware phoned home all their web-based online passwords to the bad guy, and because it attacked the browser (and not the network connection), SSL would not make any difference. One would have to change every online password used the duration of the infection.
In this respect I believe that Paychoice gave its customers very bad service in not warning against this threat which I know has hit several users. My suspicion is that their "world class" security consultants (SecureWorks) gave them terrible guidance, and that they (PAI or SecureWorks) simply ignored my evidence that it was something other than they had.
Paychoice customers were less safe because of this, and it's the single biggest failure of their security response (as far as I know).
- Most of what makes you safe or unsafe is in your control, and adopting best practices will go a long way to mitigating dangers. Things like running as a limited user (not an admin!) on your own desktop, running multiple antivirus products, and seriously training staff to use their heads: these make a huge difference.
Nothing in this incident really suggests that the Paychoice platform should be avoided. Everybody has issues, and the important measure is how one responds: Paychoice's response really sucked early on, but they're driving the boat properly now, and I'd not be worried about having my payroll data on their servers (with the exception of my password in cleartext - see below).
I have not advised or hinted anybody to leave Paychoice.
I believe that Paychoice users have the ability to see login logs for the Online Employer portal (though perhaps not now, with the functionality being rewritten), but it does not include IP addresses. I understand that PAI will make more detailed logs to their licensees upon request, especially if they have high-profile customers (say, a bank or an internet security company) that need some more detailed reassurance. This is excellent.
I'm also sure that Paychoice has to be bleeding money at consultants, lawyers, and overtime. I suspect lots of late-nite and weekend pizza delivery at PAI HQ for the programmers too.
If I could ask Paychoice two questions, they would be:
- Why have you consistenly ignored the botnet and stolen passwords?
- Do you store Online Employer passwords in cleartext?
To the many in the Paychoice community who've helped me fill in the pieces, thank you; please keep those emails coming.
And to Payroll Associates: If I am wrong in any detail, I will always post prompt corrections. I would rather be correct than right.