I had a backup. Really.

Paychoice update

Paychoice continues to have a bad time of it: they had a second flurry of attention this week when a customer of one of their licensees noticed a bogus payroll with four unknown employees with checks of around $8,000 each. The company noticed this before any money was transferred.

Brian Krebs reported this for the Washington Post, and Jai Vijayan did so too for Computerworld.

This is the first evidence Paychoice has seen of the bad guys actually using an Online Employer credential, though of course with logs having zillions of records, it's gotta be hard to really know for sure. It does seem like a stretch to imagine that the bad guy has had these credentials all along and not done anything with them, but I have no idea about the particulars.

Paychoice provided information on these four bogus employees (names plus partial bank account numbers) to its licensees, scanned their databases for these names and found at least one other use. This was caught before money changed hands as well. They've also forced another password change for all their users.

From what I can tell, PAI ("Payroll Associates, Inc.", the Paychoice people) jumped on this right away with substantial and detailed notifications to its customers, plus a lot of actions behind the scenes. Though this didn't so much touch on the areas I've been working on, I have the impression their security response is far better this time: they're learning, making less of the amateur mistakes seen in the past.

Payroll Associates has understandably been somewhat vague about the technical details of what happened, but the pretty clear impression I get is that they had never entirely nailed down the precise mechanism of the original breach late last month, but something about this new event led them to the "smoking gun". I don't know what it was, but assume it was SQL Injection. I could be wrong.

This suggests that the bad guy has had ongoing access to limited data on the OE site (full name, email address, username, password) since the beginning, and it's only now that they finally got the last piece of the puzzle. This suggests that this latest incident is part of the initial ongoing breach, not a second breach. I could be wrong on this too.

They've taken down portions of their Online Employer related to user and password management, so for now everybody has to call the Paychoice support line to get these functions taken care of. I understand they've geared up support staff a bit to handle the extra load, and they're working like mad over the weekend to recode the features for security.

At this point I am engaging in rank speculation, but if the code requires this much work to fix, it was probably not coded properly with security in the first place. An architecture that anticipated the common web threats might have had just a chink in the armor — requiring a narrow patch — but wholesale re-architecting suggests more fundamental flaws (such as passwords in cleartext, perhaps?).

Application programmers are commonly poorly versed in security coding techniques (especially web developers), and I've been previously called in for emergency mitigation of poor website software. It's never a matter of fixing just one bug, but making wholesale changes to the entire application to avoid the shoddy coding altogether.

My Tech Tip on SQL Injection has been consistently #2 or #3 on Google or Bing on the subject: it's a great place to learn about this area of secure web development. Other areas such as least-privilege use on the database — really fine-grained permissions — are also good lessons.

---

Taking a turn in the story, about a week and a half ago I got frustrated watching a botnet that nobody at Paychoice seemed to care about: they were advising their customers about a different malware threat, and completely ignoring the stolen passwords (A/V cleanup is not enough).

I finally decided to email Paychoice directly, laying out my case, and giving them information they could verify on their own without taking my word for it. This is not about me, but about customers, and if there is a threat they're not addressing, they (and their security consultants) need to know.

They had been telling their licensees that they were powerless to do anything about a botnet, and this was either outright ignorance, or baldface BS. My spidey sense puts this on the heads of the security consultants, not Paychoice. I'm just guessing.

I got a polite thank-you, and soon after that had received some queries from law enforcement asking for more information: of course I provided everything I had, and have been providing ongoing updates as I have uncovered additional technical details (all of them relatively minor).

Sadly, I've not seen any sign of revised guidance about the malware, so either they don't believe me, they think it's not their problem, or they're hoping the problem just goes away.

---

I am contacted by Paychoice licensees or their customers daily, and am giving pretty much the same advice throughout:

• Though this has been a serious matter, it's not nearly as bad as a whole raft of easy-to-imagine alternative nightmare scenarios could have been. I have no evidence that the bad guy got actual personal information, and there is no reason to have everybody change bank accounts or purchase credit protection services for their customers.

• Paychoice licensees should encourage their customers to review payrolls for unknown employees or payrolls that seem larger than usual. I believe that some payroll systems can red-flag payrolls that are some % larger than usual, but I don't know if Paychoice is one of them.

  Note: — if a bogus payroll is found, contact your service bureau or Paychoice immediatey!; I'm sure they have mechanisms to research it.

• It's been suggested that requiring pre-notification for all ACH transactions is a good idea, and my limited knowledge of payroll supports this. A prenote is a special zero-dollar direct-deposit transaction that tests the path from the payroll company to the employee's bank account: if it bounces, they can fix it (account name, account number, routing number) before it affects a real payroll.

  The prenote period is something like 10 days, so the first payroll or two for a new employee has to be done with a physical check: it's not so much the prenote process itself that provides security, but the built-in delay that gives time for somebody — the service bureau, the customer — to notice it.

  Some service bureaus do in fact require prenote for everything, though I'm not sure that it's strictly for security reasons. Others have customers who prefer to be 100% direct deposit even for first payrolls to avoid having to deal with any paper checks.

  Paychoice has helpfully offered some assistance in performing mass enablement of prenote for customers of licensees who ask (to save the service bureaus from having to do each one by hand), but I don't believe they've taken a position on whether this should be done.

• If anybody actually opened that plugin_setup.exe badware referenced by the initial bogus emails, antivirus cleanup is not enough: the badware phoned home all their web-based online passwords to the bad guy, and because it attacked the browser (and not the network connection), SSL would not make any difference. One would have to change every online password used the duration of the infection.

  In this respect I believe that Paychoice gave its customers very bad service in not warning against this threat which I know has hit several users. My suspicion is that their "world class" security consultants (SecureWorks) gave them terrible guidance, and that they (PAI or SecureWorks) simply ignored my evidence that it was something other than they had.

  Paychoice customers were less safe because of this, and it's the single biggest failure of their security response (as far as I know).

• Most of what makes you safe or unsafe is in your control, and adopting best practices will go a long way to mitigating dangers. Things like running as a limited user (not an admin!) on your own desktop, running multiple antivirus products, and seriously training staff to use their heads: these make a huge difference.

• Nothing in this incident really suggests that the Paychoice platform should be avoided. Everybody has issues, and the important measure is how one responds: Paychoice's response really sucked early on, but they're driving the boat properly now, and I'd not be worried about having my payroll data on their servers (with the exception of my password in cleartext - see below).

  I have not advised or hinted anybody to leave Paychoice.

I believe that Paychoice users have the ability to see login logs for the Online Employer portal (though perhaps not now, with the functionality being rewritten), but it does not include IP addresses. I understand that PAI will make more detailed logs to their licensees upon request, especially if they have high-profile customers (say, a bank or an internet security company) that need some more detailed reassurance. This is excellent.

I'm also sure that Paychoice has to be bleeding money at consultants, lawyers, and overtime. I suspect lots of late-nite and weekend pizza delivery at PAI HQ for the programmers too.

If I could ask Paychoice two questions, they would be:

1. Why have you consistenly ignored the botnet and stolen passwords?
2. Do you store Online Employer passwords in cleartext?

To the many in the Paychoice community who've helped me fill in the pieces, thank you; please keep those emails coming.

And to Payroll Associates: If I am wrong in any detail, I will always post prompt corrections. I would rather be correct than right.

Payroll Associates doesn't care about its customers

There, I said it.

In the ongoing story of the Online Employer security incident, the story seems to be dying down in the news, and I'm sure that PAI just wants it to go away. Who wouldn't?

So far I've not seen any evidence that any actually sensitive information has been compromised, and I don't have any information whatsoever that suggests that they're running a sloppy or insecure shop. I wouldn't worry if they had my personal information.

But it's sure become abundantly clear that PAI doesn't care about its customers.

The badware that was part of the attack emails is a password-stealing Trojan horse, and if it gets on a customer machine, it steals passwords used to login to websites. Not just Paychoice logins, but presumably everything: PayPal, eBay, your bank, etc.

These stolen passwords are phoned home to a server in Sweden, and it's still operating as I write this. For the last two weeks I've been working to kill this botnet, and so far have gotten DNS for iicon-metal.org killed twice (if the badware can't find the mothership, it can't phone home the stolen booty).

DNS has moved to a new set of nameservers, and so far I haven't been able to hunt that down, nor have I been successful in getting the mothership taken down. I'm about to get some help from colleagues on that front: perhaps we'll get somewhere.

At every step, I've been told that mine are the only reports they're getting.

Why isn't PAI doing this? ? ?

Answer: because they believe that saying "Please use up-to-date antivirus" is taking care of the problem, and if the customers get hacked anyway, it's not PAI's responsibility — at least legally.

Obviously, securing their own infrastructure is vitally important, and I don't doubt that they're doing a good job at it, but I believe they owe it to their customers (and customers of their customers) to address the entire Paychoice ecosystem, not just CYA for themselves.

Remember: if individual employers get hacked, they don't take it out on Paychoice, they take it out on Paychoice licensees, the service bureaus who process their payrolls.

Paychoice has said repeatedly that the investigation is ongoing, they have engaged the authorities, blah blah blah. If they're working this angle to take down this botnet, I haven't seen it. To date, I've not received any contact whatsoever from Payroll Associates.

I'd sure rather believe that they have gotten bad advice from their consultants: perhaps because they have no experience themselves with this kind of security incident response, they relied on their experts. If this is the case, PAI, please ask your experts why they're not doing the right thing by your customers.

I've heard from many Paychoice licensees complaining about the poor information coming from the company, and though I understand the anxiety of not knowing what's going on, that poor customer-service response doesn't make anybody less safe.

But by not aggressively attacking this botnet, their customers are actually less safe. That is what matters, and as far as I can tell, they're doing a terrible job.

Edit - Note that this attack is still ongoing, the mothership is still up, and there is nothing to prevent Paychoice or its experts from re-engaging in this effort to protect customers. They are welcome to all the information I have on this should they care to pick up the baton.

How not to respond to a targeted malware attack

For the last week, I've been working on this Paychoice data breach, and I'm getting a little concerned about how Payroll Associates, Inc. is handling it: they're giving terrible advice to their licensees (and by proxy, the customers/employees of their licensees).

The attack was a realistic email to customers of their Online Employer portal inviting them to download a required update, which was of course badware. It is a password-stealing Trojan, and it phones home the stolen booty to a mother ship located in (at least) Sweden, and reportedly another in Poland. I believe there were several variants.

The badware itself ("plugin_setup.exe") was hosted on servers at Yahoo!, but I was able to get them taken down on Thursday and Friday: I'm not sure why PAI or their security experts (reportedly SecureWorks) weren't able to do this themselves. The Yahoo! Security guys rock.

Opening the fraudulent emails after the Yahoo!-hosted sites were down means you couldn't download the badware — you're safe — but if you did install one of those updates, you are infected and phoning home passwords used in all of your online transactions.

I have been on multiple customer systems this week to clear up infections, and in every case, Symantec/Norton missed it, but the new Microsoft Security Essentials found and cleaned it. MSE had the definitions more than a week ago. Not bad for free, eh?

I've been told that PAI engaged Symantec (in an unknown capacity) to help them with the malware, but I find it hard to imagine how this could happen and still take a week to get their signatures updated, or why somebody (PAI or their security experts) didn't submit this malware to the other A/V vendors immediately. How come some random guy who does it on the third day of the attack was the first that many of these A/V vendors had seen it?

In any case, I believe the advice is to run an antivirus scan and to remove the infection if found. "If you're clean, you're fine".

This is dangerous advice because it's just not true, for two reasons.

First, just this morning was on a customer's system with the latest Norton definitions, and it didn't pick up the infection: only installing Microsoft Security Essentials found it and removed it.

But second, getting clean is not enough: from late last week until this morning, the trojan was phoning home passwords used in online transactions, and we have evidence that this is actively being exploited (he had accessed his eBay account from that system, and that account was compromised). The password-stealing is not limited to just OnlineEmployer: it's going for everything, and will continue to do so as long as the botnet C&C (command and control) mothership is up.

If you have been infected, you must change every password used online before the infection was removed. Period. If OnlineEmployer (or your payroll company) gave you new credentials earlier in the week, assume the bad guy has them: get a new password.

Curiously, one of the "malwares" was actually notepad.exe — harmless — and I suspect the bad guy used it for testing but forgot to put it back. If multiple independent up-to-date A/V scans report nothing, you probably are safe, though I do recommend running more than one to help keep you safe.

Repeat this process for any other online service: eBay, Paypal, Facebook, MySpace, your bank, DSLReports, whatever. If you used a password, change it.

Furthermore, for financial sites, research the login history to see if anybody came from an IP address you don't recognize. If the service doesn't give you a way to do this via the online tool, contact the provider and insist that they research this for all access since last Wednesday.

If the login history shows only access from your own sites, you're probably OK, but you still have to change your password (the bad guy knows it!). But if it shows access from other places, you have to assume that the bad guy rooted around your system and took all the information he could find. For a payroll portal, this would be an identity theft orgy, and evidence of an individual account compromise probably triggers legally-required notifications in many jurisdictions.

Any advice that doesn't include the above precautions is simply ignoring the problem and hoping it will go away, and is irresponsible.

Furthermore, as of 2PM PDT Friday, the mothership in Sweden was still up, accepting connections from infected systems. I don't know what steps PAI or its experts have taken to get these taken down, but it's not obvious that any have. I'm still working this via other avenues to get this addressed.

WANTED: There have been reports of another C&C in Poland: if anybody has information about this, I'd sure love to see it.

Make no mistake, Payroll Associates is a victim here, on the business end of a sophisticated criminal act, and I have always had tremendous sympathy for them. They also positively have their hands full researching what happened and to insure that their own infrastructure is safe. Protecting their own stuff protects their customers.

But they are not the only one facing threats, and I really don't see much evidence of them Doing The Right Thing to proactively and aggressively take care of their customers (as opposed to themselves).

It's very common for companies new to this kind of security nightmare to treat it as mainly a PR problem, especially since I still believe the bad guys didn't actually get the really juicy data from Paychoice directly.

But by not aggressively helping their licensees keep their customers safe, they have shifted the burden of legally-mandated privacy-breach disclosures from themselves onto their customers: "PAI did not send the badware, we didn't open it, we didn't send the passwords to the bad guys: you may have to disclose to your employees/customers, but we don't".

My hero Bruce Schneier would probably call this an "externality": a cost imposed on others that is not a concern to me. I predict that if this happens to customers, Paychoice licensees will asking Paychoice to pay for it (I don't know anything on this front beyond idle chitchat from licensees).

When dealing with this kind of horrible event, you really have to fall all over yourself to keep your customers in the loop — consistent with conducting an investigation — and to make customers feel like they're being taken care of. The worst thing you want is for your customers to have their imagination go wild — it never goes to a good place.

I've called this the warm fuzzy feeling for years, and I haven't gotten that vibe from many Paychoice licensees in the last week.

I have heard nothing from Payroll Associates, though a lot of their licensees are talking to me, but I'd love nothing more than to find out that they have taken far more steps than I've seen, and that I'm just uninformed. We can only hope.

Note: Here, and throughout this incident I am commenting on Paychoice's security response, which is how they handle an incident. I am making absolutely, positively no comment on their actual security as a whole, because I don't have the first bit of information, or even a hint, to provide an assessment (and probably never will). Really - I have no idea.

Furthermore, I've not seen anything that would make me avoid using Paychoice to run my payrolls except for what I perceive as very poor customer service during a security incident.

Disclaimer: I consult to the payroll industry, including to a Paychoice competitor, but this is an independent, unpaid, uncoordinated effort.