I had a backup. Really.

Strange password prompt from Outlook

A customer called in a kind of panic today when everybody started getting this popup when launching Outlook:

Which says (so Bing/Google will pick it up):

This is your first login attempt or your password has expired.
You must change your password before proceeding.

These happened for multiple versions of Outlook on several different operating systems, and these did not correspond with real Windows password expirations.

This turned out to be a toolbar from a VoIP system (which has both Outlook and IE toolbars). This is a branded product, and I've seen references to the same toolbar from Alteva, DSCI, and Broadsoft: I can't tell who is the real vendor.

Vendor, if you're reading this, please consider putting some kind of branding on the dialog box so the purpose is obvious - a bit of searching on the internet shows that more than one person has been surprised by this.

Friedl's rule of tech support: the best support is that which is not needed.

Payroll Associates doesn't care about its customers

There, I said it.

In the ongoing story of the Online Employer security incident, the story seems to be dying down in the news, and I'm sure that PAI just wants it to go away. Who wouldn't?

So far I've not seen any evidence that any actually sensitive information has been compromised, and I don't have any information whatsoever that suggests that they're running a sloppy or insecure shop. I wouldn't worry if they had my personal information.

But it's sure become abundantly clear that PAI doesn't care about its customers.

The badware that was part of the attack emails is a password-stealing Trojan horse, and if it gets on a customer machine, it steals passwords used to login to websites. Not just Paychoice logins, but presumably everything: PayPal, eBay, your bank, etc.

These stolen passwords are phoned home to a server in Sweden, and it's still operating as I write this. For the last two weeks I've been working to kill this botnet, and so far have gotten DNS for iicon-metal.org killed twice (if the badware can't find the mothership, it can't phone home the stolen booty).

DNS has moved to a new set of nameservers, and so far I haven't been able to hunt that down, nor have I been successful in getting the mothership taken down. I'm about to get some help from colleagues on that front: perhaps we'll get somewhere.

At every step, I've been told that mine are the only reports they're getting.

Why isn't PAI doing this? ? ?

Answer: because they believe that saying "Please use up-to-date antivirus" is taking care of the problem, and if the customers get hacked anyway, it's not PAI's responsibility — at least legally.

Obviously, securing their own infrastructure is vitally important, and I don't doubt that they're doing a good job at it, but I believe they owe it to their customers (and customers of their customers) to address the entire Paychoice ecosystem, not just CYA for themselves.

Remember: if individual employers get hacked, they don't take it out on Paychoice, they take it out on Paychoice licensees, the service bureaus who process their payrolls.

Paychoice has said repeatedly that the investigation is ongoing, they have engaged the authorities, blah blah blah. If they're working this angle to take down this botnet, I haven't seen it. To date, I've not received any contact whatsoever from Payroll Associates.

I'd sure rather believe that they have gotten bad advice from their consultants: perhaps because they have no experience themselves with this kind of security incident response, they relied on their experts. If this is the case, PAI, please ask your experts why they're not doing the right thing by your customers.

I've heard from many Paychoice licensees complaining about the poor information coming from the company, and though I understand the anxiety of not knowing what's going on, that poor customer-service response doesn't make anybody less safe.

But by not aggressively attacking this botnet, their customers are actually less safe. That is what matters, and as far as I can tell, they're doing a terrible job.

Edit - Note that this attack is still ongoing, the mothership is still up, and there is nothing to prevent Paychoice or its experts from re-engaging in this effort to protect customers. They are welcome to all the information I have on this should they care to pick up the baton.

How not to respond to a targeted malware attack

For the last week, I've been working on this Paychoice data breach, and I'm getting a little concerned about how Payroll Associates, Inc. is handling it: they're giving terrible advice to their licensees (and by proxy, the customers/employees of their licensees).

The attack was a realistic email to customers of their Online Employer portal inviting them to download a required update, which was of course badware. It is a password-stealing Trojan, and it phones home the stolen booty to a mother ship located in (at least) Sweden, and reportedly another in Poland. I believe there were several variants.

The badware itself ("plugin_setup.exe") was hosted on servers at Yahoo!, but I was able to get them taken down on Thursday and Friday: I'm not sure why PAI or their security experts (reportedly SecureWorks) weren't able to do this themselves. The Yahoo! Security guys rock.

Opening the fraudulent emails after the Yahoo!-hosted sites were down means you couldn't download the badware — you're safe — but if you did install one of those updates, you are infected and phoning home passwords used in all of your online transactions.

I have been on multiple customer systems this week to clear up infections, and in every case, Symantec/Norton missed it, but the new Microsoft Security Essentials found and cleaned it. MSE had the definitions more than a week ago. Not bad for free, eh?

I've been told that PAI engaged Symantec (in an unknown capacity) to help them with the malware, but I find it hard to imagine how this could happen and still take a week to get their signatures updated, or why somebody (PAI or their security experts) didn't submit this malware to the other A/V vendors immediately. How come some random guy who does it on the third day of the attack was the first that many of these A/V vendors had seen it?

In any case, I believe the advice is to run an antivirus scan and to remove the infection if found. "If you're clean, you're fine".

This is dangerous advice because it's just not true, for two reasons.

First, just this morning was on a customer's system with the latest Norton definitions, and it didn't pick up the infection: only installing Microsoft Security Essentials found it and removed it.

But second, getting clean is not enough: from late last week until this morning, the trojan was phoning home passwords used in online transactions, and we have evidence that this is actively being exploited (he had accessed his eBay account from that system, and that account was compromised). The password-stealing is not limited to just OnlineEmployer: it's going for everything, and will continue to do so as long as the botnet C&C (command and control) mothership is up.

If you have been infected, you must change every password used online before the infection was removed. Period. If OnlineEmployer (or your payroll company) gave you new credentials earlier in the week, assume the bad guy has them: get a new password.

Curiously, one of the "malwares" was actually notepad.exe — harmless — and I suspect the bad guy used it for testing but forgot to put it back. If multiple independent up-to-date A/V scans report nothing, you probably are safe, though I do recommend running more than one to help keep you safe.

Repeat this process for any other online service: eBay, Paypal, Facebook, MySpace, your bank, DSLReports, whatever. If you used a password, change it.

Furthermore, for financial sites, research the login history to see if anybody came from an IP address you don't recognize. If the service doesn't give you a way to do this via the online tool, contact the provider and insist that they research this for all access since last Wednesday.

If the login history shows only access from your own sites, you're probably OK, but you still have to change your password (the bad guy knows it!). But if it shows access from other places, you have to assume that the bad guy rooted around your system and took all the information he could find. For a payroll portal, this would be an identity theft orgy, and evidence of an individual account compromise probably triggers legally-required notifications in many jurisdictions.

Any advice that doesn't include the above precautions is simply ignoring the problem and hoping it will go away, and is irresponsible.

Furthermore, as of 2PM PDT Friday, the mothership in Sweden was still up, accepting connections from infected systems. I don't know what steps PAI or its experts have taken to get these taken down, but it's not obvious that any have. I'm still working this via other avenues to get this addressed.

WANTED: There have been reports of another C&C in Poland: if anybody has information about this, I'd sure love to see it.

Make no mistake, Payroll Associates is a victim here, on the business end of a sophisticated criminal act, and I have always had tremendous sympathy for them. They also positively have their hands full researching what happened and to insure that their own infrastructure is safe. Protecting their own stuff protects their customers.

But they are not the only one facing threats, and I really don't see much evidence of them Doing The Right Thing to proactively and aggressively take care of their customers (as opposed to themselves).

It's very common for companies new to this kind of security nightmare to treat it as mainly a PR problem, especially since I still believe the bad guys didn't actually get the really juicy data from Paychoice directly.

But by not aggressively helping their licensees keep their customers safe, they have shifted the burden of legally-mandated privacy-breach disclosures from themselves onto their customers: "PAI did not send the badware, we didn't open it, we didn't send the passwords to the bad guys: you may have to disclose to your employees/customers, but we don't".

My hero Bruce Schneier would probably call this an "externality": a cost imposed on others that is not a concern to me. I predict that if this happens to customers, Paychoice licensees will asking Paychoice to pay for it (I don't know anything on this front beyond idle chitchat from licensees).

When dealing with this kind of horrible event, you really have to fall all over yourself to keep your customers in the loop — consistent with conducting an investigation — and to make customers feel like they're being taken care of. The worst thing you want is for your customers to have their imagination go wild — it never goes to a good place.

I've called this the warm fuzzy feeling for years, and I haven't gotten that vibe from many Paychoice licensees in the last week.

I have heard nothing from Payroll Associates, though a lot of their licensees are talking to me, but I'd love nothing more than to find out that they have taken far more steps than I've seen, and that I'm just uninformed. We can only hope.

Note: Here, and throughout this incident I am commenting on Paychoice's security response, which is how they handle an incident. I am making absolutely, positively no comment on their actual security as a whole, because I don't have the first bit of information, or even a hint, to provide an assessment (and probably never will). Really - I have no idea.

Furthermore, I've not seen anything that would make me avoid using Paychoice to run my payrolls except for what I perceive as very poor customer service during a security incident.

Disclaimer: I consult to the payroll industry, including to a Paychoice competitor, but this is an independent, unpaid, uncoordinated effort.

Security Response: the Paychoice/Online Employer incident

For the last week or so, I've been working on a security incident involving the online web portal of Paychoice, a very large payroll software and service provider in the United States. They provide payroll for many companies directly, plus their software licensees are service bureaus that do the same thing. It's a large operation.

A week ago, users of this web portal — Service bureau users and employer/customer users — received fake emails purporting to be from the Online Employer web-based service, inviting users to download a "required update" to use the online portal. It was badware, of course, and to make the email more believable, they contained the username and partial password of the user. This looked really scary.

I got involved when a customer casually mentioned these difficulties, and I spent the next quite a few days digging in on several angles. This was an independent effort, to date I've had zero contact from the Paychoice people.

First, BIG kudos to the great folks at Yahoo! Security (hi Mark!) who managed to shut down the malware-serving sites within an hour of being reported. The Y! Paranoids rock, and all of my customers are grateful for your fast service to protect users.

Also, serious thanks to my Microsoft Security MVP colleagues, who provided invaluable guidance while handling this incident. They thought of things I'd not have.

Surprisingly, in the end I concluded that this was probably much less serious than we first thought: if the bad guy had real-deal payroll data, it wouldn't have been necessary to go on this well-orchestrated phishing expedition, so I came to the conclusion that the bad guy had only the usernames and partial passwords.

It's still a big deal, of course, and I don't doubt that Paychoice is pressing numerous resources (both with security experts and with law enforcement) to get to the bottom of this. As they should. Godspeed, folks: I know it's been a sucky week for you.

I'll also note that Paychoice kept in more or less regular contact with its licensees: updates daily with what's going on (though never enough information for licensees).

I kept a running log of this incident, which by now is mostly calming down, plus wrote up in more detail my analysis of why this may not be the data-breach catastrophe it first appeared to be.

Brian Krebs reported this story in his Security Fix column in the Washington Post.

New Tech Tip: Configuring Windows 7 for a Limited User Account

Last week, I got a copy of Windows 7 Release Candidate 1 (build 7100) and loaded it on my laptop. Though I help customers with limited Vista issues on occasion, I had let it pass me by on my own equipment. So this means I've never gotten to really play with UAC myself.

Now I have, and I really like it (both Windows 7 and UAC). I'll write more about Windows 7 another time, but I had some bumps in the road getting a true limited user working. I'm a huge fan of least privilege, having run it on all my XP workstations, and I was determined to make it so with Windows 7 as well.

I found there was scant guidance on this, so I created a Tech Tip on how to do it safely. Both new-install and existing-user scenarios are covered, and the instructions have worked well on the several configurations I've been testing in virtual machines and on my laptop.

• Unixwiz.net Tech Tip: Configuring Windows 7 for a Limited User Account

It's clearly a best practice to run as a limited user, and I hope these notes help bring that about (and almost all of this applies to Vista as well).

SBC and Privacy - Huh?

I recently called SBC (née Pac*Bell) to convert my ISDN line into just a regular POTS line. I've not made an ISDN call in years, and I only needed one phone line anyway; no need to be dependent on my trusty Motorola Bitsurfr Pro ISDN TA.

Putting aside the fact that it took hours to find the right person to take the order (and to fix it several days later), I received a confirmation of the service change via US Mail. I was shocked to see my Social Security Number in the address line in the envelope's window (click the thumbnail).

I can't see how the SSN should appear anywhere in a service confirmation order, especially since they ask me for the last four digits when I'm talking to them about it, but maybe there is actually some good reason. It doesn't appear on my monthly bill.

But there is no conceivable reason why it should appear in "cleartext" in a postal address, as if it's somehow contributing to routing through the US Mail. Because it appears inside the address, it's not just a case of private data inadvertently appearing in a window after the page shifted.

Checking the SBC Privacy Policy, they talk generally about how securely they keep my private data, but I don't see how this practice could possibly square with it. I'm going to contact SBC and see what they say about this, but I'm skeptical that I'll find anybody who even understands what this is about. Has anyone else seen this?

Update - It seems that California law explicitly forbids doing what they're doing. Thanks to Kasia for her peerless research skills, Section 1798.85(a)(5) of the Civil Code generally forbids mailing of Social Security numbers, but provides some exceptions that seem generally reasonable. But in any case:

"A social security number that is permitted to be mailed under this section may not be printed, in whole or in part, on a postcard or other mailer not requiring an envelope, or visible on the envelope or without the envelope having been opened."

It's hard to see how they might talk their way out of this one. From what I can tell, this is a self-contained "Steal Steve's Identity Kit".

Update #2 - I called SBC and got a really helpful agent; he found that some previous agent had typed my SSN in the wrong field, which he removed immediately after apologizing and agreeing that it was a bad thing. I don't think I could have asked for a better response, and this suggests a one-time screwup rather than a systemic programming issue and/or sloppy data security practice. Not sure there's really anywhere to go with this.

As an aside, why is ISDN still handled out of the Emerging Products Center? I wonder if Touch Tone is "Emerging" too?

Running as non-admin: Office 2003

I have been trying valiantly to run as a non-Administrator, non-PowerUser on my XP box - this is supporting the Limited User Access (LUA) principle that many of us have been espousing for some time. Only by getting experience as LUA can we seriously make this recommendation for others.

About a month ago, Office 2003 started giving me "Preparing to install..." messages every time I started Word/Excel/Outlook, and it burned about 45 seconds every time even though it appeared to run successfully. This, not surprisingly, got very old.

Google was no particular help, and neither were my contacts at Microsoft (somebody suggested it might be related to an MSN Messenger upgrade, but nothing came of that), so this morning I really dug in with the lovely RegMon (Registry Monitor) tool from the smart guys at SysInternals.

The idea, of course, is to look for access denied entries, but it's not so simple to just open up every registry key to which the user does not have rights. Lots of software anticipates these failures by attempting parallel opens on either

1. the same key but requesting less rights, or
2. a parallel key in HKEY_CURRENT_USER

Searching for Access Denied through the hundreds of thousands of entries in RegMon showed many pairs of failure-then-success accesses, but one key didn't seem to have this pairing:

HKEY_CLASSES_ROOT\Software\Microsoft\MasterAggregatorForIPP\MSDAIPP

I generally expand permissions with a special group ("LUA Hacks") so it documents that the key was opened for this purpose, and now it seems to work just fine.

I have no idea what caused it to break, or why this fixes it, but I can't be the only one having this problem: hopefully this blog entry will help somebody else.

Advisory: USPS Information Disclosure Vulnerability

I've found what appears to be a pervasive, longstanding security issue in a service offered by the US Postal Service, and the details are found in my advisory.

Unixwiz.net Advisory: USPS Information Disclosure Vulnerability

Lame Journalism 101

Well it seems that not everybody at Security Focus should be writing about Security: this article, Linux Kernel Security, Again breathlessly talks about how Linux is "vulnerable" to the venerable fork bomb and takes Linux distributions to tasks for not "doing something" about it.

Said by Jason Miller

---

It's a sad day when an ancient fork bomb attack can still take down most of the latest Linux distributions.

---

Background: In *ix of all kinds, new processes are created with the fork() system call (more or less like CreateProcess() in Win32), and a "fork bomb" is a program (shell script or C code) that does nothing but recursively launch copies of itself over and over. Eventually this fills up the process table, and it's long been an effective way to drop a machine because you have to kill all of the forkbomb processes at the same time (they keep respawning themselves).

It's so easy to do that friends do it to friends all the time. Ha ha ha.

This is a local denial-of-service attack, and all systems provide ways of limiting this behavior: max # of overall processes, per-user limits on process creation, and reservation of the last process slot for root, who presumably is going to run the "shutdown" command to make it all go away. "ulimit" is the admin's friend for this.

Jason chastises Linux vendors for shipping with unreasonably high defaults:

Said by article:

---

Even though a local user should be somewhat trusted, that doesn't mean you should hand them a silver platter with the ability to take down the entire machine.

---

It's not really a silly argument until one looks at the wider picture: this is just a single example of a local resource-exhaustion attack, of which the fork bomb is just one. There is also the "disk space bomb", a "CPU bomb" or a "memory bomb". I can think of at least three more, and surely others can too.

But only the fork bomb warrants this special attention? By Jason's reasoning, shouldn't disk quotas be enabled by default? After all, how hard could it be to pick a reasonable quota that applies to everybody out of the box?

When a user has the ability to consume a resource, he can abuse that resource, and I challenge anyone to come up with a broad set of default options works for everybody in limiting bad stuff but not the good stuff.

This is the classic "security versus usability" tradeoff, but unlike the Windows-insecure-by-default that many of us have noted, resource-exhaustion attacks are second-level attacks that are achieved by users who are already on the box (and "compromise" doesn't mean that you have to flatten your box).

If valid users are doing this, you RTFM, tune the limits and/or kick them off the system, and move on. If launched by an unauthorized user, then you have security issues a lot deeper than running out of process slots.

But there is no set of limits that works for everybody, and leaving them for the admin of the box seems perfectly reasonably to me. He suggests that BSD has picked some reasonable limits, but if he gives me fifteen minutes on his box with a C compiler, I'll drop it. There are lots of resources to exhaust (syslog bomb, anyone?).

I am dramatically more concerned about what unauthorized users are doing on my boxes than I am about the authorized ones, and "local denial of service attacks" are very low on my list of concerns. In more than 20 years of commercial system administration across a wide range of enterprises, I have yet to come across a customer with a hostile local user that required these ameliorations. Not even one.

Justin put this really well:

---

the issue is really only of concern to administrators who run multi-user boxes with shell access in tough environments... and they have probably been more than aware of the various resource limits necessary to keep things running safely, for ages.

---

The author could have written a perfectly useful article that addressed how to deal with local resource tuning - I suspect most *ix users don't even know such things exist - but instead resorted to sensationalism about a legitimate tradeoff that has been made for decades.

It is indeed a sad day: for technology journalism.

What's in a Security Certification?

Those of us who don't have any certifications in the InfoSec industry often glaze over at the acronym soup of certs offered by various groups. Daniel Miessler (who just passed his GSEC) has blogged about the popular ones on several dimensions (difficulty, respect gained, renown, etc.), including his own comments as a holder of some and observer of all. A worthy resource:

Information Security Certifications