Unhappy surprise: Dell servers misreport RAID failures

Today a customer found out to her unhappy surprise that the Dell PERC 6/i Integrated RAID controller (perhaps others) doesn't report RAID status properly, and probably lost a full array.

When a drive fails, the drive light goes red/amber, but so does the front-panel display that shows status of the entire server. Responsible system admins make it a point to eyeball the servers regularly, looking for the "all is well" cool blue indicator.

This front-panel reflects not just RAID status, but everything else -- memory, power supply, fans, etc. -- and is a great early warning system even if SNMP traps or alert emails or whatever aren't working right.

But what it doesn't reflect is the status of virtual drives, and this is seriously bad news.

If a failed drive is inadvertently replaced with the wrong kind, be it SATA instead of SAS (and many RAID controllers talk to both), or with a drive that's size incompatible (just slightly smaller than the old size), the RAID controller will adopt the drives with "Ready" status, turn off the front-panel amber status even though the new drive won't rebuild the array.

It's devastatingly bad design to indicate "all is well" when RAID arrays are in degraded status.

The poor customer who obtains a stack of the wrong drives, dutifully replaces them when failed, and is seduced into thinking that all is well when the lights all go green (on the drives) and blue (on the server) may well lose everything if the wrong two drives fail in succession.

We found a different server that had been limping along with a degraded array and a bad drive:

Thankfully we caught this second server before it was truly gone and were able to replace the bogus drive with a proper one. We've ordered a stack of additional spares just to avoid this in the future.

The moral of the story is that you can't be sure of a RAID rebuild until you can query the controller directly regarding virtual-drive status, either in the BIOS (requires downtime), or with vendor-provided utilities under the host operating system.

Ouch.

Parking at Peters Canyon Regional Park

click for larger image

The Peters Canyon Regional Park in Tustin is one of the most popular hikes in Orange County, offering trails with challenges for all different levels. It's a gem.

The official public entrance is on the north side at 8548 E. Canyon View Ave. in Orange, but most hiking groups meet at the south entrance to avoid having to pay for parking. The south entrance is at the intersection of Peters Canyon Rd and Silverado Terrace in Tustin, though many GPS navigators don't do a very good job of getting you there based on this.

Note: when navigating, be sure to enter Tustin, as there's also a Peters Canyon Rd in Irvine.

But parking at the southern/back entrance is universally problematic, and many hikers become lost or frustrated (or both) when trying to join the group for the first time. I've been to three or four hikes now, and I'm still getting used to it.

This map shows the local streets around the southern trailhead, with details and pitfalls noted:

Peters Canyon Road is rather long, and there's no parking at all on the left side, and it's only allowed on the right side once you pass the reservoir access road. People do receive tickets here.

The residential neighborhoods on the left are gated communities, so there are no parking opportunities there either.

Peters Canyon Elementary School is on the corner of Pioneer Rd and Peters Canyon Rd, and though they have two parking lots, it's not clear they are open to the public (signs say "parking by permit only"). I know some hikers do park here, and if it's on an evening with no school events going on, there may be no harm. If you park here, just walk to the trailhead along the main road.

Further to the south is Cedar Grove Park (run by the City of Tustin), and there don't seem to be any prohibitions about parking here as long as it's not overnight. Though it's possible to walk along the main roads (Pioneer to Peters Canyon), you can also cut through the park and shave off some time.

It's about 3/4 mile from the Cedar Grove parking lot to the trailhead, so if you use a walking app, start it here. It counts, right? Be sure to allow sufficient time to park and get to the hike (especially the first time).

I understand that there is additional parking in the neighborhoods just north of the trailhead, but I haven't done those yet: I'll document them once I have.

Ascending Sierra Peak - my first real hike

Sierra Peak seen from my house

After power-walking in the hills of my neighborhood for months, I joined the HikingOC MeetUp group in March. It's been great for conditioning and even better for getting me out of the house.

From my place in the hills of Yorba Linda (Orange County, California), I can see Sierra Peak, the northernmost summit of the Santa Ana Mountains, which looms over the 91 freeway from the south as you drive into Riverside County. You can't miss it.

Sierra Peak in context

There are a zillion antennas on top, and as a long-time radio geek (amateur radio callsign KA8CMY), it's been calling to me for years.

There are two main paths going up: via Skyline Drive (in Corona) from the east, and via Coal Canyon on the west. Coal Canyon is closer and (I believe) a more difficult hike, so that's what another hiker and I from HikingOC decided to try.

Sadly, the combination of the relatively poor camera in my iPhone 4S along with my terrible photography skills means that these pictures will all be marginal. I'll be consulting the family photo expert for gear suitable for my skill level and intended use.

Coal Canyon used to be a real exit off the 91 freeway, but crossing under the roadway was found to be the only wildlife corridor connecting the Santa Ana Mountains with the Chino Hills to the north, so it was removed from service in 2003 or so. There's still an unmarked freeway exit going eastbound, but it's for construction or emergency traffic only. The freeway-accessible parts are fenced off so the mountain lions don't wander around on the 91.

Heading Up

At 7AM sharp, we met at the Green River Golf Course, the closest car-accessible parking spot, and walked ~1.3 miles the entrance of the park which is just south of the freeway via the wildlife undercrossing. Hikes like this always start early to avoid the heat of the day, at least going up.

The ascent starts up Pipeline Trail to the east along a well-groomed fire/service road, and though it's definitely uphill, it's entirely non-technical and easy footing.

But a bit more than 3 miles into the hike we reach the end of the fire roads and start an unmarked non-vehicle trail. This is where it gets very steep, at times surpassing 45 degrees, and this is difficult, slow climbing. We called it "Holy Crap! Hill", the first of four like this.

Unfortunately, I left my trekking poles at home. Duh. These look like ski poles, and they're really useful for ascent and descent, providing stability and extra oomph going up. Mine are Black Diamond Ultra Distance poles, made of carbon fiber that snap down to fit in a backpack. They're very light and very nice, and I wish I'd had them with me. Nevertheless, their absence was not a show-stopper.

Finally at ~5.5 miles, we reached Leonard Road, a maintained fire road, and from there it was another half mile to Sierra Peak, looping past it to the south and approaching from the east. Getting past the non-vehicle trails onto regular roads was a great relief.

It was here that we saw a rattlesnake sunning himself (herself?) in the middle of the road; these are common sights on the trail.

Rattlesnakes are much more afraid of us than we are of them, and there's really no danger unless you step on one by mistake, so giving them a wide berth means we can all enjoy the trail together. I'd seen one in Chino Hills SP two weeks prior.

Even though it was a hazy day, the views going up were tremendous; I could see the area of my house most of the way up, and the views kept getting better and better. My previous highest hikes had been in Chino Hills State Park (both San Juan Hill and Gilman Peak), but they were like anthills in the sand from 3000 feet.

(click image for more detail)

Heading down

After having arrived at the peak at 10AM and spending a few minutes enjoying the view, we took off and started heading back down. But rather than return the same way, we took the long way down that stuck to fire roads — going downhill on the original path is difficult and dangerous, especially without poles. Besides, we wanted a different view.

Heading southwest along Leonard Road, we got great views to the south of a huge canyon that I don't think is visible from any highway, and it was lovely to see the variety of geology and biology heading down. At times was kinda steep, but nothing too challenging, and we made very good time.

At the southernmost portion of the trail we made an almost 180 degree turn to head north on Carbon Canyon Trail. Had we missed this turn, we'd have been taking Windy Ridge Trail, which leads over by the toll plaza on the 241 freeway. That would have been a long walk home.

It was at this point that I ran out of water. My backpack is a Camelbak Octane 18x, and it has a bladder holding 3 liters of water along with a drinking hose. I typically fill mine with ice and then water, because cold water tastes soooo much better than hot water. But I got a good lesson in rationing.

We had met a couple of obviously experienced hikers up on the peak, taking the same path, and when they passed us it served as a great motivator to speed up a bit to chase the rabbits. In places the descent was of a steepness that made it easier to jog than to walk, so we made great time heading down. It was much easier going down even though it was around 2 miles longer.

Recap

I have a great app on my iPhone for hiking, Walkmeter, and it's just fantastic: It records every hike, times and distances (it says I've hiked 279 miles this year), and it can track GPS points to plot our hike in a map:

click image for more detail

Our 14.12 mile hike took around 5-1/2 hours on the clock, but Walkmeter doesn't count time stopped (resting, viewing, etc.) we had 4:14 of real hiking time.

The elevation at the starting point was around 450 feet above sea level, which puts Sierra Peak's 3045 feet ASL at more than 2500 feet of gain (almost half a mile straight up). Though other peaks in Orange County are higher — Santiago Peak is more than 2000 feet higher — Sierra Peak is considered by many one of the hardest hikes in Orange County due to the difficulty of the terrain. I believe it.

I'd like to do it again sometime, this time taking better notes and photographs to make a real guide to this trail: many portions are not marked well (or at all) and it's easy to go down a wrong turn. Though one is not likely to get truly lost here (visibility is very good), backtracking these inclines would be tiring. Thankfully, we didn't take any wrong turns.

I had a cellphone signal pretty much the whole hike, we're in an essentially urban area, but I'd thought ahead to get a PowerSkin battery case for my phone; this fits like a regular phone case, but it includes a built-in battery that pretty much doubles the available battery time. My battery would have been dead by the time we got back to the cars without it.

I'd been very nervous about doing this hike, but I don't think I was really all that close to my real limits: I could have done a harder hike. In the past I'd had problems with blisters, but I got away scot free on this one. I did crash/relax for most of the rest of the day, and though I'm certainly sore today, I'm ready to do whatever is next.

Big thanks to my hiking partner Diana, and to HikingOC group leader Marlin for her helpful encouragement and advice in advance.

"The Net interprets censorship as damage and routes around it." — John Gilmore

"The Net interprets bad licenses as damage and forks around it." — me.

Space Aliens, Sports Radio, and a Really Good Book

A year ago I would have not bet anything that I'd have come across—and been enthusiastic about—a book by a sports radio host, but stranger things have happened. This is the odd story of how I came to read a fantastic book, one I cannot recommend highly enough.

The story starts a few years ago.

Living in the hills of northeast Orange County California, my crappy clock radio only gets one station, powerhouse KFI AM 640 out of Los Angeles, and though the daytime fare is entertaining enough, it's the nighttime programming that put me over the edge.

All UFOs, All The Time

Overnight, KFI airs syndicated Coast to Coast AM hosted by George Noory (taking over for Art Bell), dedicated to every kind of wierd nonsense you can name: UFOs; remote viewing; conspiracies; numerology; vampires; the afterlife; Atlantis; paranormal reptilians; prophesies. Everything ridiculous.

The Mayans/Dec 2012 were to Coast to Coast AM what Sarah Palin was to Tina Fey.

A recurring guest is one Richard Hoagland, who at first comes across as exceptionally bright and articulate, but it doesn't take long before "NASA is hiding evidence of life on Mars" came oozing out, along with a 'tude of epic condescension, to really ruin the experience. If there's anybody whose life ought to be consigned to whacking off a stallion read the book, it's this guy.

After reaching my limit of Sasquatch, chemtrails, and ghosts, I finally bought myself a new radio that opened up a new world of things to listen to.

Why did it take me years to do this? Duh.

Go Angels!

I spent a coupla days just tuning around to see what's what, but I found to my surprise and delight that local ESPN affiliate (and Angels flagship station) KLAA carries the Angels game live, plus — this is the good part — a replay every midnight.

Woot! I love baseball! It's a great way to drift off to sleep.

I totally got back into the sport of my youth, enjoyed following the team, and became completely ensorceled with then-rookie phenom Mike Trout, who was tearing up the game, unanimously won the American League Rookie of the Year, and—most importantly—shares August 7 with me as a birthday.

For the first time in years, I went to Angel Stadium with some very dear friends, and loved it.

Mike and Mike in the Morning

Having never been even remotely interested in sports radio, I nevertheless woke up to the national show Mike and Mike in the Morning, one of the most popular radio shows in the country, I had never heard of it, but even though I only tuned in because it was playing from the night before, I was quickly captured by it because it's so well done (albeit from 3-7AM Pacific time).

Mike Greenberg is the journalist go Jets, Mike Golic played pro football as a 6'5" defensive tackle go Irish for years, and I could identify with both of them from my high-school days. Greeny would have been in chess club with me, and Golic would have been one of the jocks who depantsed both of us periodically. Ah, the memories :-)

I don't really care for most of sports, but these guys are both pros, and I got to really appreciate the whole "inside baseball" aspect where I could see the subtle and nuanced backstories that had never occurred to me before.

Professional sports is professional. In some other life, these guys would be close friends of mine.

All You Could Ask For

After listening to Mike and Mike for a year, and really coming to respect both Mikes, it turns out that Greeny wrote his first novel, "All You Could Ask For", and he was out on a book tour.

I knew nearly nothing about the book itself, other than it was reported to be "smart, sensitive, and very funny", and that all author proceeds were going to the V Foundation, named for famous basketball coach and broadcaster Jim Valvano, whose life was taken by cancer at age 47 (3 years younger than me).

This has all the hallmarks of an Oprah Book Club selection— the very mention of which makes me want to throw up in my mouth, just a little—but I was drawn to it anyway and bought it for my Kindle.

I loved this book, it having captured my attention from the very first paragraph.

The problem is: describing the book makes it sound like an Oprah Book Club selection, which could not be less appealing to me, having already thrown up in my mouth. Again. But I couldn't put this book down.

It's written entirely in the voice of three unrelated women as they go through their lives, and this is where the smart and funny comes from. It's exceptionally witty and funny.

As a side note, one of the early vignettes includes a very minor bit of computer hacking; as an internet security consultant, I'm used to embarrassing depictions of technology (even ones that would have been easy to get right), but it was absolutely authentic. There's no way that Mike knows how well he got this right. F Larry Bird indeed read the book.

I didn't know it was coming—but should have—but partway through the novel we saw the introduction of cancer into the lives of these woman. As one who's been very, very close to a women with breast cancer (a former spouse), I was just a tad more in tune with this, but looking back I'm not sure it would have made any difference were my life to have been unafflicted with this curse on humanity.

This is the sensitive part, and it was done very, very well. It's not a sports novel, it's not a tech novel. I can't believe I'm writing this.

Unlike many "cause" books, this was utterly non-preachy, and I didn't even know it was a "cause" book until the cause made itself known, and even then it was still nothing but a really great book.

Mike has written other books, but the only one for Kindle (the wonderful Amazon e-reader) is Why My Wife Thinks I'm an Idiot: The Life and Times of a Sportscaster Dad, which was also a wonderful bit of writing. It reminded me of Dave Barry, but for sports.

The only thing that was somewhat unnerving about "All You Could Ask For" was just how well Mike Greenberg captured the female voice (three of them). Either he is an astonishingly good writer, or he has the world's largest collection of shoes in his closet. For a man. Or both.

Not that there's anything wrong with that.

Read this book. Thank you Mike.

Comodo - phone spamming for SSL cert biz

A customer of mine got a phone call from one Pedro Santos of Comodo saying that an SSL cert was coming due, and offering to help renew it. My customer (who didn't realize that this was spam) innocently asked for more information in an email.

I happened to be on the phone with the customer later, who mentioned it, and forwarded me email: Comodo is spamming and attempting to sell via misrepresentation.

From: pedro santos [mailto:pedro.santos@comodo.com] 
Sent: Wednesday, March 27, 2013 2:12 PM
To: (customer)
Subject: SSL Certificate Solutions By Comodo

Hi (customer).

It was nice speaking with you today!

The SSL Certificate on the (hostname) is coming up for expiration shortly.(36 days) 

Currently you are with Thawte on a 1 year DV paying $150 for the year 
  
With Comodo, I can offer you the SAME DV for 1 year for $79.85 or 5 years for $259.75 

Comodo is the largest privately held certificate authority in the world!! 
  
Here is a list of some of our customers:
http://en.wikipedia.org/wiki/Extended_Validation_Certificate
https://secure.wish.org https://www.davidsbridal.com http://drillcomp.com/ https://itunesconnect.apple.com/WebObjects/iTunesConnect.woa
www.utahcentral.com
www.fila.com
www.westernunion.com 

Find out a bit more about our company: 
www.comodo.com www.enterprisessl.com 
  
Click on the link below to take a look at our big named customers.
http://www.enterprisessl.com/ssl-certificate-corporate/ssl-certificate-customers.html 

-- 
Pedro Santos
Account Executive
 
COMODO
 
Direct: 201-620-6731
E-Mail: pedro.santos@comodo.com
www.comodo.com

I'm not a fan of cold calling, but that's how the world works and I'm generally going to politely turn them away, but in this case we have some special sauce:

1. The particular domain had no public links that we know of, it's for internal use, so we're reasonably confident that Comodo is generating leads via automatic scanning for open 443/tcp and sucking down the cert details. This just seems scummy.

2. Pedro claims that we're spending $150/year for our cert, but it's simply not true: we spend around $30/year from a Thawte reseller. Thawte's website does list DV (domain validated) SSL certs for $150/year, but my suspicion is that's just a retail price that nobody pays, them instead relying on resellers to sell at a quarter of the price.

   I find it very, very hard to believe that a salesman in the SSL cert business doesn't know this, and was happy to lie in the hopes of getting a sale.

I replied to the guy, disabused him of the $150/year notion, suggesting that he never call us again.

click for PDF (not from Comodo)

This kind of trolling for SSL business is redolent of registrars who send out invoice-like USMail to domain-name owners hoping to snag the unsuspecting into paying much more for services than they are now.

The example to the right is from DNS Service earlier this year (to the same customer, it turns out)s, who seems to be notorious for this, and I've had to send notes to my customers telling them to throw these things away and not ever countenance this kind of spam.

Now Comodo is doing the same thing. Shame on you. Congratulations, Pedro, now you're famous.

Looks like I'll be sending yet another warning email to my customers. Sigh.

New Tech Tip: Microsoft's confusing "Not enough..." error messages

I've burned a lot of time tracking down the cause of errors from various software, and have found a lot of cases where the message is entirely misleading. The biggest offender is "Not enough storage is available to complete this operation", which sends everybody off to My Computer to check disk space.

But it's not that kind of storage, and as I dug into the error messages I found so many that looked similar, but were not, so I wrote up a Tech Tips that surveys every Windows error message that includes the phrase not enough, along with whatever additional information I could find. I hope to update this as I find more out.

Unixwiz.net Tech Tip: Microsoft's confusing "Not enough..." error messages.

Unhappy surprise: Firebird SQL's backup/restore uses magic words for stdin and stdout

I'm writing this mainly for Google/Bingle; it's not of general interest.

I work on a project that uses the Firebird SQL database extensively, and one of the command-line tools is gbak, which performs backup and restore operations. I (and my customers) use this all the time.

Normally the backup and restore operations are done to explicit filenames (backup CL_123.gdb to CL_123.gbk), but a recent project suggested we wanted to pipe these together (credentials removed for clarity):

gbak -backup CL_123.gdb /dev/stdout | gbak -restore /dev/stdin CL_123.gdb   # FAILS

The problem is that it doesn't work: the restore half is trying to seek on the input, and since seeks aren't allowed on a pipe, it fails with "gbak: ERROR:expected backup description record".

Digging into the source code shows that the literal magic words "stdin" and "stdout" (rather than the Linux-provided /dev-based files) send the process down a different code path that works properly without seeks:

gbak -backup CL_123.gdb stdout | gbak -restore stdin CL_123.gdb   # WORKS

This feature is very thinly documented, and I hope this blog posting helps somebody else save some time. I've filed an issue in the Firebird Tracker asking that the docs be clarified.

Unhappy surprise: PuTTY's PSFTP doesn't reflect batchfile errors in the exit code (+ patch)

Many of my customers use the PuTTY suite of secure remote access tools (mostly on Windows), and I've written about it here:

Tech Tip: Secure Linux/UNIX access with PuTTY and OpenSSH

PuTTY is great, but I made an unnerving discovery this weekend about PSFTP.

PSFTP is the SFTP client, which is sort of like FTP over SSH, and it allows for secure file transfers between the local machine and the remote server. Most of the time it's used interactively by a human, but it also supports a batch file option to run a bunch of commands all as a group, and it's often used to automate repeated file-transfer tasks (say, to upload a file to another server every night).

The unnerving discovery is that though PSFTP produces error messages during batchfile processing, the program's exit code is always "success" even if things went wrong. This means that the calling script has no real way of knowing that it didn't work.

Network or login errors do reflect an error exit status, but once it starts processing the batchfile, it's only going to exit with success status even if batchfile commands fail. Even a missing batchfile still causes a success exit code.

I suppose one way to deal with this is to parse the program's output looking for known error messages, but this is terribly unreliable, and is no substitute for PSFTP properly reflecting success or failure in the exit status: if anything goes wrong, the calling script should know about it so it can take some action (usually to notify somebody to have a look).

Note that the -be command-line parameter, which says to exit batch processing on error, doesn't impact the program's exit code.

This came as a total surprise, and in my application it mattered a lot because the calling script had to know if the file transferred or not. Needing this functionality, I created a patch against PSFTP.C that introduces a -batchfail command-line option that causes errors during batch file processing to exit with error status (if this option is not provided, there's no visible change in PSFTP's behavior).

This has been working well for me in my application, and I've submitted this patch to PuTTY's maintainers for consideration; I'm hoping they either accept it, or at least give some guidance on how I can make it better (and in the spirit of the rest of the code). The patch is against the current production release, v0.62.

psftp-batchfail-patch.txt

I'd sure love feedback among those who try this.

Tech Tip: Deploying Mitel IP Phones in a VLAN/DHCP Environment

I recently burned way too much time implementing a Mitel IP phone system for a customer, and it should not have been this difficult. Doing it the "easy" way - giving each phone a static IP on the same VLAN as the rest of the network - is just sloppy, especially considering that the phones are happy to play well on a more properly segregated network.

Our local support person was a great and helpful guy, but he simply didn't have this information, and my impression was that we were expected to go to Mitel Professional Services for network consulting (presumably for a fee).

Um, no.

So, I wrote this paper to help the next guy accomplish what we did, and maybe the general direction will be useful for other kinds of IP phones: feedback and related experiences are welcome.

Tech Tip: Deploying Mitel IP Phones in a VLAN/DHCP Environment