I had a backup. Really.

"The Net interprets censorship as damage and routes around it." — John Gilmore

"The Net interprets bad licenses as damage and forks around it." — me.

Unhappy surprise: PuTTY's PSFTP doesn't reflect batchfile errors in the exit code (+ patch)

Many of my customers use the PuTTY suite of secure remote access tools (mostly on Windows), and I've written about it here:

Tech Tip: Secure Linux/UNIX access with PuTTY and OpenSSH

PuTTY is great, but I made an unnerving discovery this weekend about PSFTP.

PSFTP is the SFTP client, which is sort of like FTP over SSH, and it allows for secure file transfers between the local machine and the remote server. Most of the time it's used interactively by a human, but it also supports a batch file option to run a bunch of commands all as a group, and it's often used to automate repeated file-transfer tasks (say, to upload a file to another server every night).

The unnerving discovery is that though PSFTP produces error messages during batchfile processing, the program's exit code is always "success" even if things went wrong. This means that the calling script has no real way of knowing that it didn't work.

Network or login errors do reflect an error exit status, but once it starts processing the batchfile, it's only going to exit with success status even if batchfile commands fail. Even a missing batchfile still causes a success exit code.

I suppose one way to deal with this is to parse the program's output looking for known error messages, but this is terribly unreliable, and is no substitute for PSFTP properly reflecting success or failure in the exit status: if anything goes wrong, the calling script should know about it so it can take some action (usually to notify somebody to have a look).

Note that the -be command-line parameter, which says to exit batch processing on error, doesn't impact the program's exit code.

This came as a total surprise, and in my application it mattered a lot because the calling script had to know if the file transferred or not. Needing this functionality, I created a patch against PSFTP.C that introduces a -batchfail command-line option that causes errors during batch file processing to exit with error status (if this option is not provided, there's no visible change in PSFTP's behavior).

This has been working well for me in my application, and I've submitted this patch to PuTTY's maintainers for consideration; I'm hoping they either accept it, or at least give some guidance on how I can make it better (and in the spirit of the rest of the code). The patch is against the current production release, v0.62.

psftp-batchfail-patch.txt

I'd sure love feedback among those who try this.

Dumb software: CPAN

CPAN is the Comprehensive Perl Archive Network, and it's a huge repository of perl modules covering an enormous array of computing needs. It's an amazing resource and one of the best features of perl.

The client side is the cpan command, which can find and install requested modules, and it knows about and offers to install dependencies:

# cpan
cpan shell -- CPAN exploration and modules installation 
cpan> install HTML::Foo

I'm taking a look at the RT ticketing system, and it has a long list of requirements. This is exactly what CPAN was meant for. But as I installed package after package, it kept trying to install the junoscript-perl package each time. Huh? This package is used for talking with Juniper routers, and is reportedly excellent, but I'd not requested it. Hmmm.

We do use Juniper routers, so perhaps this is stuck in a cache somewhere, so I looked all over the system and killed any reference to junoscript. It kept coming back. WTF?

Several hours later I figured it out: in addition to the shell version, one can launch an install directly on the command line, and I'd been using:

$ cpan install HTML::Foo        surprise!

Well guess what: install is not a keyword on the command line, it's the name of a package or module, and cpan located a package in its directory that provides the install module. This directory is found in the .cpan/sources/modules/02packages.details.txt file, and it contains:

...
INSPEC::BibTeX     0.01 V/VP/VPLA/INSPEC-0.01.tar.gz
INSPEC::Retriever  0.01 V/VP/VPLA/INSPEC-0.01.tar.gz
install           undef J/JU/JUNOS/junoscript-perl-6.4I0.tar.gz
integer            1.00 N/NW/NWCLARK/perl-5.8.6.tar.gz
interface          0.02 S/SW/SWALTERS/interface-0.02.tar.gz
...

So junoscript is providing an install module, and this is what I was asking for. Over and over. For hours. The proper syntax is:

$ cpan HTML::Foo                correct

This was technically my fault — I used the wrong syntax — but this is a clear violation of the Principle of Least Astonishment, and I can't believe how frustrating this was. A bit of google shows that I'm not the only one who's had this problem either.

Why isn't this a keyword, or at least have the install module removed from Junoscript?

Building LIBGD on FreeBSD

While building Thomas Boutell's excellent libgd library (ver 2.0.33) on FreeBSD 4.11, I ran into a problem that Google shows lots of others have had, but without a solution. After configuring the library for the platform, undefined pthread variables are found:

---

$ make
...
/usr/local/bin/bash ./libtool --mode=link gcc  -g -O2   -L/usr/local/lib \
   -L/usr/local/lib -o annotate  annotate.o ./libgd.la -ljpeg -lfreetype -lpng \
   -lz -lm  /usr/local/lib/libiconv.so -Wl,-rpath -Wl,/usr/local/lib
gcc -g -O2 -o .libs/annotate annotate.o /usr/local/lib/libiconv.so \
   -Wl,-rpath -Wl,/usr/local/lib  -L/usr/local/lib ./.libs/libgd.so \
   -ljpeg -lfreetype -lpng -lz -lm -Wl,--rpath -Wl,/usr/local/lib
./.libs/libgd.so: undefined reference to `pthread_mutex_unlock'
./.libs/libgd.so: undefined reference to `pthread_mutex_destroy'
./.libs/libgd.so: undefined reference to `pthread_mutex_lock'
./.libs/libgd.so: undefined reference to `pthread_mutex_init'
*** Error code 1

---

Looking around found no libpthread.a library - huh? - in any of its forms, but it was hard to imagine that this machine simply didn't have this facility.

Several hours of hunting this down showed the problem: pthreads are found in libc_r (the re-entrant C library). Aha! This is enabled with the -pthread flag to gcc.

I typically use a simple wrapper script around ./configure so that I can document my settings, and I just added CFLAGS="-pthread" to the ../configure-gd script:

---

#
# requires LIBPNG
#
#       sh ../configure-gd
#       make
#       make check
#       sudo make install
#

export CFLAGS='-pthread'   # for FreeBSD

exec ./configure \
        --without-x

---

The proper solution is to fix the configure.ac script to check for this properly, but it looks like very ugly territory; many flavors and variants and workarounds. Sadly, not up for that today.

I'm documenting this here mainly so Google will index it and perhaps help some future GD-builder who's been having the same problem.