Evil software idea: Cancel Roulette

When doing some task on Windows and clicking the ESCape key, I often see dialogs like this:

and today was thinking: some software asks if I want to save, other software asks if I want to cancel, but a software developer who hated her customers could have some real fun with this by varying the messages at random within the same program:

requiring the user to really pay attention each time. There's lots of room for imagination here.

How fun would that be? :-)

Quickbooks Verify Error 196

I use Quickbooks 2017 desktop, and while running a routine backup, the file verify operation generated an "Error 196" and suggested that I re-enter online banking information for a certain credit card account. It's not clear why this was a problem, as I had no issues going online to download transactions on a regular basis.

I was unexcited about fully disconnecting and reconnecting the account from my bank, but after going into the Bank Feed Setting it showed a curious error:

Odd: everything works, but it's somehow unhappy with it (and MyUsername is how I login to my bank).

Looking at the other account with the same bank, the same screen showed myusername - in all lower case - so on a hunch I went to the failing account and changed it from MyUsername to myusername.

Poof: that fixed it. How about that.

And as a curious aside, for some time I had wanted to rename the account to reflect the last four digits of my updated account number (say, from 1234 to 9876) just so it's easy to correlate with account activity, but Quickbooks would just freeze when I tried to edit the name, but after this username fix that started working too.

Mapping iSCSI filesystem mounts on Linux

A customer's installation had some Linux systems and an external drive array, with the array data mounted on Linux via iSCSI, yielding a number of mounted partitions. This installation used the LVM (logical volume manager), which added another layer of abstraction, and it took me a while to sort out where things came from. This post memorializes how I tracked this down.

Implementation notes: I was reverse engineering a fairly old system, so almost everything is dated. In particular, the Linux systems were running CentOS 5.3, which long past being supported. As I write this, current CentOS is around version 7.4.

I have not researched command differences between the old CentOS and anything newer.

Mounted filesystems

Our ultimate goal is to figure out which iSCSI targets are found on the local system, and almost everybody starts with either the "mount" command, which shows only mount points, or "df", which shows mount points plus size information.

# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VOLGR_SYS-VOL_ROOT
                       5078656   2076892   2739620  44% /
/dev/mapper/VOLGR_SYS-VOL_USR
                      30472188   1891600  27007724   7% /usr
/dev/mapper/VOLGR_SYS-VOL_TMP
                      10157368    154348   9478732   2% /tmp
/dev/sda1                93307     27744     60746  32% /boot
/dev/mapper/VOLGR_DATA-VOL_DATABASE
                     304721920 256764784  32228496  89% /var/local/database
/dev/mapper/VOLGR_DATA-VOL_EXPORT
                     633788608 532349916  68724760  89% /export
/dev/mapper/VOLGR_DATA-VOL_HOME
                     101573920   5461968  90869072   6% /home
/dev/sdd1            1922470928  67098676 1757716332   4% /rsync-backups

Some of the above are found on internal hard drives inside the Linux system itself, while others come from the drive array; you can't tell for sure from here (though you could take some guesses), which is why we're reverse engineering it.

Drive array configuration

This depends entirely on the array product: they are using a Promise m210i VTrak array, and this is always the first place to start in order to determine the logical unit numbers (LUNs). Each one is a separate "device" that can be exported to another system.

Though most disk array products can provide data in many forms — NFS, CIFS (windows network sharing), Appletalk, rsync, etc.—but this case uses iSCSI. It's a fairly efficient protocol, but allows each volume to be exported to one system. If further sharing is required, the computer system mounting the data has to export it somehow.

I won't discuss how to create or manage logical devices—consult the product manual for that—we're only looking at logical devices that are already there.

Our example has two logical units:

The two logical devices LD0 and LD1 are exported via iSCSI to a Linux system, and though the size of each device is seen by the client, the name and RAID level are not.

The question now is how these are mapped on the Linux system.

ISCSI block devices

Our example assumes that the disk array is exporting only to a single Linux system, though this is not inherently the case: it's possible (and common) for a disk array device to provide space for many systems, though I believe each logical device can be mounted by only one machine at a time (hence: many clients means many logical devices).

In iSCSI parlance, "initiator" is the Linux system trying to use the data, and "target" is the disk array device. There's an odd naming scheme for devices that we don't really need to get into for our purposes of just mapping what mounts where. These terms—initiator and target—will come up now and then.

The iscsiadm command allows us to query what's mounted where, and it's an odd format that takes a bit of getting used to (some fields removed):

 
# iscsiadm -m session -P 3
iSCSI Transport Class version 2.0-724
iscsiadm version 2.0-868
Target: iqn.1994-12.com.promise.4e.67.54.55.1.0.0.20
    Current Portal: 10.97.1.6:3260,1     ← which array device are we talking to?
    Persistent Portal: 10.97.1.6:3260,1
        ...
        ************************
        Attached SCSI devices:
        ************************
        Host Number: 5  State: running
        scsi5 Channel 00 Id 0 Lun: 0
            Attached scsi disk sdc  State: running   ← array LUN#0 = Linux /dev/sdc
        scsi5 Channel 00 Id 0 Lun: 1
            Attached scsi disk sdd  State: running   ← array LUN#1 = Linux /dev/sdd

The IP addresses show the connection to the disk array, and in a multi-array configuration you can use this to figure out which unit it's connected to.

But the part we care about is the last section, "Attached SCSI drives", and it shows the mapping of each logical unit on the array to the attached drive on the Linux system, and we can marry this to the logical unit information on the array itself.

Array LUN	Array name	Array size	Linux block device
#0	DARY00_00	1 TB, RAID 5	/dev/sdc
#1	DARY00_01	1.82 TB, RAID 0	/dev/sdd

The Linux system will surely have other block devices, and in this case I'd expect to see /dev/sda and /dev/sdb used for internal hard drives. They will have filesystems as well, but they won't involve this iSCSI array.

Block devices are usually partitioned with the fdisk command— which brings back MS-DOS memories—and we can use this to see how each one is divided up. We are only looking at the partitioning, not changing anything, so tread carefully here and exit the tool (with the "q" command) as soon as you have what you need.

# fdisk /dev/sdc
...
Command (m for help): p    ← p=print partition table

Disk /dev/sdc: 1099.5 GB, 1099512152064 bytes
255 heads, 63 sectors/track, 133674 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1   *           1      133674  1073736373+  8e  Linux LVM   ← Our LVM partition

Command (m for help): Command (m for help): q    ← q=quit without saving

For systems not using the logical volume manager, there may well be multiple partitions: sdc1, sdc2, etc., each as a mountable filesystem, but in this case we're obviously using the logical volume manager so we have to dig further.

But, for /dev/sdd the partitioning shows not "Linux LVM", but "Linux", for /dev/sdd1, and we see that in our original "df" listing as the /rsync-backups partition. This is not going through the logical volume manager, and this was in fact set up later with a single (non-RAID) hard drive just to give me a place to park backups.

So, we have now taken care of array LUN#1, with array LUN#0 requiring the LVM deeper dive.

LVM Mapping

The Logical Volume Manager is a powerful system that supports a very flexible storage subsystem, with filesystems spanning drives and many other options that are far beyond the scope of this post. But I'll touch on three concepts we'll need to decode this.

The LVM subsystem has physical volumes (PV), volume groups (VG), and logical volumes (LV), with each having its own set of commands. In every case we are only looking, not changing anything. The goal here is to reverse engineer what's going on.

First is looking for all physical volumes (with uninteresting fields elided):

# pvdisplay
  --- Physical volume ---
  PV Name               /dev/sdc1
  VG Name               VOLGR_DATA
  PV Size               1023.99 GB / not usable 26.68 MB
  ...
   
  --- Physical volume ---
  PV Name               /dev/sdb1
  VG Name               VOLGR_SYS
  PV Size               110.72 GB / not usable 4.59 MB
  ...

This shows the /dev/sdc1 shown in the fdisk command listing, and it's associated with volume group VOLGR_DATA. These names are arbitrarily given, though it's easy enough confuse volume groups with volumes that the notion of "volume group" was helpfully encoded in the name by the person who set this all up.

We also see /dev/sdb1, which is the internal hard drive on the system, not included in the external array, and we do not see anything from /dev/sdd because that device was partitioned directly and outside the LVM system (though still hosted on the array).

We don't see anything for /dev/sda either, and that's because it's the Linux system root drive. We don't care about that for this process.

So, we have a single physical device (/dev/sdd1) associated with the drive array, and it's included in the VOLGR_DATA volume group. Let's look there as well.

# vgdisplay
  --- Volume group ---
  VG Name               VOLGR_DATA  ← hosted on iSCSI array
  VG Size               1023.97 GB
  ...
   
  --- Volume group ---
  VG Name               VOLGR_SYS   ← hosted on internal Linux hard drive
  VG Size               110.72 GB
  ...

A single volume group can be composed of more than one physical volume (say, two hard drives spanned together to form a single volume), but we're not seeing that here: we just have a single volume group we care about (VOLGR_DATA) on the array, with VOLGR_SYS found on the internal hard drives.

A volume group can contain one or more logical volumes, so let's query that now with the lvdisplay command, providing the name of the volume group:

# lvdisplay VOLGR_DATA
  --- Logical volume ---
  LV Name                /dev/VOLGR_DATA/VOL_DATABASE
  VG Name                VOLGR_DATA
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                300.00 GB
  ...
   
  --- Logical volume ---
  LV Name                /dev/VOLGR_DATA/VOL_EXPORT
  VG Name                VOLGR_DATA
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                623.97 GB
  ...
   
  --- Logical volume ---
  LV Name                /dev/VOLGR_DATA/VOL_HOME
  VG Name                VOLGR_DATA
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                100.00 GB
  ...

Now we're seeing some familiar items: each of the logical volumes corresponds to a mounted filesystem on the Linux system, though via a slightly different path name.

A side note: if the # Open count shows zero, then the partition is not mounted.

Each volume group has its own directory (e.g. /dev/VOLGR_DATA/) that contains the logical volumes within, though each is a symbolic link to a common /dev/mapper directory:

# ls -l /dev/VOLGR_DATA/
lrwxrwxrwx  1 root root   35 Jul  6  2017 VOL_DATABASE -> /dev/mapper/VOLGR_DATA-VOL_DATABASE
lrwxrwxrwx  1 root root   33 Jul  6  2017 VOL_EXPORT -> /dev/mapper/VOLGR_DATA-VOL_EXPORT
lrwxrwxrwx  1 root root   31 Jul  6  2017 VOL_HOME -> /dev/mapper/VOLGR_DATA-VOL_HOME

And at this point we pretty much have it: each of the /dev/mapper/ names—each a mashup of the volume group + volume names— corresponds to a mounted filesystem as shown in the original "df" listing.

There are, of course, many more possible permutations of how this can be organized, but for the purposes of reverse engineering a configuration, this ought to be sufficient in most cases.

Tech Tip: Patient's Guide to FOLFOX6 Chemotherapy

Somewhat off topic from the usual technical subjects, I've written a Tech Tip on things I wish I'd known when I started going through chemotherapy for stage 3a colon cancer. Because the doctors believe they got it all in surgery, this is just insurance (i.e., I don't have cancer now), but I've had to discover a lot of things myself that might help others.

This is not medical advice, it's more about practical stuff and things to expect. I hope it makes somebody else's path easier.

Unixwiz.net Tech Tip: Patient's Guide to FOLFOX6 chemotherapy

Over-minimalism in browsers?

I need to run multiple types of browsers on my Windows 10 desktop: some websites won't work in Edge, some things work better in Chrome, and Firefox allows excellent privacy controls (especially the sandboxing of Facebook).

What I'm really tired of is the dramatic over-minimalization of the browser user interface such that I have to turn on the menu bar and the favorites bar for easy access. Is there anybody who doesn't turn those on?

But the worst is how there's almost no direct indication of which browser just from looking at the title bar when I have multiple windows open on my four large monitors.

Yes, you can train your eye to recognize which is which (especially Internet Explorer's "e"), but how hard would it be to put a tiny icon in the title bar with the logo of the browser itself? I feel dumb every time I look for Help » About to figure out where I am.

Somebody thought this was a good idea, but I don't get it.

Edit: Aha, it seems that in Firefox, going to Menu » Customize » (at the bottom) [X] Title Bar will do pretty much what I want. I can't find equivalent settings in the others.

Building a GPS time station with a Raspberry Pi 3

I dove in headfirst to the whole tiny-micro world almost two years ago — Raspberry Pi, BeagleBone, Arduino — and have found all kinds of uses for them around the house and in my office.

One of my favorite is the super-accurate network time server built with a Raspberry Pi 3 and a GPS hat on top. Though we all think of GPS as providing location services, it also provides highly accurate time, and that's what we're extracting with this little setup.

This Tech Tip details how I built my own tiny network time server, I hope it helps others.

• Tech Tip: Building a GPS Time Server with the Raspberry Pi 3

Fortigate: changing switch to interface mode in 5.4+

Fortinet, the makers of Fortigate firewalls, keep changing around how to convert the internal Ethernet ports from switch mode, where all the ports are members of a single interface, to interface mode, where each port is its own interface.

This weekend I spent hours tracking down how to do this on a Fortigate 60E that shipped with 5.4.3, because the procedure changed - again - and even Fortigate support was too busy to help me (!!).

But I figured it out and am writing this for Google.

A hat tip to Cyrill Gremaud whose helpful instructions allowed me to sort it out; his notes didn't exactly apply to this version of firmware, but it was close enough. Thank you, Cyrill.

The general approach has always been to remove all references to the "internal" interface, which makes it available for the shuffling required.

For an out-of-the-box unit, there are only two references, the internal DHCP server plus a firewall rule allowing traffic, and they have to be removed. If you have already configured additional references to the "internal" interface, they have to go too, and it's probably easiest to remove them from the GUI.

On the 60E models, there's no USB management port that I've been used to, so there's no friendly web-type interface via the FortiExplorer app, so as far as I know we have to do this strictly on the CLI via the serial port. This is new to me so maybe I'm missing something easy, but since we're removing the IP address from the internal interface, we can't use network methods to connect.

config system dhcp server
  delete 1
  end

config firewall policy
  delete 1
  end

# Note: some guides say set the IP to 0.0.0.0 but this works too.
# You do have to remove the "lan" role.
config system interface
  edit "internal"
    unset ip
    unset role
    end

# deleting the internal switch does the heavy lifting here
config system virtual-switch
  delete internal
  end

# or whatever parameters you want for network access
config system interface
  edit "internal1"
    set ip 192.168.1.99/24
    set role lan
    set allowaccess ping http https
    end

Now you can connect via the usual network methods and continue your configuration. Unlike previous methods that used "set internal-switch-mode interface", this does not seem to require a reboot to take effect.

This procedure has been tested in September 2017 on Fortigate 60E with 5.4.3 and 5.4.5 firmware.

Windows 7 missing mouse clicks - Solved?

For a few weeks I've had problems with my 64-bit Windows 7 system, where left mouse clicks wouldn't "take" randomly. Sometimes I had to click a few times, sometimes I had to hold the button down, really: maddening. It appeared to be independent of application.

Searching the interwebs showed this was not an uncommon problem, with suggestions pointing to changing USB bus options ("don't turn off to save power"), updating drivers, and the like. None made any difference.

One set of suggestions revolved around changing one's mouse, but it was raised by folks with more exotic, specialty/gaming mice, and since I have a plain vanilla Dell mouse that I've been using for years, I just didn't see how this could be it.

It was; replacing the mouse with one from Microsoft made the problem go away.

FortiClient SSL VPN - connecting before login

A pox on Fortinet for a lousy user interface and hours of wasted time.

When using the SSL VPN to connect a remote system to the main office — which works pretty well — Windows domain credentials are usually cached, so even without a network connection you can login to the computer. This is why even on a corporate domain-joined laptop you can login with your domain credentials but no network connection.

This also works in the VPN scenario: login with cached credentials (no network connection required), and once in you can launch the SSL VPN to connect to the domain controllers. Works fine.

But what happens when a new user takes over a desktop at the remote office? There are no credentials cached, so no login can happen without a connection to the domain controller, but without launching the FortiClient, no connection comes up. A classic chicken-and-egg problem.

The Fortinet knowledge base articles helpfully describe a "Enable VPN before login" option that seems to do exactly what we want, right under File→Settings

Just what I wanted - sign me up!

Wait: where do I click?

Really - it's not there, there's nothing to click, nothing to find, no obvious way to get to the File menu to get to settings,

Uninstalled and reinstalled many times, rebooted many times, tried installing the full package and not just VPN only, tried setting the "fortissl" network connection object as the default connection. Tried an older build of the FortiClient software.

None of this made any difference. Is it a licensing thing? Am I just stupid?

The problem is that I had to launch the FortiClient console as the administrator to get the menus, something that's not documented in the knowledge base, and absolutely not obvious from the client itself.

This is a terrible, terrible user experience.

When running as a non-admin user, I can fully manage VPN connections: define new ones, delete old ones, change configurations, and connect/disconnect to any of them. To me this smells vaguely admin-y, and because it's not asking me for elevation to do any of it, it is not obvious that I'd need to elevate to make additional items appear in the UI.

I do understand that the setting itself requires elevation, because this impacts the state of the machine beyond the current user, but that there is no clue whatsoever in the UI that something is missing, and could be enabled by elevation. What the hell kind of UI is this?

If It were me, I'd probably do something like adding the File: menu but graying it out, so it's obvious there's a "there" there, and — perhaps — an option in settings "Hide File menu for non-admin users". But make the default to expose the options to the user.

At minimum this has to be noted in the Fortinet KB article on the subject; I've created some feedback for them in the hopes that they fix this.

Fortinet: thanks for nothing.

One fix for Quickbooks' enormous QBSearchIndexerError.txt file

This is for Google; it's apparently a common problem, with no results as far as I can see.

The issue: Quickbooks 2014 (and possibly other editions) generates an enormous file, %TEMP%\QBSearchIndexerError.txt, containing reason after reason why something didn't go right building a search index, and this file can easily grow to fill the hard drive (mine today was almost 100GB).

This is lousy error handling by Intuit, which comes as a surprise to exactly nobody.

In my case, the error said:

Exception raised in IndexData
--Exception Data START--
Time: 11/17/2014 4:53:46 PM
Lucene.Net
checksum mismatch in segments file
   at Lucene.Net.Index.SegmentInfos.Read(Directory directory, String segmentFileName)
   at Lucene.Net.Index.IndexFileDeleter..ctor(Directory directory, IndexDeletionPolicy policy, SegmentInfos segmentInfos, StreamWri
ter infoStream, DocumentsWriter docWriter, HashSet`1 synced)
   at Lucene.Net.Index.IndexWriter.Init(Directory d, Analyzer a, Boolean create, IndexDeletionPolicy deletionPolicy, Int32 maxField
Length, IndexingChain indexingChain, IndexCommit commit)
   at Lucene.Net.Index.IndexWriter..ctor(Directory d, Analyzer a, Boolean create, MaxFieldLength mfl)
   at QBSearchIndexer.SearchDataIndexer.get_Writer()
   at QBSearchIndexer.SearchDataIndexer.WriteDocument(String searchIndexData, String searchStoreData)
   at QBSearchIndexer.SearchDataIndexer.AddDocumentToLuceneDataStore()
   at QBSearchIndexer.SearchDataIndexer.IndexData(Object obj)
--Exception Data END--

... a few bazillion times while it ran through my 100M Quickbooks db file, and since this process normally refreshes every 30 minutes, it doesn't take long for this to fill the drive. Ugh.

Running a file verify on the .QBW file didn't make any difference, nor did a rebuild of that same file. Ugh.

For this particular error, the fix was straightforward:

• Close Quickbooks
• Delete the %TEMP%\QBSearchIndexerError.txt file. If the file is in use, look in Task Manager for dbextclr11 and kill it.
• In the directory holding your .QBW file and look for the associated SearchIndex folder (such as STEVE.QBW.SearchIndex); delete all the files found there.
• Launch Quickbooks and nav to Edit:Preferences, then Search → Company Preferences, then click [Update Now]. This ought to rebuild the indexes without error, though it might take a few minutes.
• Verify that the QBSearchIndexerError.txt file isn't out of control.

This worked for me, but there are lots of other kinds of errors that may not have the same easy fix. It may be that turning off indexing may be the only real way to address this, save for running the gauntlet with Qiickbooks support

My environment:

• Quickbooks Pro 2014
• Windows 7 Professional, 64-bit
• Running as a non-admin user (no auto-elevate)
• Single-user QB with the data file on my workstation