Software engineering researchers can be bozos.
Software engineers love to sit around tables and complain about how
nobody values their work. It's true, too: people will happily pay for
physically tangible resources but always expect software to be cheap
or free, even if it's much more complex and difficult to engineer than
the physical object. So you would think that a
software engineering
conference, of all things, would appreciate the value of software
and be willing to spend a little money to save a lot of time and
effort down the road.
As you can guess by the very existence of this entry, you would be
spitting-in-your-soup wrong. They've saved themselves a pocketful of
change by using the free version of CyberChair (not a great
conference manager even in its commercial version, but that's a story
for another day), but in the process they've bought the program
committee a right-royal headache.
Let's take the bidding interface. I would love to show you some
screen-shots, but I can't do it without violating the confidentiality
of the authors (or my own bids). This makes me sad, because words
cannot do justice to how bad this program is. (I tried to include a
redacted version of the HTML here, but Blogger will not accept a
frameset
tag, more's the pity. But in the process of
generating this I had a look at the page source. Not only do many of
the tags not nest, there are even closing tags with no corresponding
opening tag. Wonderful Perl code, circa 1997.)
The idea is that you bid on all the submitted papers to indicate
whether or not you'd like to review them. The bidding interface
consists of two frames (yes, frames, in 2007). The left
frame is some indifferent text, while the right frame is a long
vertical list of all the papers, consisting of a paper number and a
drop-down where you select your preference. That's right, paper
number: like you know or care.
In the left frame you can select the list of all paper titles, with
or without abstracts. So now you have two lists of 419 entries
each...in two different frames. Guess who is responsible for keeping
them in synch while scrolling through each one independently?
Let's say you've made a dozen bids. You'd hate to lose them because
you accidentally killed your browser or something like that. But is
there a way of saving bids? No, not unless you scroll down another
407 entries to find a save button (which is cleverly disguised under
the name Update Bids, which has exactly the wrong meaning). And if
you do save, you'll have to scroll your way all the way back up. (I
later found that there's another copy of this button at the very top,
which still leaves open a Texas-sized wasteland for the middle 395
papers.)
By the way, be sure to click on the correct button. For reasons that
beggar comprehension, the system also offers a Reset button which,
presumably, restores the defaultsnot, I think, because there is
any good reason for it (there absolutely is not), but solely because
HTML offers a way of doing this. This is brain-dead user-interface
design. And it's about to get worse.
For each paper you can enter a bid, which is an indication of how much
you'd like to review it. There's a special category for papers with
which you have a conflict-of-interest (a standard term-of-art in this
area). Concretely, the bidding categories are that you'd love to
review it (A), that you're willing to do so (B), that you'd rather not
do so (C), or that you're conflicted with the paper (D). Now which
letter would you think goes with Conflict?
As you're looking down the list of all papers (left frame), you might
notice that there are two little columns labeled A and C. These are
the counts of how many people have already bid A or C on that
paper. Let me explain why this is about the stupidest information
to provide. You want people to bid bidding independently of each
other; you don't want someone to look at a paper and think,
Hmm, I could go either way on this; oh look, we've had ten
people bid C and nobody bid A, which suggests nobody really wants to
read this paper...I'd be crazy to get stuck with it, so I'll go with
C. I can't imagine a system better designed to make life
maximally hard for a program chair.
Anyway, after well over two hours of this last night, I submitted my
bids (my hand poised shakily over Update Bids, fearful of what would
happen when I clicked). I then sent mail to Kathi, who is also on the
committee, warning her to look for any other interface than the
list-of-all-papers I'd just been through.
Happily, there is another interface: you can select just an area at a
time and view all the papers in that area. Kathi happily marched off
to use that, little knowing her life would be worse than mine.
First, the list of papers in an area only changes your view in the
left frame. The right frame is still the list of all papers. So if
you pick an area and its papers are paper numbers 11, 17, 89, 103, and
so forth, guess how you enter your bids for those papers?
Second, you can't choose a single bid for all the papers in an area.
That means it's not enough that you don't want to read a single paper
on the Semantic Web; you have to find every paper in the category and
manually decline (C) each of them. (Hmm; perhaps an Ontology would be
able to help with that.)
Third, curiously, the list of papers does not include the A
and C columns. In other words, they didn't reuse code! The good
software engineering practices just keep on coming.
Finally, many papers fall under multiple categories. But if you bid
on paper 11 under one category, it still shows up in sorted numeric
order in all its other categories, without even indicating that you've
already bid on it. (In other words, you would definitely see far
more than 419 papers by the time you're done.)
This is bloody-minded software development in the extreme, a process
of mindless, thoughtless, soulless application of algorithm to data
structure. It is difficult to imagine that a living, breathing
combination of human sinew, blood and fiber could have created an
interface so awful.
The real moral, though, lest we lose sight of it, is the waste of it
all. There are 34 regular program
committee members, 16 people on the Expert-Review Panel
(including Kathi and me), and two program chairs. Suppose each of us
wastes just two hours of our time because of bad software design
(seeing as I believe I've alredy wasted one hour, the odds of this are
fairly high). Some of these people are fancy consultants, so on
average say our time is worth USD 100/hour. That's a collective USD
10,400 just in terms of wasted billable time (never mind the
frustration, the lost goodwill, etc).
That's far more than the cost of a better conference management
package. What does that tell you about what the committee thinks of
our time? I can't wait to meet one of these people and hear them
complain at the conference about the popular unwillingness to pay for
quality software.
Maybe I should send them a bill. Maybe we should all send them bills,
so they would come to their senses.