Re: Put up yer dukes! It's that laptop user issue again.....

["KimBoo York" <kimboo@watermarkonline.com>]

-----Original Message-----
From: Roger Dingledine <arma@mit.edu>
To: seul-research@seul.org <seul-research@seul.org>
Date: Friday, August 07, 1998 4:51 PM
Subject: Re: Put up yer dukes! It's that laptop user issue again.....


>I'm sorry; I wasn't very clear about how the laptop issue currently stands.
>I should have been more clear.
>
>There are currently 6 user types that we've agreed on. The possibility also
>exists that we will have a *separate question* on the front page of the
>survey, asking people if they also use a laptop. Thus, they will be able to
>check a user-type and also check if they're a laptop user.

Oh! Okay, that works. Mea culpa, I did not understand that was the plan.

>This means that people who checked 'laptop user' on the first page will
have
>an extra section of questions for them on the second page.
>
>Several points:
>
>a) It is imperative that we stick to only two 'sections' for the survey.
The
>   front section must be relatively short, and it must be clear that they
>   aren't answering the actual survey yet. The main section can be pretty
>   long if we want it to be, but people taking the survey need to have a
>   good idea of how long it is, and how much there is left. Essentially,
>   having more than one page is really bad for morale. So we're going to do
>   everything we can to avoid it.

I agree with that. The problem, of course, is getting as much into as little
as possible. So, far, I think the survey is doing pretty well by that yard
stick.  :-)

>b) I still am not convinced that we should have the separate laptop
question
>   on the front section. My argument is simply that I don't see the laptop
>   issue as being any more important than, say, "do you use a word
>   processor?" Other people disagree on this. I will not be convinced by
>   people saying "But trust me, it's important." I will be convinced by
>   people coming up with the questions that should go in the laptop part
>   of the main section. Good questions -> laptop section. No questions ->
>   no laptop section. I think that's pretty fair, right?

Well, I'm not so sure. Let me put it this way: what information BEYOND the
fact that the user utilizes a laptop is the survey trying to get? That is
why there are few laptop specific questions out there, I think. The survey
is trying to gather raw data on user types and their preferences; the fact
that a certain type of user may have a laptop is important, while what they
do with the laptop is more defined by their overall user type. I like your
recommendation above that after selecting a user type, the respondee can
simply check whether they use a laptop. In fact, it might behoove us to
create one simple question, thus:

Choose your prefered computer environment:
[] stand alone tower system/desk system <<whatever, I'm not good on the
jargon here....you know, the big systems that come in big boxes!! :-)  >>
[] workstation
[] laptop
.....and perhaps one or two I can't think of at 8 PM on Friday night (do I
have a life? Of course not....)

We don't need to query what brand, how powerful, what OS, said environment
has...as you pointed out, that is getting into the "do you use a word
processor" type of errata. But I think it would be an elegant way of
gathering useful information. This is why I am not pushing for laptop
specific questions, just the information of whether it is used or not.


>c) Actually, I *was* planning on having people fill out the survey
>   multiple times if they felt they fit into multiple categories.
>   I wasn't very clear on the chaos you were describing, Kimboo --
>   what's wrong with the scenarios you described?

No, that is not what I was describing. What you write makes sense; ie, for
every user def. that respondee fits, they fill out the survey each time. My
concern was with a respondee filling out the survey *once* under several
user definitions. Sounds like, though, my concerns were unfounded.

>   Do you think we can't
>   trust people to be rational and answer questions from different
>   perspectives depending on which user-type they're answering for?

Absolutely I do think we can trust the respondees. But your statement
depends on them answering from *one* perspective at a time, that is, perhaps
filling out the survey twice from different user profiles. My concerns were
addressing skewed data results should the respondees fill out the form
*once* from different perspectives. Like I said, though, it seems my
concerns were unfounded.

>    Or
>   I guess that might skew our survey results? We'll be able to keep
>   track of whether people answered multiple times, so if we want to
>   get fancy we can ask our data questions like "how many soho+sysadmin
>   users care about foo?" But we won't want to do this, because there will
>   be enough noise that that question will probably be meaningless.
>   Anyway, do people think this is a bad solution? How else do we deal
>   with people who *do* fit into multiple categories?

Just the way you planned....have them fill out the survey once for each
profile they fit. I was just brining up the scenario based on my
misunderstanding of laptop as a seventh user definition.

>d) About the 'extra questions for each user-type' that we are supposedly
>   working on. Here is the original plan:
>
>   There will be a set of core questions that are applicable to all user
>   types. For each user type, the *order* of presentation of these
>   questions can be different, to emphasize different concerns and
groupings
>   of issues for that particular user type. In addition, for each user type
>   we might add *new* questions that pertain only to the interests of that
>   user type, to get more detailed answers on eg whether a large business
>   sysadmin is happy with the current Microsoft server solutions.
>
>   However, this means we actually have to come up with those new
questions,
>   and also come up with orderings for questions for each user-type. We
>   might not want to put that much work into it (also, it might not be
>   worth it to do so). Thoughts? If we do decide to skip the re-ordering
>   of questions on a per-user-type basis, then the survey gets much
>   simpler... (this isn't necessarily a good thing).

This is the thing: in all surveys (I am a trained anthropologist -- my BA
was in cultural anthro, which is completely useless except in cases like
this!!  :)  you have to remove the bias of the surveyor as much as
possible -- technically impossible, but if you don't try, people will laugh
at you (no, not really, just a bad college flashback there....). To first
have a selection of user profiles, and then to have the survey 'formatted'
to that user type, you better have a *darn* clear picture of that user type,
based on previous surveys and statistics, or else you are formatting the
survey to what you *think* that user type is like, which is totally biased
and ends up skewing the raw data.

I kind of think that the best solution is to format the survey as broadly,
generically as possible (which is why the inclusion of a "don't care/don't
know" answer to the questions is kind of important), and let the users
themselves identify the important issues within the survey. This survey is
structurally valid in that aspect, since the questions are being ranked 'in
order of importance' to the user. This way, lumps of data from the survey
can be taken and applied both across the board and comparatively (fancy talk
for: "wow, every person who identified as user def. #3 felt that question
#32 was incredibly important, compared to people who responded as user def.
#1, who all felt that question #32 was totally unimportant."). So actually,
a simpler survey might actually be a more accurate one. Just a few thoughts
and suggestions.....


KimBoo.........calling it a night and going out to dance to bad disco (but
hey, the beer is cheap!!)