Fwd: Re: Initial survey info

[Roger Dingledine <arma@mit.edu>]

Oopsie. I'll go fix that Reply-to line now.

------- Forwarded Message

From: "Karsten M. Self" <kmself@ix.netcom.com>
Date: Sat, 13 Jun 1998 22:50:40 +0000
To: Roger Dingledine <arma@MIT.EDU>
Subject: Re: Initial survey info

This long.  Hope it's worth it.

Contents:
 - About me
 - About SEUL?  (My take)
 - Research methods
 - Misc responses
 

Roger Dingledine wrote:
> 
> Welcome to seul-research. Our current goal right now is to figure out
> what we have to do to write a good survey, and then actually write it.


About Me:

> kmself@ix.netcom.com
>   Author of the SAS survey
>   (http://pw1.netcom.com/~kmself/SAS/sas_user_survey.html) and somebody who
>   is clueful about writing surveys. He'll going to tell us when we're being
>   stupid.

After an intro like that, I have to add:  it's only because I'm a
recognized expert in the field <g>.  More background -- I'm a SAS
programmer, most of my experience is with medical claims data, and
lately signal processing.  I've been associated with marketing
activities and had tangential exposure to some market research and
survey projects, though it's hardly a forte.  OTOH, I try to maintain a
fundamental level of incompetence in everything, and know some of the
more fundamental blunders.

On the programming side, I'm pretty good at SAS, SQL, and shell
scripting.  I'm quasi C-literate (read, don't write), do some fairly
heavy shell & awk scripting.  Familiar with PC and Unix hardware.  Known
Unix for > 10 years, Linux for 1.  RH 4.2 & 5.0.

I'm also relatively familiar with the software quality literature
(Yourdon, Constantine, Lister, DeMarco, McConnell, Boehm, Jones,
others).  

I'm drawing on Larry Constantine's "Constantine on Peopleware" (Yourdon
Press, 1995, Prentice Hall Building, Englewood Cliffs, NJ ISBN
0-13-331976-8, $15.96 US at Amazon:
http://www.amazon.com/exec/obidos/ASIN/0133319768/002-8466517-1066213). 
Chapters 29 - 37 are devoted to user needs, interface, and design.  Book
also has a decent bibliography, a big plus.


About SEUL?
 
> Motivation: why I want to get this survey written.
> 
> I want to figure out what end-users want in an operating system, and I
> want to let everybody else know.

Rephrasing:  how can we configure a Linux install, and bundle additional
applications and utilities, to maximize usefulness for the non-technical
end user.  The ideal would be to make Linux the *only* logical choice
for a non-technical computer user.  (OTOH: could we just sell them
Macs?). 


> >The end-user (as we use the term) is a person who just plain uses Linux, and
> >the applications and accessories produced for it. His main goal is not to use
> >Linux for the purpose of creating more Linux, whether for himself or the
> >public (e.g. hacking around in his config files and scripts, or writing new
> >Linux applications for redistribution, respectively). Thus he is at the "end"
> >of the production line. He wants compiled code, in the form of an intuitive
> >graphical icon on his screen, which he can click on and correctly expect it
> >to do what it seems like it should.

Isn't this really similar to what Caldera and Red Hat are doing now?  I
wouldn't say that RH is quite ready for Joe Sixpack, but Joe College
(and Jonny Hacker) can manage it quite well.  A lot of RH's work is
based on RPM and polishing the UI, which hit on two key areas --
maintenance & interface.  I suspect most of our work will be there.

RH is aiming at a broad market, from ISPs to grandma.  Their product out
of the box tends to assume you'll be throwing up a website and a whole
mess of other stuff.  I've made a suggestion based on Core/Layers for
different configurations of Linux. 

Broadly:  portable, workstation (WS), server (SV). 

More narrowly:  
  WS: gamer/home, SOHO (small/home office), technical/development
  SV: web, file/print/com, enterprise apps, dedicated/embeded systems

More at Freshmeat:
http://www.freshmeat.net/forum?mode=view&item=897189861 


> This can be put to several uses:
> * Developers need a goal: if we can figure out which applications or
>   environments are most important to potential new Linux users, then the
>   developers can focus on those aspects.
> * Advocacy people need a goal: if our survey finds that feature foo is
>   very important to our target users and it's already implemented, then we
>   can tell the Linux advocates to say "Linux has foo!" more often. While
>   the Linux advocacy effort has a lot of zealous members right now, it's
>   very disorganized and most advocates aren't as clueful as they could be
>   about what will actually sell Linux. I want to have something concrete to
>   provide to the Linux community to help education and understanding.



Research Methods:

> Some initial very basic thoughts on format:
> 

> The first part of the survey will be ...

Back to squares.  Is a survey the best way to gather the information you
are interested in?

I don't know what your resources or the geographic distribution of this
group is (I'm assuming global), but I strongly suggest a primary
emphasis on other research methods:

 - Panels -- gather subjects to discuss uses/needs
 - Observed use/lab -- subject(s) use product(s) under observation at a
test lab
 - Observed use/native -- subject(s) use product(s) under observation in
their customary environment.
 - Meta analysis -- review existing research data for own needs


The problem with surveys are that you find out what people think they
do, need, or want.  Getting the survey to the desired target audience
(sampling) is also difficult especially WRT the current project -- how
do you get people who don't know you exist and don't hang out where you
do to tell you what they think of a product they've never used?


Constantine: 

User survey are an obvious tool, but the truth is that most users just
won't take the time to respond to questionnaires, and  those who do
often do so casually, with little attention to needed detail....For most
software developers, beta test sites are a major source of
feedback....[M]ajor software companies [are] build[ing] usability labs
or usability research centers.  ....[P]eople do not behave in the
laboratory like they do when they are on their own turf.  If you want to
know how people work with a particular piece of software, you need to
study them *in context*, [cite: Holtzblatt and Beyer, 1993, "Making
Customer-Centered Design work for Teams," Communications of the ACM
36(10), October].  [Emphasis in original]

<end quote>

He goes on to describe an ideal system:  user at computer with camera
over their shoulder, and a mirror placed so the user's face is reflected
(many cues are facial).  This eliminates many of the biases of a human
observer, who is liable to point out methods and such.


So, a suggestion would be to set up a user monitoring system which might
be incorporated into another setting such as a training center, a
friendly corporate environment, selected home user sites, and/or a
school (college, secondary/high school) facility.  You don't need a
whole mess of users, just a representative range, watching them for
periods ranging from several hours to several months.  

On the HW/SW implementation side, a web cam mounted on the monitor could
track facials, mousing, and keyboarding.  I suspect that there's a
utility (xwd?) which could periodically sample screen images to a file
for recovery later (gives better images than a video, simplifies
configuration).  Hmm... you could also monitor total system load, apps
in use, clock/cpu time by app, uptimes, and a whole bunch of other stuff
on Linux (I love real OSs).   You'd also want to provide for scheduled
submissions of data to a central collection site.  Communications
bandwidth will be high.

A few (5 - 20) of these setups would give you much better feedback than
a thousand survey responses.  Friends, family, and office environments
would provide a long-term profile of a small range of users.  A training
center or drop-in lab site would allow shorter-term study of a wider
range of users.  If anyone has connections to Sun, Xerox, HP, SGI, can
you find out if these folks have done similar usability studies?


The web cam is the only extraordinary bit of HW required.  This is the
sort of thing which could even be set up on a distributed basis like
rc5ds -- though info from more than a few sites would be too much to
wade through.  It could potentially be one of the largest
usage/usability studies on the planet, and could very well be worth
development on its own merits -- this is the sort of information ISVs
kill for.  Ok:  name:  LUser (Linux User).  As a client, an icon (you
are being watched), and options to turn on/off recording, blank specific
client displays from being saved (for confidentiality/privacy),
communications/upload options.  Thoughts?


You'd probably also want comparative information from systems such as
Win95/NT, and Macintosh.  The visual component should be reproducible,
though system info may be harder to get.  

You'd also want to compare reactions/responses under different
configurations as SEUL is modified and developed.  The controls
(Win95/NT, Mac) won't suffer from this, natch.


> The first part of the survey will be gathering information on the person,
> ...profession, age?, and experience with different operating
> systems. 

A survey would still be useful for the following:
 - Easier to process/interpret large(r) number of responses.
 - Ask about features not implemented/desired.

Respondent profile:
 - name, age, gender, user profile (gamer, small/home office (SOHO),
professional, business systems (eg: point-of-sale, warehouse, inventory,
database, customer service system, etc.), company/corporate sector,
computer experience (none, games, job, writing, programming, etc.,
multiple responses possible).


> Then I want to know what they *do* with their computer. Do you do

This is probably the best way to approach the OS issue.  I think we're
less interested in this as an OS than we are in it as a *System*. 
Probably uses by category.

Other issues we'd be interested in:
 - do multiple people use this system?  At same time/ w/o logging out?
 - do you have a computer network? (Opens option for NCs or terminals)
 - is user privacy important (one user can't read another's stuff)
 - is user security important (one user can't change another's
files/settings)
 - do you have access to tech support/skilled users
 - current eval of tech support issues


> Something else to consider is learning what hardware the person is using,
> and whether it's sufficient for each of the above activities.

> We also need to figure out how to word questions about the OS features
> themselves -- most people aren't going to know or care if it's virtual
> memory or not, but they will care if it can handle multitasking correctly.
> But we need to be sure not to bias our questions towards Linux -- if we do
> that, we might as well be surveying people with "Do you prefer an OS that
> works?"

;-)

See the "other issues" -- this gets to some of the security/mulit-user
capabilities of Linux from a capability rather than an implementation
angle.
 



WRT other comments:

virtanen <hvirtane@cc.jyu.fi> wrote:

> For example win3 was an attempt to put together some basic applications
> like write, paintbrush, calendar, notepad etc, which looked similar and
> could combine data from each other. The problem with win3 or win95 is
> that the 'applications' which come with the OS itself aren't powerful
> enough for real work, but kids' play only. I've never seen anyone to write
> something serious with write... all the people buy a real wordprocessor
> like word, wordperfect etc. But you can anyway get an idea, what is
> possible with the real applications by trying the toy-applications of win. 

This is an approach which virtually every integrated desktop environment
has duplicated:  Win3.x/95/NT, Macintosh, Openlook, VUE, CDE, KDE,.... 
Do we really need to write another NOTEPAD.EXE?

If we're looking at providing real applications, the approach would seem
to be bundling suitable powerful, capable commercial or OSS apps, not
writing our own from scratch.  There's a place for lightweight tools for
quick & dirty use, but I'd discourage a development effort around this. 
OTHOH, providing better documentation for existing tools would be a
bonus.

And yes, the perfect OS is like the perfect lover:  never gets in your
way, cleans up after you, is stable, and makes you happy.

> We might perhaps put together a primary seul with a kernel, gui, and basic
> applications like server, web-browser, wordprocessor, drawing-program, ...
> what more? Make it an easy to install packet and working as well as -
> win95 or better and distribute that free some time and ask then what more
> people want... 

Too technical.

Again, WRT to apps, the way I see it, computers are an information,
communications, and entertainment tool.

They manipulate text, data, graphics, and sound.
You want tools to view the above.  (browsers, A/V players)
You want tools to manipulate the above. (editors, spreadsheets,
databases, dialers)
You may want tools to make more tools (compilers, languages)
You want games.

FWIW, thinking about word processors (paper-output based) strike me as
an anachronism.  A weak point of all current major products is that they
are paper centric.  We're finally starting to see an era of documents
which rarely if ever leave the screen, whether HTML, XML, SGML, LaTex,
etc.  If you want to develop a new KA, this is a place to apply some
thought.


 
"Pete St. Onge" <pete_st_onge@iname.com> wrote:

> I checked kmself's
> SAS survey, and it didn't seem to be too long to me. I fear that it
> might be perhaps too short for the info we want to collect, however,
> given the complexity of the applications and our goals.

Which illustrates the double bind of a survey.  You don't know my
numbers:  550+ hits to the survey page.  41 responses.  6 weeks.  This
is actually a pretty good response rate.  My primary advertising channel
is a mailing list/newsgroup with probably 2-5,000 readers, plus LDP,
LWN, and Linux Gazzette spots.  So < 10% of advertising generates a hit,
< 10% of hits generate a response.  I'm getting a very highly
self-selected 1% response rate.  This skews my results heavily to people
interested in the concept I'm promoting.

For my purposes  (advocacy), survey acts more as an indicator of
interest than a source of design concepts, and is better suited.  As a
researcher, you want as long a survey as possible, to gather info, but
people don't like answering nosy (or just plain too many) questions.

For a related ISV study, I've decided that one-on-one, free-form
interviews are a better way to fly.
 

> --Roger

- -- 
Karsten M. Self (kmself@ix.netcom.com)

    What part of "gestalt" don't you understand?
    Welchen Teil von "gestalt" verstehen Sie nicht?

        web:       http://www.netcom.com/~kmself
        SAS/Linux:
http://www.netcom.com/~kmself/SAS/SAS4Linux.html        

  3:46am  up  1:16,  2 users,  load average: 0.00, 0.15, 0.59

------- End of Forwarded Message