[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Initial survey discussions
The following are some email conversations so far with some good ideas and
suggestions. Feel free to discuss anything in here.
----------------------------------------------------------------------------
From: virtanen <hvirtane@cc.jyu.fi>
Date: Tue, 9 Jun 1998 20:59:54 +0300 (EET DST)
To: Roger Dingledine <arma@seul.org>
Subject: Re: SEUL end-user survey?
On Thu, 4 Jun 1998, Roger Dingledine wrote:
> Do you happen to know anything about survey techniques? I'm very interested
> in coming up with a large survey that would figure out once and for all what
> end-users wanted in an OS. If it was well-written, I could probably get several
> thousand people to fill it out in the space of a week. That would be absolutely
> invaluable to the Linux community at this stage.
I think that one big problem is that
people don't care or even know so much about OSs but applications!
It is difficult to get knowledge what people wanted in an OS, because they
don't so much realize the whole thing...
The real work with (micro)computers is done nowadays by using huge
applications, which run about the same way on different OSs like macOS,
Win or Linux. I mean for example Wordperfect or Freehand or the like...?
OS is there mainly to combine applications uniformly together...
For example win3 was an attempt to put together some basic applications
like write, paintbrush, calendar, notepad etc, which looked similar and
could combine data from each other. The problem with win3 or win95 is
that the 'applications' which come with the OS itself aren't powerful
enough for real work, but kids' play only. I've never seen anyone to write
something serious with write... all the people buy a real wordprocessor
like word, wordperfect etc. But you can anyway get an idea, what is
possible with the real applications by trying the toy-applications of win.
In that way seul could perhaps be something similar as the idea of win3,
but with *real* applications, which could be used for real work... The
applications should work or look somehow similar so that if you learn the
basics, you could work with any of them on basic level...
So, actually in my opinion we needed knowledge, what people do with their
computers and what they wanted to do. Better not to ask what the people
want in an OS nut what they want in an hd... For example this old macOS
here is OK as long as it doesn't crash and I'm *able to switch easily
between web-browser, wordprocessor, e-mail etc*... the OS is OK as long as
you don't notice it at all!?
>
> I'm looking for perhaps three people to get together and work on it. I've
> identified a couple of other people who might be useful, but I haven't yet
> written the mail describing exactly what I want to see; I'm just probing for
> interest at this point. :)
>
But how to do the survey? In what way do you think to ask the questions?
We might perhaps put together a primary seul with a kernel, gui, and basic
applications like server, web-browser, wordprocessor, drawing-program, ...
what more? Make it an easy to install packet and working as well as -
win95 or better and distribute that free some time and ask then what more
people want...
The installation should perhaps be with using a webbrowser saving files
directly on the hd...
It should be able to make the necessary partitions and installation
just by invoking one command... Install!
There could be for example a system of 'toy-applications' for the people,
who dont have much space on hd, 'real-applications-system for others...
Put them available in various places in web...
I mean that the first distribution for the survey could be done by using
the net. The survey could be done by asking the people who installed the
systems, what they think about it? If they could do with that the same
things they in general want to do with a computer...
> Some URLs to look at:
>
> http://www.linux.or.jp/~pjotr/mc-doc/results.html
> http://aachen.heimat.de/alug/fragebogen2/
>
> a good example of survey/results format (though not our kind of survey):
> http://www.isdmag.com/Surveys/Linux.html
>
> The beginnings of a description of what I want are under the 'Market
> Research' heading of www.seul.org/seg/
>
> Please let me know if this sounds interesting to you.
>
> Thanks,
> --Roger
Just checked URLs. It seems to work somehow. But it seems that we really
need to get people to use linux somehow to get the knowledge what they
want?
virtanen
----------------------------------------------------------------------------
From: Roger Dingledine <arma>
Date: Tue, 09 Jun 1998 20:18:47 EDT
To: virtanen <hvirtane@cc.jyu.fi>
Subject: Re: SEUL end-user survey?
Ok, I'm going to put you on the seul-research list, which is slowly
accumulating people interested in the survey idea. I'm busy for the
next day or two, but then I'll try to start discussion going in the
right direction. Do you mind if I forward your response here to the
list, once it has people on it? (I expect about a half dozen people
on it, currently.)
In message <Pine.SOL.3.96.980609191018.27914A-100000@kanto.cc.jyu.fi>, hvirtane@cc.jyu.fi writes:
>On Thu, 4 Jun 1998, Roger Dingledine wrote:
>
>> Do you happen to know anything about survey techniques? I'm very interested
>> in coming up with a large survey that would figure out once and for all what
>> end-users wanted in an OS. If it was well-written, I could probably get seve
>ral
>> thousand people to fill it out in the space of a week. That would be absolut
>ely
>> invaluable to the Linux community at this stage.
>
>I think that one big problem is that
>people don't care or even know so much about OSs but applications!
>It is difficult to get knowledge what people wanted in an OS, because they
>don't so much realize the whole thing...
True. I guess I misworded that -- the trick is to figure out what questions
we have to ask, and how to ask them, so we can get to the information
we really need. I guess instead of 'OS' I should have said 'operating
environment'. What do they want to do, how do they want to do it, how do
they want it to work, etc.
>The real work with (micro)computers is done nowadays by using huge
>applications, which run about the same way on different OSs like macOS,
>Win or Linux. I mean for example Wordperfect or Freehand or the like...?
>
>OS is there mainly to combine applications uniformly together...
>
>For example win3 was an attempt to put together some basic applications
>like write, paintbrush, calendar, notepad etc, which looked similar and
>could combine data from each other. The problem with win3 or win95 is
>that the 'applications' which come with the OS itself aren't powerful
>enough for real work, but kids' play only. I've never seen anyone to write
>something serious with write... all the people buy a real wordprocessor
>like word, wordperfect etc. But you can anyway get an idea, what is
>possible with the real applications by trying the toy-applications of win.
>
>In that way seul could perhaps be something similar as the idea of win3,
>but with *real* applications, which could be used for real work... The
>applications should work or look somehow similar so that if you learn the
>basics, you could work with any of them on basic level...
>
>So, actually in my opinion we needed knowledge, what people do with their
>computers and what they wanted to do. Better not to ask what the people
>want in an OS nut what they want in an hd... For example this old macOS
>here is OK as long as it doesn't crash and I'm *able to switch easily
>between web-browser, wordprocessor, e-mail etc*... the OS is OK as long as
>you don't notice it at all!?
yes, exactly
>> I'm looking for perhaps three people to get together and work on it. I've
>> identified a couple of other people who might be useful, but I haven't yet
>> written the mail describing exactly what I want to see; I'm just probing for
>> interest at this point. :)
>>
>
>But how to do the survey? In what way do you think to ask the questions?
>
>We might perhaps put together a primary seul with a kernel, gui, and basic
>applications like server, web-browser, wordprocessor, drawing-program, ...
>what more? Make it an easy to install packet and working as well as -
>win95 or better and distribute that free some time and ask then what more
>people want...
There are currently a couple of projects out there that are starting to
work on this. Seul isn't working towards a distribution right now, but
instead we're working closely with projects like Independence and Freelinux
to let them make a distribution happen, while working on advocacy and
research things on the side to help the spread of information.
>The installation should perhaps be with using a webbrowser saving files
>directly on the hd...
>It should be able to make the necessary partitions and installation
>just by invoking one command... Install!
>There could be for example a system of 'toy-applications' for the people,
>who dont have much space on hd, 'real-applications-system for others...
>Put them available in various places in web...
>
>I mean that the first distribution for the survey could be done by using
>the net. The survey could be done by asking the people who installed the
>systems, what they think about it? If they could do with that the same
>things they in general want to do with a computer...
It's true. I think that if we develop a good survey, we can both put
it out now for initial comments, and then try to bundle it with other
linux distributions (hey, we could even tailor it to redhat and convince
redhat to put it on their site, tailor it to debian and convince debian
to put it on their site, etc.) But while we should keep in mind as we're
writing it how we might deploy the survey, we shouldn't focus on that
until we've actually got something started. :)
>> Some URLs to look at:
>>
>> http://www.linux.or.jp/~pjotr/mc-doc/results.html
>> http://aachen.heimat.de/alug/fragebogen2/
>>
>> a good example of survey/results format (though not our kind of survey):
>> http://www.isdmag.com/Surveys/Linux.html
>>
>> The beginnings of a description of what I want are under the 'Market
>> Research' heading of www.seul.org/seg/
>>
>> Please let me know if this sounds interesting to you.
>>
>> Thanks,
>> --Roger
>
>Just checked URLs. It seems to work somehow. But it seems that we really
>need to get people to use linux somehow to get the knowledge what they
>want?
Well, we don't really need to focus on linux entirely -- there is plenty
of information we can get from current windows and mac users about how
successful their os really is at doing what they want.
But that presents a much more difficult problem of survey distribution --
it's really tough to get the survey to the end-users themselves, since they
don't live on newsgroups and monitor mailing lists.
>virtanen
--Roger
----------------------------------------------------------------------------
From: "Karsten M. Self" <kmself@ix.netcom.com>
Date: Sat, 06 Jun 1998 04:23:58 +0000
To: Roger Dingledine <arma@seul.org>
Subject: Re: linux vendor survey
I've attached a draft of the vendor survey I worked up. I haven't
touched it for a month or so. You're welcome to do with it as you
please.
You can also look at my website for the "SAS User Interest Profile" for
other ideas.
The questions you ask depend on the information you're looking for.
I've decided for the time being that:
1). One survey is enough to wrestle with for the moment.
2). I can collect vendor-related information better through informal
interviews -- I'm still learning what the issues are.
If you have access to a university library or bookstore, you should be
able to find a text on social research. Marketing magazines (usually
trades, check your marketing department if you're with a larger firm)
have some information as well, though I'd tend to go with bound texts
rather than "10 survey tips" articles.
Survey questionnaire design is a factor, but it's not the most critical
factor -- distribution and sampling are, particularly if you're looking
to get some sort of "representative response". The problem with any
"voluntary response" type survey is that you'll find your results are
skewed toward the interests/opinions of people most inclined to
respond. The "how are we doing" postcards you'll find in many stores
will almost always paint a negative picture -- if everything's going
well, nobody says anything. A measure such as that is useful as a pulse
check. You find out qualitiative information -- what sorts of things go
wrong -- not quantitative: how do we compare with the Joneses?
Pretty much the same arguments hold to web-based and mail (electronic or
snail) surveys -- those with an interest will respond. I'm using the
current SAS Interest Profile to guage both issues, and measure
quantitative change over time. It's also a propaganda piece -- the fact
that it's there is affecting impressions -- but I'm pretty open about
this, and I intend to report results fairly. I've had about 550 hits to
the survey page and 35 responses, in one month. Main advertising is on
a SAS mail list/newsgroup, and a few Linux-related sites.
Doing in-person surveys at an opportune location (trade show) might be
good if you can swing it. You'll be skewing your responses by the
attendees of the show, naturally.
If you really want to design a robust experiment, you'll want help from
someone who does this sort of thing professionally (I'm not a survey
professional -- I just play one on TV). You probably don't need this
though, and this should work so long as you're candid in interpreting
your results as to the likely informational biases.
General guidelines for formulating questions:
1. Keep the overall survey as short as possible. You're asking someone
for a favor, respect their time. Both my surveys are too long.
2. Web based: put the entire survey on one page. I started filling out
someone's survey which was split across multiple pages -- I had no idea
how long it would be, and quit after three or four pages. If you *have*
to split your survey (and you don't), number pages (page 1 of 500), and
tell the respondant how many questions they've answered/have remaining
on each page. Because each page is submitted independently (usually),
partial responses will really gum up your study.
3. Try to limit free-form questions. Multiple guess is much easier to
interpret. Put a comments section for additional information. One of
the sample survey pages had a neat trick of including all the free-form
("Other") responses in the results page.
4. For preference questions, try to stick to a two or four level scale,
with no "indifferent" measure. I strongly prefer a scale based on four
levels: strong positive, mild positive, mild negative, strong negative
(usually given as agreement or preference). The reason to exclude the
middle is to force the respondant to choose one way or the other. More
levels don't give you higher accuracy (this comes from your sample size
-- it washes out), and dilute the statistical strength of the results.
5. For other categorical response (pick one of, pick all matching),
choose reasonable categories. The "pick one or more" question is harder
to score than you think -- you're really asking a question for each
possible response.
6. Consider scaling quantitative responses (how many, how much). I use
log scales because they cover a lot of ground, and capture significant
detail (as opposed to worrying about pennies in a billion dollar
response). They also simplify and standardize scoring, and reduce
response error (illegible/invalid responses).
7. Think of how you'll validate responses, if at all. I am trying to
collect at least an email, company name, and city/state. I can validate
companies via Chamber of Commerce or Better Business Bureau if necessary
(I did this on a suspicious and very favorable response -- company did
not exist at the stated address).
8. Give some thought to how you plan to process and present results
*before* designing the survey.
9. I ran across a couple of tools, many, many moons ago, which both
created survey forms *and* the response entry screen (for paper-based
surveys). I'd be interested to know whether there's anything out there
right now which might do a similar trick with web-based
survey/response. Otherwise you end up writing your own parsing code for
the CGI or response-mailer data you get. This is my current headache
(I'm trying to learn Perl by doing this project).
10. I can't give you one more tip, or I'll be as bad as the magazines I
was warning you about <g>
Good luck, let me know if there's anything else I can do to help.
----------------------------------------------------------------------------
From: Roger Dingledine <arma>
To: kmself@ix.netcom.com
Subject: Re: linux vendor survey
Date: Wed, 10 Jun 1998 03:45:14 EDT
Thanks for your detailed response. I've contacted several other
people, and now I'm stalling a couple more days for people to get
back to me, while I deal with real-world work and such.
I've started adding interested people to the seul-research list; do
you mind if I add you as well, in case you have any responses to the
discussions? Also, do you mind if I forward the mail you sent me around
to a couple other people?
In message <3578C45E.7DCD490C@ix.netcom.com>, kmself@ix.netcom.com writes:
>1. Keep the overall survey as short as possible. You're asking someone
>for a favor, respect their time. Both my surveys are too long.
This is going to be a good thing to keep in mind. My goal is to make as
complete and comprehensive a survey as possible. But too long a survey
will get fewer responses. Especially because my target audience isn't
necessarily dedicated enough to Linux to be willing to put in the time
to fill out a long survey. Ick.
>3. Try to limit free-form questions. Multiple guess is much easier to
>interpret. Put a comments section for additional information. One of
>the sample survey pages had a neat trick of including all the free-form
>("Other") responses in the results page.
Yes, my goal above all is to make it a quantitative survey. I want to
actually be able to point at statistics, rather than hand-wave about the
various paragraphs of response I get.
>9. I ran across a couple of tools, many, many moons ago, which both
>created survey forms *and* the response entry screen (for paper-based
>surveys). I'd be interested to know whether there's anything out there
>right now which might do a similar trick with web-based
>survey/response. Otherwise you end up writing your own parsing code for
>the CGI or response-mailer data you get. This is my current headache
>(I'm trying to learn Perl by doing this project).
I'll look into this. I know perl well, so this shouldn't be as big an
issue as it is for other people. (Though I'm probably going to try to
pawn off the actual writing of the survey on somebody else. I have way
too much going on right now to do this whole thing myself. :(
>Good luck, let me know if there's anything else I can do to help.
Thanks,
--Roger
----------------------------------------------------------------------------
From: "Pete St. Onge" <pete_st_onge@iname.com>
Date: Fri, 05 Jun 1998 09:37:23 -0400
To: Roger Dingledine <arma@seul.org>
Subject: Re: Survey writing volunteers needed
Roger,
I had a look at the questionaires you mentioned in your letter.
Being an applications programmer myself, I use VB instead of C or C++ to
build user tools (the site I use for this is www.trentu.ca\~erpds ). In
this way, I think I could bring an interesting perspective to the
building of the questionaire.
Please count me in.
Pete
--
Pete St. Onge - McGill U. Limnology - Fun with Ropes & Buckets
pete_st_onge@iname.com http://wwp.mirabilis.com/4322052
----------------------------------------------------------------------------
From: "Pete St. Onge" <pete_st_onge@iname.com>
Date: Fri, 05 Jun 1998 22:38:11 -0400
To: Roger Dingledine <arma@MIT.EDU>
Subject: Re: Survey writing volunteers needed
Roger,
> Excellent. I'm going to give people a couple more days to get back
> to me; so far you're the only real volunteer. I don't really have
> the time to manage this project right now, so I was hoping to find
> somebody interested in leading and a couple people to help out.
I don't mind taking an active role, so long as the work is shared
(I'm in the process of trying to wrap up my M.Sc.).
> Do you have any experience building questionnaires? It would be
> really nice if I could find somebody who actually had knowledge of
> how to word things, so we weren't just winging it.
Writing questionaires, as such, no. However, before coming back to
school to do my M.Sc. I taught operating systems and applications at the
college level. As part of that, I had to build exams, tests,
assignments, etc., so I think that I might have a grasp on that; I also
have several collegues in other disciplines who build questionaires
frequently, so I imagine I might be able to draw on their input.
> I guess if that doesn't happen, we can get a pretty good idea of how
> to set things up from the other questionnaires on the web.
True, but if I can go back to an email Aldo sent on your behalf on
Christmas day, you expressed so well:
"One of the most important jobs of the SEUL project is to provide
convincing evidence and arguments for why each user should make the
change from using Windows to using Linux."
The issue, I think, goes beyond the view from the perspective of
the average Windows user that Linux seems quite complex and difficult to
use. It has to not only show that the software these users need exists,
but also that the ease of use is approximately equivalent to those
applications in Windows
In order to convince users that the change to Linux would be
beneficial, it might be advantageous to approach this from the
perspective of the average Windows user (academic or otherwise),
examining their needs and seeing how these needs can be met within the
framework of the applications available for Linux. The former I can help
quite a bit with, from my experience as a Windows instructor, programmer
and support person. The latter, however, is where I fall short. But
then, that's why people work in teams...
Cheers,
Pete
--
Pete St. Onge - McGill U. Limnology - Fun with Ropes & Buckets
pete_st_onge@iname.com http://wwp.mirabilis.com/4322052
----------------------------------------------------------------------------