Thursday, September 9, 2010

How I got feedback from 200 people for just $17

Many entrepreneurs are familiar with Amazon Mechanical Turk, a service where real people perform small tasks for equally small pay. Being located in Canada, I’m unable to use the service directly (as it’s only open directly to US users). Thankfully there is a startup called PickFu, which enables startups worldwide to ask a simple A/B question through the Amazon Mechanical Turk API.

For $17 you can get 200 people to take a look at your site, and reply to one question with a single line comment. This is the question I posed:
We recently launched www.anynewbooks.com, a free new book notification service. Based on your first impression, would you sign up for it?
The possible answers were “Yes” and “No”. As well each participant was required to leave a comment briefly explaining what motivated their choice.

Why I think this is great

$17 is very little to pay for a service that enables you to receive first impressions from people who visited your site for the first time ever. Moreover, you get detailed statistics about who voted for what based on their gender, education, income, et cetera. In my case, a few people even signed up for the service (albeit not a high percentage of those who answered that they would do so).

Limits of the method

Before you run out and use this site for all your feedback needs, please note that there are a few limits to this approach:

  • Mechanical Turk users are not the general population, and they are paid to visit your site. Depending on your niche, these paid visitors may be quite different from organic traffic.
  • Some people will not even visit your site (if such is the case, this will often be clear by their comments).
  • Some users will not take the task seriously and post inane comments.
  • Users may have a tendency to click on the first answer just to get through the task as quickly as possible. Posing two questions with inverted answers may help counteract this. Update: PickFu takes care of this (see comment by John Li).
  • Some users may tell you whatever you want to hear (after all you are paying them, even if it’s indirectly through PickFu).

I’m not giving too much weight to the actual outcome (which is almost equally split in my case, anyhow), but am instead focusing on the few insightful comments I received (plus some details that stick out in the demographic section).

Practical outcome for Any New Books?

I’m happy that I did this little “experiment” (technically it's still running). Despite its limitations and a few unpleasant comments, I learned something about my service through it.

On the sign up form used on Any New Books?, I asked users whether or not they had a Kindle reader. The motivation behind this question was that at some point in the future I may offer Kindle ebook listings in the weekly selection (of new titles) for those who own a Kindle.

Rather surprisingly however, a commenter thought that our whole service was strictly for Kindle users. This comment - coupled with the fact that some users don’t know what a Kindle is - made me realize that the sign up form was more complex than it needed to be. As such I decided to A/B test the sign up page both with and without the Kindle question (note, I didn’t test this point through PickFu). The results were very convincing, so I opted to remove the Kindle question (after all, I can always ask active subscribers what format of ebooks they read at a later stage).

Other people seemed to have a problem with understanding the value offered by the service, when a combination of Google and/or Amazon could also help you discover new books. This tells me that perhaps I should add a F.A.Q. explaining how the service works, what the benefits from using it are (a verbose version of what's on the homepage), and why many early adopters absolutely love the service.

One commenter suggested reducing the number of categories, as some of them are too broad (which can lead to a “Paradox of Choice” for the person signing up). I agree with this point, and reducing the number of categories is something I plan to change as soon as I tackle other pressing points first.

Rather surprisingly there was a person who thought that our prices were too high. This remark is interesting because it brings up something I hadn’t thought about at all. There are people out there (perhaps less technically minded ones) who may assume that we are actually selling books, rather than just bringing new releases to their attention via weekly emails. (Our service is absolutely free.)

Other users questioned the overall look and trustworthiness of the site, which gives me some further points to mull over. Recently I commissioned a professionally designed logo for Any New Books?, which should help boost the site’s look. Additionally I plan to add quality testimonials to the homepage. These things should definitely help improve the general look of the site.

Some people questioned whether there are enough books released (each week) for this service to be useful (to address that point, yes, there definitely are!). Perhaps I should highlight (on the site itself) that every week our team sorts through thousands of books to hand-select the hottest new releases.

Demographic details (or Know Thy Customer)

From the demographic data below (A is Yes, B is No), I noticed a correlation between higher education and interest in the service (not surprising). As well female visitors seemed to be more interested in the service than males. Also, younger crowds were stereotypically less interested in the service (and, according to their comments, in reading books in general). People with incomes between $30-60K were by far the biggest supporters of the service.


Conclusion

I'm taking this test's results with a grain of salt. The comments I received were a neat snapshot into the minds of 200 (sort of) random people, but I don't consider them to be a representative sample of all the potential users that Any New Books? may bring in. Still, I have to say that for a mere $17, the insight I received was money well spent, making this experiment a successful one for sure.

11 comments:

John Li said...

Hi Antonio,

I'm one of the creators of PickFu. Thanks for sharing your experience with our service - we love hearing that something we built for ourselves is useful for others too.

One clarification on your "Limits of the method" section - we actually randomly swap the A/B choices when showing them to responders, so responders who just click on the first response will be evenly balanced across both choices. We're also working on ways to better filter the inane comments.

Thanks again for using PickFu!

Cheers,
John

Antonio Cangiano said...

Thanks for stopping by, John. I updated my post in light of your comment. It's good to know that you already take care of that issue.

Maintenance Man said...

So if your service is free, how are you going to make any money? Or is that secret for now? I would want to know.

Antonio Cangiano said...

We plan to make money through the Amazon Associates program, as well as inconspicuous (read non-obnoxious) advertisement within the emails at a later stage.

Cassie Rice said...

Just wanted to let you know, a great way to ask the same question is by using Pop Feedback. On your sign-up page, you can ask visitors that very question and get responses instantly. Which means, you'll be able to get responses from people that are much closer to your target market. Hope this helps. :)

Ben Congleton said...

Hi Antonio,
I'd take your feedback approach one step further and add a tool like http://www.olark.com to your website. This way you can talk to some of your organic visitors, answer their questions, and than use this knowledge to iterate.

(I'd argue that any organic traffic is better than a random turker, but sounds like you were able to get some great feedback for a small investment.)

-Ben

Antonio Cangiano said...

Thank you, Cassie and Ben. It's a good suggestion. :)

Emeric said...

Well, I just erased my first initial comment on how effectively the site captured my email based on a first impression then lost it when I tried to get back to the homepage. Here is that part as I remember it (I hadn't read anything about the site on the site, but it was a review of a few hundred words that sent me this direction. And it was the ease of selecting 4 genres and submitting my email address that made it work. That's effectiveness) That still holds true, but I suggest adding a "Home" button on this page making easier and clearer how to get back to the opening pages. The link below the About sidebar is too nondescript. Can't wait to see what you suggest I read.

Emeric

Antonio Cangiano said...

Thank you for your feedback, Emeric. I added SITE HOME and BLOG HOME links at the top because of your suggestion.

jamie dalgetty said...

How long did it take you to get your results? I posted a question yesterday and while its already been over 24 hours I've only received about 1/4 of the responses I paid for.

I like the results (not complaining) but I was expecting a faster turn around time.

Antonio Cangiano said...

Hi Jamie,

sorry to hear about the delay you experienced. In my case I believe it was completed within a couple of days.

Post a Comment