How I got feedback from 200 people for just $17

Many entrepreneurs are familiar with Amazon Mechanical Turk, a service where real people perform small tasks for equally small pay. Being located in Canada, I’m unable to use the service directly (as it’s only open directly to US users). Thankfully there is a startup called PickFu, which enables startups worldwide to ask a simple A/B question through the Amazon Mechanical Turk API.

For $17 you can get 200 people to take a look at your site, and reply to one question with a single line comment. This is the question I posed:

We recently launched www.anynewbooks.com, a free new book notification service. Based on your first impression, would you sign up for it?

The possible answers were “Yes” and “No”. As well each participant was required to leave a comment briefly explaining what motivated their choice.

Why I think this is great

$17 is very little to pay for a service that enables you to receive first impressions from people who visited your site for the first time ever. Moreover, you get detailed statistics about who voted for what based on their gender, education, income, et cetera. In my case, a few people even signed up for the service (albeit not a high percentage of those who answered that they would do so).

Limits of the method

Before you run out and use this site for all your feedback needs, please note that there are a few limits to this approach:

• Mechanical Turk users are not the general population, and they are paid to visit your site. Depending on your niche, these paid visitors may be quite different from organic traffic.
• Some people will not even visit your site (if such is the case, this will often be clear by their comments).
• Some users will not take the task seriously and post inane comments.
• Users may have a tendency to click on the first answer just to get through the task as quickly as possible. Posing two questions with inverted answers may help counteract this. Update: PickFu takes care of this (see comment by John Li).
• Some users may tell you whatever you want to hear (after all you are paying them, even if it’s indirectly through PickFu).

I’m not giving too much weight to the actual outcome (which is almost equally split in my case, anyhow), but am instead focusing on the few insightful comments I received (plus some details that stick out in the demographic section).

Practical outcome for Any New Books?

I’m happy that I did this little “experiment” (technically it's still running). Despite its limitations and a few unpleasant comments, I learned something about my service through it.

On the sign up form used on Any New Books?, I asked users whether or not they had a Kindle reader. The motivation behind this question was that at some point in the future I may offer Kindle ebook listings in the weekly selection (of new titles) for those who own a Kindle.

Rather surprisingly however, a commenter thought that our whole service was strictly for Kindle users. This comment - coupled with the fact that some users don’t know what a Kindle is - made me realize that the sign up form was more complex than it needed to be. As such I decided to A/B test the sign up page both with and without the Kindle question (note, I didn’t test this point through PickFu). The results were very convincing, so I opted to remove the Kindle question (after all, I can always ask active subscribers what format of ebooks they read at a later stage).

Other people seemed to have a problem with understanding the value offered by the service, when a combination of Google and/or Amazon could also help you discover new books. This tells me that perhaps I should add a F.A.Q. explaining how the service works, what the benefits from using it are (a verbose version of what's on the homepage), and why many early adopters absolutely love the service.

One commenter suggested reducing the number of categories, as some of them are too broad (which can lead to a “Paradox of Choice” for the person signing up). I agree with this point, and reducing the number of categories is something I plan to change as soon as I tackle other pressing points first.

Rather surprisingly there was a person who thought that our prices were too high. This remark is interesting because it brings up something I hadn’t thought about at all. There are people out there (perhaps less technically minded ones) who may assume that we are actually selling books, rather than just bringing new releases to their attention via weekly emails. (Our service is absolutely free.)

Other users questioned the overall look and trustworthiness of the site, which gives me some further points to mull over. Recently I commissioned a professionally designed logo for Any New Books?, which should help boost the site’s look. Additionally I plan to add quality testimonials to the homepage. These things should definitely help improve the general look of the site.

Some people questioned whether there are enough books released (each week) for this service to be useful (to address that point, yes, there definitely are!). Perhaps I should highlight (on the site itself) that every week our team sorts through thousands of books to hand-select the hottest new releases.

Demographic details (or Know Thy Customer)

From the demographic data below (A is Yes, B is No), I noticed a correlation between higher education and interest in the service (not surprising). As well female visitors seemed to be more interested in the service than males. Also, younger crowds were stereotypically less interested in the service (and, according to their comments, in reading books in general). People with incomes between $30-60K were by far the biggest supporters of the service.

Conclusion

I'm taking this test's results with a grain of salt. The comments I received were a neat snapshot into the minds of 200 (sort of) random people, but I don't consider them to be a representative sample of all the potential users that Any New Books? may bring in. Still, I have to say that for a mere $17, the insight I received was money well spent, making this experiment a successful one for sure.