Laying some ground rules for Software Recommendations

Question

Proposal: Software Recommendations

We're going to let this site go ahead, and see what you all can do with it. While we discussed the topic, we had to take some time to identify our chief concerns with recommendation & shopping type questions. The number one problem that we agreed upon was pretty simple:

They tend to overrun a site.

This entire site is dedicated to this type of question, so this is obviously a moot concern. With that said, the second chief concern was quality - so we do want to lay down some ground rules when it comes to asking and answering this type of question - and then close with something everyone might want to think about. We're honestly excited to see where this goes, and will support you to the best of our ability.

Asking recommendation questions

It's important to keep your questions as narrowly-scoped and specific as possible, because this is what's going to steer the types of answers that you receive. This is best illustrated by example, starting with titles:

What's the best IDE for Python?

This will be closed immediately, as it does not:

• Explain in detail what you hope to get out of an IDE in terms of features, license or cost
• List any applicable frameworks that you might be using, or other possible compatibility quirks with your workflow
• Indicate any that you had previously tried, and why you didn't like them
• Do anything else to encourage answers to be as specific as possible
• Obviously, 'best' without context is never going to work - in fact please try to just avoid the word in titles. If you ask about an IDE for Python on Linux that has (features), we're pretty certain people aren't going to recommend the worst.

What Python IDEs support Django?

This is a much better question, as you've really narrowed the possible answers with some additional criteria. If you can add more criteria in the question body, then you've got the makings of something that is useful, and very unlikely to receive a bunch of mostly link answers.

What Windows Python IDE supports Django and ships with a pink color theme?

... I think you get the point, more specifics means a better question. Ideally, questions will on average have only 5 or so answers that actually address the question constraints to the best of their ability.

You must also be prepared for answers that simply inform you that no single thing meets your needs, but goes on to recommend things that cover what you've identified to be the most important. Hence, make sure you're clear about what you must have out of something, and what would simply be nice to have but not absolutely essential.

Good questions describe specific needs, contain bullet lists of constraints and inform anyone that cares to answer as to which constraints are the most important.

When it comes to libraries, show your ideal use of a prospective library. What features should it provide? What kind of interface would you like in ideal circumstances? Does it need to be able to compile even under certain compiler settings, or platforms? Do you have a size limit in lines of code, allocated memory, or both? Just like in the IDE example above, it's very helpful when you indicate how important these constraints are.

Answering recommendation questions

We need to put just as much work into answering these questions as folks put into asking them. Answers that contain little more than a link will be deleted without question, comment or other ceremony. Additionally, our policy on excessive self-promotion will be even more heavily enforced on the site. If you're asked to stop promoting a product that you're affiliated with by a moderator or community manager - you need to stop, or you'll likely be asked to leave the site. Spam filters will be adjusted accordingly - so be careful.

If you work on a product, be it free, gratis or proprietary - it's fine to recommend it to people opportunistically provided that you include full disclosure of your involvement with the product. If more than a small percentage of your posts mentions your product, we're probably going to need to have a talk.

Good answers on this site will be in the form of honest testimonials that share first hand experience with something and why it meets the needs of the asker.

You should answer questions when you have first hand knowledge of things that satisfies at least most of the constraints given in the question. You should be ready to talk about your experience with it, what you liked about it, what features it does (or perhaps doesn't) have, within the context of the question. You should also talk about any potential quirks that you can think of, and even things that you found to be a down side of the product. In short, share your experience with something, not simply your knowledge that it exists.

Answers that do not even come close to meeting the constraints described in the question should be flagged as not an answer, and will be removed quickly.

Something to think about - what if this works?

We wouldn't be supporting this proposal if we didn't think it had a chance of working. Looking at the users involved so far, and some of the better example questions, we think if any community has a shot at making this work it's this community. However ...

What if it works? What if you finally get these types of questions down to a science? If Stack Overflow decides to relax just a little when it comes to these types of questions, well - you get the drift, and need to keep it in mind. The ultimate form of success for this community might ultimately be rejoining larger, established sites.

It's way too early to say, and I've probably portrayed this as much more complicated than it actually needs to be. Just focus on the quality, favor depth over breadth during your private beta and get active asap on your meta site once it becomes available. We will be watching, we will be helping to the best of our ability with advice and guidance, and we're going to make sure you get a fair shot.

I'll now open the floor for anything additional anyone would like to add (just write an answer) - and we'll see you when the beta is ready :)

Answer 1

This has probably already been stated, but I would like to make it clear that this site should not be questions in the form of:

What do you think about software package XYZ?

Folks that want that can simply go to product review sites.

To me, this site is about people who know the features or concepts they are looking for but don't know the names of existing products or don't have the technology-specific vocabulary to get there. It helps ease someone into a new technology genre, not provide analysis of a software package they have already chosen.

Note that this does not preclude questions in the form of:

I am looking at software package ABC to do items:

1. some task
2. some other task
3. some other task again

Does software package ABC perform these tasks?

Just my thoughts. Comments?

Answer 2

They tend to overrun a site.

I beg to differ.

Until a few months ago, Stack Overflow did not have a specific policy against recommendation questions. (There is now a specific close reason which declares them systematically “off-topic” while pointing to a meta thread that explains how to write a good recommendation question.) Yet was not overwhelmed by recommendation questions.

Super User has long had a policy against “shopping questions”. The policy was in fact applied to most recommendation questions. Super User does get a lot of recommendation questions, but it was never overrun by them.

On Unix & Linux, recommendation questions have never been a big issue. There are a couple of meta threads on the topic, but no specific policy.

Gaming has a blanket ban on recommendations. I didn't follow all the discussions, but as far as I know “they overrun the site” wasn't the primary argument against them — the argument was a lack of quality.

When Science Fiction & Fantasy started, there were 20% of list questions. Some discussions on SF&F conflated list questions and recommendation questions, but they are in fact two very different categories. Recommendation questions were banned because most of them were bad.

The now defunct Literature site tried out a book recommendation policy (live copy here). Having observed it at work, I would say that:

• Neither the fact that recommendations were allowed, nor the policy in itself, had a major impact on the site's viability. It died because there were too few participants.
• When the policy worked, it generated good, interesting questions with good answers.
• What made the good threads work was specific requirements — not “what shall I read”, but “here's a problem I want to solve” — and detailed answers — all the good answers recommended multiple books.
• A majority of recommendation questions were bad, but closing them was poorly enforced.

---

There are two keys to making recommendations work.

The first key, which you mention, is to solve a problem. A question that asks “what's the best X” should be closed. A good recommendation question asks “how do I do X” — and if the answers are “use product Y”, that's fine.

The other key has to do with answers. It's important not to let the answers degenerate into a poll. Stack Exchange sucks at polls. The voting on polls shows:

1. who posted first;
2. how popular the item is;
3. how well the item answers the question.

For the answers to be useful, #1 must as much as possible not be a factor. #2 is a side concern, #3 is the really important one. So it is vital that an answer is not “use this product” but “use this method” (which happens to use a product). If the method uses no specific product, or multiple products, or if an answer recommends different products for different scenarios, that's good.

I find it a bad sign that, even before the site starts, a proposal to require answers to contain a single suggestion is rather popular. Such a policy would be highly toxic to the site. On the contrary, for the site to be viable, answers with multiple suggestions should be encouraged. Answers that just say “use this product” should be deleted on sight.

---

And yes, it's weird to single out recommendation questions and have them on a separate site. Why not have them on the site where they're on-topic?

The only reason I can see is that recommendation questions require moderation, which sites such as SO and SU are poor at enforcing. On a site where an answer like “You should totally use jQuery” scores well instead of being summarily deleted, there is no hope to have good recommendation questions.

Answer 3

This might be a little bit early for a site that has not even started yet.

We have to take care of old questions. Software changes and so new software might meet the original requirements, a good example for this is syntax highlighting.

Several years ago, syntax highlighting wasn't standard. Sure, a few products might have it, but now it is standard. Almost everything except windows notepad supports that feature now.

So I expect that a growing number of software meets the requirements of old questions. How should we deal with that? Close them as obsolete?

Answer 4

As a moderator on Cooking, I have seen quite a few recommendation questions, and have had to reason about whether they are good or bad questions for a SE-network site. We didn't have a big discussion on "what should I buy" type of questions, but on the similar "where should I buy" type of question, which is generally a recommendation of stores.

The rule I have noticed is:

Askers who don't know if what they need exists are asking good questions (=seeking to expand an empty set of good solutions). Answers who are trying to choose between multiple products they know are asking bad questions (=seeking to make a choice from a set of multiple good solutions).

Technically, these two categories of askers tend to sort themselves into the categories mentioned in the "question" post. The ones who don't know if a product out there satisfies their needs already know their specific needs and list them in a question. The ones who are overwhelmed by a choice are the ones who ask "which is better, A or B" (Insert ATI vs Nvidia, Emacs vs Vi, etc.)

The reason I am posting this here is that it is a good lithmus test which can help a moderator make a decision about which questions to keep and which to close. I find it a more precise criterion than "specific enough", because specificity is a continuum, and it is harder to judge what is "enough". So my suggestion to the mods is to try applying this criterion and see if it fits with their gut feeling of which questions are good and which are bad. If it provides a good fit, you might want to codify it in the site policy.

A corollary is also that there is not much sense in urging askers of bad questions to make their questions more specific. The reason is that we are trying to recommend something which fits a set of needs, and the person who seeks "the best" doesn't know his own needs. You should of course gather data on this one too before basing policies on it, but you may not have noticed it (I am a requirements engineer, so a bit more sensitive to need fit etc.), so I thought that it is worth mentioning.