This is a post in a discussion thread. Please bear with me that it's not an answer as in "definite answer". I start seeing some limitations to the standard SE format for such kind of "debate". Even the thread title is not a question.
Demystifying AI is to me the main target of this proposal. A Q&A format seems particularly relevant. I understand and agree with your concerns. Yet does it really mean the proposal and possible site are doomed?
You, as a specialist in ML, probably (pun intended) see that there are many unreasonable expectations with "AI" in the world. These expectations come from very serious people that just see AI as magic, wonderful SciFi authors, and others. I think something needs be done to better understand what to expect from "AI". This is not about "killing the dream"---we need the dream to go on. This is about clarifying the muddy situation we have been in for so long with AI. This means getting naive questions, whether we like it or not. There are a lot of naive questions, even on SO, that many want to answer with a fully capitalized RTFM.
Now I'd like we keep in mind that specialists (or experts, if you will) are the very ones who "kill the dream by making it real" (which is terrific). Getting real, McCarthy is customarily cited at this point: "As soon as it works, no one calls it AI." So we seem to end up in a situation where the best people to talk about AI-related stuff (the Experts) are also the ones who don't want to talk about "it" because it does not look expert enough (I mean, answer questions deemed naive by experts). One issue here is to mix "weak AI" that progresses fast these days and contributes to our daily routines, with "strong AI" that is just plain elusive. Isn't it still "AI", and the very term that some initiatives like this Q&A could help clarify?
So, is the seemingly very low quality of the example questions on your grading system the real issue with this proposal? I tend to think the framing is more of an issue (which is related, yes), and the expectations: AI is no magic, and this Q&A would be here to make it clear.
I really like your final remark: "the majority of users of this site will end up being non-experts that speculate without any basis on unclear questions, which would result in very low-quality content." I think that describes why showing that AI is no magic matters: To curb speculations and "correct" understandings.
A couple extra comments:
- "What's the difference between AI and genetic algorithms?" If some people are confused with the terms, you and I would probably answer "go read Wikipedia or a text book". Fair enough. Now, if there is a confusion that is upvoted, we may also see it as a mission for "people who know" to clarify it. Disclaimer: I don't like this question.
- "This is a question that I wouldn't be surprised to see on WordBuilding, but not in a scientific site." Great point, related to the proposal framing: Is Science the right category? Science may entail that people expect "definitive" answers (as in state of the art). I chose the Science category at first, as in scientific method. I would like to see the proposal going toward enlightening people with logical (hypothesis, try, measure, repeat) statements, in other words dealing with "non-experts that speculate", as well as "expert" questions.
- "How is deep learning a step towards self aware AI ?" Bingo. This question is pure mud we want to clear out. So many people get confused with this one. Wouldn't something like a Q&A help with such kind of confusion? And I would be thrilled to hear about alternatives too.