17

The list of sites in the beta phase has now reached two pages in length, which probably serves to "cloak" some of the newer beta sites, and some of these sites have been going on the better part of two years. If they were going to make some substantial traffic move, they would have done so by now. Right now, it feels like there's essentially simply a "Tier 2" group of sites that are established, but low traffic enough that Stack Exchange can't be bothered to move them to the main site. Which, as I understand it, isn't the purpose of either Area 51, or the notion of beta testing sites.

I looked at the metrics reported for the 53 beta sites that are currently more than 10 days old (the metrics for brand new sites are bonkers), and looked at some of the relationships between age of the site and metrics for "success". Here's what I found:

Questions/Day: This is, in my mind, the currently most flawed metric being used by Area 51. None of the beta sites reach the "healthy" threshold, and to be perfectly frank, if the 15 questions/day threshold is going to be continued, sites actually trying to reach non-beta status (rather than languishing in at-best a perpetual beta) should pack up, go home, or seek other alternatives. But I stray from the point.

If we're expecting older sites to "age into" higher questions/day, we're going to be waiting a long time. A simple linear fit of the two (more complex fits didn't perform better) shows a significant, but tiny increase in that ratio. An increase of 0.0042 per day older. To put that in perspective, that's an increase of 1.53 per year. At that rate it will take the most successful site (by that metric, it's Code Review) another 1047 days to reach a target of 15. That's another 2.9 years in beta.

Answered Percentage: There's really no relationship at all, and most sites are doing fairly well on this metric anyway. It's not what's holding sites back.

Avid Users: Unsurprisingly, this one does have a very strong association with site age - an "avid user" is added, according again to a simple linear model, every 2 days or so. Given reputation takes time to accumulate, this is as expected. But it also can't be what's holding sites back - a wealth of the older sites (and some newer ones, like Christianity) have 200+ avid users, and there's an even greater number nearly there.

Total Users: Again, unsurprisingly, older sites have more users. As there's no entry for what's considered a "good" number of total users, I can't evaluate what's considered a "successful" site, but there's several older sites in the tens of thousands of total users.

Ratio of Questions to Answers: There's another very small linear relationship - another waiting years for a single digit increase in the ratio. Beyond that, officially 2.5 is "good". Only 7 of the 17 sites more than a year old have a lower ratio than that, and none of the oldest 5 do. This, again, can't be what's holding them up.

Visits: Visits is another very strong relationship - about 4 visitors are added per day, according to the fit. But, once again, 1,500 is supposedly "good". There's 7 year old sites that are above that threshold - some by a considerable margin.

What's the point of all this musing? Letting betas run clearly has the intended effect - the older sites are, by and large, stronger. But some of these sites are now quite old for betas, have established communities, and it appears are waiting for the "Questions/Day ratio" - or some metric that's hidden behind the curtain - to get across the starting line.

My assertion is that the 15/day number is a flawed one - eventually niche sites simply won't be able to reach it, and it either needs to get adjusted, or sites that don't get there need to be put out of their misery. But is there really any reason to keep say, Personal Finance, Startups or RPGs as betas? They've got large numbers of avid and total users, a healthy Q&A climate, 1500+ visitors per day...what purpose is served by keeping them off the main site? What's the path offered for them to reach main site status given they haven't already, and the one metric where they're struggling is something that, based on the evidence, they won't reach in a meaningful amount of time?

4

I've written extensively about what we're looking for before moving sites to graduation. These are the two most relevant blog posts:

Does this site have a chance of succeeding?

When will my site graduate?

Of the statistics you cited, the only two that are part of the minimum requirements for graduation are as follows:

  • 90+ days in beta
  • 10+ editors (users with 2000+ reputation)
  • 5+ closers (users with 3000+ reputation)

The rest of the statistics we look at, but they are primarily for the communities to know how they're doing.

Once the site reaches their minimum requirements, we start looking for that turning point in the number of visits that indicates they have reached full network effects. It's all outlined in those blog posts.

We recognize that site are of varying sizes and potential audiences. That's why we don't base graduation solely on the numbers. But that "report card" in Area 51 gives you a basis for comparison.

  • 11
    The problem is that last bit, whether or not your site traffic has reached some sort of critical mass, is something you're putting behind the curtain, even if you're writing about it. "When your site finally reaches that tipping point..." isn't something that's actionable by the community. Can you, for example, say what Personal Finance needs to do - and why you think it can be done if it hasn't been in the last 586 days? – Fomite Mar 13 '12 at 17:55
  • 7
    You keep citing those two posts any time anyone asks why there are sites in betas for going on 2 years, but they are woefully unhelpful and fail to address any real criticism to the process. At the end of the first blog post (from July 2011), you even state: "A large part of this summer will be spent looking at the traffic data we’ve accumulated over the last three years to make sense of it all." So what was the result of that analysis? What has SE learned? What new information can you provide us on this process? What specifically is holding back the sites that have been in beta for 2 years? – user16359 Mar 15 '12 at 3:14
  • 1
    @Mark I quote the posts because they are wholly relevant and current. Graduation indicates that a site is self-sustaining and growing under it's own momentum (i.e. we can "lock it in" as successful). The last hurdle for graduation, specifically, is typically the same as it always was; waiting for those network effects to kick in. Studying past sites hasn't shown a predictable way to determine when a site will have enough interesting questions to intrigue the search community to find and keep using the site. That type of intangible quality isn't easy to quantify or predict. – Robert Cartaino Mar 15 '12 at 12:54
  • 1
    Somewhat related, since you mention it is the only definitive requirement, is that the users statistic (including users above 2k/3k reputation) just shows the maximum, not the number of people that have stuck around with that privilege. Jeff had hinted several months ago that some kind of "citizenship rating" was in the works to better track active users and make it easier to tell if the user base for a site is growing or shrinking. Do you guys have any more information about that or other improvements to more easily track the growth of a site? – Troyen Mar 15 '12 at 20:16
  • 1
    @Troyen There has been (recent) discussion about "citizenship ratings" that give an more-current indication about recent activity on the site. I don't recall any more details than that. – Robert Cartaino Mar 15 '12 at 20:37
  • Money now has 900+ days in beta, 53 users with 2000+ rep, and 31 users with 3000+ rep, and has demonstrated significant traffic growth recently. Network effect attained? – Chris W. Rea Feb 1 '13 at 20:41
  • @RobertCartaino will it help reposting Chris W. Rea's comment as a new question to get it answered? Same goes for EpiGrad's comment. How do you determine that a site has reached "the tipping point" or not? – Andrew Savinykh Apr 23 '13 at 10:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .