20

By popular demand, here is a place for people to post regularly-updated metrics of traffic and other indicators of how Mathematica.SE is developing.

Please post sets of analysis from different sources as (community wiki) separate answers.

2
  • Thanks Verbeia! I wasn't sure it is legal to openly ask this question. Commented Mar 25, 2012 at 10:50
  • We shouldn't obsessively update too frequently, but there is clearly interest in this stuff.
    – Verbeia
    Commented Mar 25, 2012 at 11:14

5 Answers 5

19

I have been scraping information from the Area 51 page every day since launch, sometimes more than once a day. I've been saving these data into an Excel file which I can visualise in Mathematica.

As at 28 June 2012 (9:00pm AEST) - the 178-day mark - This will be my final posting of Area 51 data

Visits per day and users with 200+ rep
Committers and followers that signed up
Questions per day and total number of users Compulsiveness ratio - visits per day/users

11
  • I like the suggestion in Leonid's remark here. Could you also look at 1) number of users with an accepted answer and 2) number of unique users for accepted answers, both as a function of time (days in beta)?
    – rm -rf Mod
    Commented Mar 31, 2012 at 21:43
  • Accepted answers will require some trawling through the API. I don't think time awarded is accessible in the API. The Scholar badge tracks users with upvoted answers: Currently 225 out of 880-odd users.
    – Verbeia
    Commented Mar 31, 2012 at 21:59
  • I didn't mean time the checkmark was awarded, but rather a growth in number of users over time. Of course, some users log in after a break and accept answers to a whole bunch of questions at once, leading to spikes, but it might be a good indicator of a growth in the userbase that's answering questions. Any way, it's not a big deal, and I suggested only because you have the basic framework setup. By the way, re: trawling and accessing time awarded via API, look into the docs for timeline view :)
    – rm -rf Mod
    Commented Mar 31, 2012 at 22:06
  • Sorry, I meant the Teacher badge earlier, of which 158 have been awarded
    – Verbeia
    Commented Mar 31, 2012 at 22:31
  • I hadn't realized our sign-on rate (follower/committer) was as high as it is. :)
    – rcollyer
    Commented Apr 2, 2012 at 17:00
  • 3
    I love the almost perfectly linear growth in total users. At roughly 10 new users per day, the whole world will be on this site in about 1.8 million years!
    – Simon
    Commented May 6, 2012 at 0:07
  • I should note that the number of Teacher badges is now 195 - a really good sign!
    – Verbeia
    Commented May 6, 2012 at 4:49
  • 2
    I can't believe I missed this. This is awesome. You're awesome. This needs to be on Area 51.
    – Shog9
    Commented May 22, 2012 at 1:48
  • @Shog9 I have updated the data and added a new metric based on Area 51 data. I'm just about to update the other post based on the API data.
    – Verbeia
    Commented May 23, 2012 at 11:15
  • There's a (very) slight up-tick in the number of users graph at the very end. I'd like to point out, yesterday, we added 11 users when our normal rate seems to be in the 3 - 5 range. Let's see if it keeps up.
    – rcollyer
    Commented May 25, 2012 at 4:15
  • @rcollyer I'll keep an eye on it.
    – Verbeia
    Commented May 25, 2012 at 5:45
15

It is also possible to get some good information from the StackExchange API, as shown in this question on the main site.

As at 10 June 2012

Number of users
registered 1428
moderator 20
unregistered 115

Number of days since last visit Cumulative users New users daily Cumulative users who joined and never came back

This shows if drive-by (single-visit) users are becoming more frequent relative to regular users.

Ratio of single-visit users to all users

For these histograms, I'm using the {"Log", "Knuth"} binning specification.

Reputation Histogram
Reputation Histogram (excl 1 and 101 rep people)
Weekly reputation change

566 users have no badges, implying two-thirds of users have done something on the site, even if it is only to upvote or complete their bio.

Histogram of badge counts

0
13

I thought I'd have a go at making a graphic showing the progress of some Area51 sites in a single image, as a learning exercise. I did 16 sites originally (chosen at random), now 31, and tried to use the same colors as used on the separate pages (red, orange, green). It was harder than I expected, so there's still much to learn.

Graphic updated Jun 21 2012:

area51 stats

Code, as of today, follows. Warning: viewing this code may be unpleasant for seasoned Mathematica-users... :/

siteNames = {"Mathematica", "Money", "Biology", "Code Golf", "Signal Processing", "SmugMug", "Software testing", "Cryptography", "Startup business", "Board games", "Brewing", "Quant. Finance", "Biblical H", "Lego", "Writers", "Graphic Design",  "AudioVideo", "Poker", "Martial Arts", "Code review", "Project Mgmt", "Fit & Nutrition", "Motor Maint.", "Parenting", "Gardening", "Personal prod", "French", "Linguistics", "Cog. Sci.", "Comp. Sci", "Computational"};

siteList = {
"http://area51.stackexchange.com/proposals/37304/mathematica",
"http://area51.stackexchange.com/proposals/1721/personal-finance-and-money",
"http://area51.stackexchange.com/proposals/12502/biology",
"http://area51.stackexchange.com/proposals/4570/code-golf-programming-puzzles",
"http://area51.stackexchange.com/proposals/1691/signal-processing",
"http://area51.stackexchange.com/proposals/25069/smugmug",
"http://area51.stackexchange.com/proposals/2241/software-quality-assurance-testing",
"http://area51.stackexchange.com/proposals/15811/cryptography",
"http://area51.stackexchange.com/proposals/6243/startup-business",
"http://area51.stackexchange.com/proposals/5220/board-and-card-games",
"http://area51.stackexchange.com/proposals/1619/homebrewing-beer-wine-etc",
"http://area51.stackexchange.com/proposals/117/quantitative-finance",
"http://area51.stackexchange.com/proposals/1817/biblical-hermeneutics",
"http://area51.stackexchange.com/proposals/10919/lego",
"http://area51.stackexchange.com/proposals/1623/writers",
"http://area51.stackexchange.com/proposals/1924/graphic-design",
"http://area51.stackexchange.com/proposals/4/audio-video-production",
"http://area51.stackexchange.com/proposals/3425/poker",
"http://area51.stackexchange.com/proposals/4470/martial-arts",
"http://area51.stackexchange.com/proposals/11464/code-review",
"http://area51.stackexchange.com/proposals/10947/project-management",
"http://area51.stackexchange.com/proposals/7080/fitness-nutrition",
"http://area51.stackexchange.com/proposals/1321/motor-vehicle-maintenance-and-repair",
"http://area51.stackexchange.com/proposals/4861/parenting",
"http://area51.stackexchange.com/proposals/1369/gardening-and-landscaping",
"http://area51.stackexchange.com/proposals/4296/personal-productivity",
"http://area51.stackexchange.com/proposals/29480/french-language-usage",
"http://area51.stackexchange.com/proposals/6673/linguistics",
"http://area51.stackexchange.com/proposals/2149/cognitive-sciences",
"http://area51.stackexchange.com/proposals/35636/computer-science",
"http://area51.stackexchange.com/proposals/28815/computational-science"};

srcList=Map[Import[#,"Plaintext"] & , siteList];

(* gustavo delfino's code for scraping the site*)

getSiteData[siteSource_] := With[
  {cmt = Whitespace ~~ comment : ("Excellent" | "Needs Work" | "Okay"),
   src =  StringReplace[siteSource, "," -> ""]},
  StringCases[src,
   {percent : NumberString ~~ "%" ~~ Whitespace ~~ "answered" ~~ 
      cmt -> {"answered", percent}, 
    grade : NumberString ~~ Whitespace ~~ "questions" ~~ Whitespace ~~
       "per day" ~~ cmt -> {"questions", grade}, 
    avidUsers : NumberString ~~ Whitespace ~~ "avid users" ~~ 
      Whitespace ~~ totalUsers : NumberString ~~ Whitespace ~~ 
      "total users" ~~ cmt -> {"avidusers", {avidUsers, totalUsers}},
    answerRatio : NumberString ~~ Whitespace ~~ "answer ratio" ~~ 
      cmt -> {"answerratio", answerRatio},
    visitsPerDay : NumberString ~~ Whitespace ~~ "visits/day" ~~ 
      cmt -> {"visits", visitsPerDay}}]]

drawBox[x_, y_, text_, color_] := Module[{},
   {Opacity[0.3],
    color,
    EdgeForm[{Black, AbsoluteThickness[0.2]}], 
    Rectangle[{x, y}, {x + boxWidth, y + 1}, RoundingRadius -> 0.2],
    Opacity[1], 
    Text[Style[text, 12, Black, FontFamily -> "Helvetica", 
      FontWeight -> "Bold"], {x + boxWidth/2, y + .45}], 
    color,
    Disk[{x, y + 0.5}, 0.5]}];

(* build a stack of boxes for each site *)

siteGraphics[siteName_, siteData_] := Module[
   {data = Reverse[getSiteData[siteData]]}, 
   x = 0; y = 0; boxWidth = 5;
   results = Reap[
     Sow[(* 
      site name*)
      {Black, 
       Text[Style[siteName, 11, FontFamily -> "Helvetica", 
         FontWeight -> "Bold"], {x + boxWidth /2, y - 0.5}]}];
     Do[
      key = el[[1]];
      value = ToExpression[el[[2]]];
      Switch[
       key,
       "visits", 
       Sow[
        drawBox[x, y, value, 
         Which[value < 500, Red, value < 1500, Orange, True, Green]]],
       "answered", 
       Sow[
        drawBox[x, y, value, 
         Which[value < 80, Red, value < 90, Orange, True, Green]]],
       "avidusers", (* two values, avid then total returned in a single list *)
       {Sow[
         drawBox[x, y, value[[1]], 
          Which[value[[1]] < 100, Red, value[[1]] <= 150, Orange, 
           True, Green]]];
        y += 1.2;
        (* don't know what colors for total user values ...? *)      
          Sow[drawBox[x, y, value[[2]], 
          Which[value[[2]] < 300, Red, value[[2]] <= 600, Orange, 
           True, Green]]]},
       "answerratio", 
       Sow[
        drawBox[x, y, value, 
         Which[value < 1, Red, value <= 2.5, Orange, True, Green]]],
       "questions",(* 15 is good *)
       Sow[drawBox[x, y, value, 
         Which[value <= 5, Red, value < 15, Orange, value >= 15,  
          Green]]]
       ];
      x += 0; y += 1.2; (* next box *), 
      {el, data}] (* End Do *)
     ]; (* end Reap *)
   Graphics[{results[[2]]}] (* 
   why does Reap do that Null thing? *)
   ];

finalGraphic = 
  siteGraphics[#[[1]], #[[2]]] & /@ Transpose[{siteNames, srcList}] /. 
   RGBColor[0, 1, 0] :> Darker[Green, 0.3];

g = GraphicsGrid[Partition[finalGraphic, 7, 7, {1, 1}, { }], 
  ImageSize -> 800, Spacings -> {25, 100}]
8
  • Thanks, it will take a while to get that monstrosity of a code out of my head. And, no I didn't really read through it.
    – rcollyer
    Commented May 4, 2012 at 18:17
  • Thanks cormullion! How come that the neat RoundingRadius effect is not observable in the figure? Commented May 11, 2012 at 14:08
  • 1
    @IstvánZachar Well, mainly because I'm a klutz, and uploaded the wrong one :)
    – cormullion
    Commented May 11, 2012 at 14:48
  • 1
    The round corners of the rectangles create an illusion of dark dots inbetween them.
    – Szabolcs
    Commented May 12, 2012 at 13:16
  • @Szabolcs Ah yes - the Hermann Grid (en.wikipedia.org/wiki/Grid_illusion)! It would probably go away if I increased the vertical or horizontal margins ... Perhaps next week. :)
    – cormullion
    Commented May 12, 2012 at 16:17
  • I like the addition of the circle on top of the bar. I don't know how it would look, but have you considered coloring the circle according to the prior weeks color as an indicator of change?
    – rcollyer
    Commented May 19, 2012 at 0:48
  • @rcollyer A progress indication would be a great thing to add - though it would require a complete overhaul to store last week's numbers somehow. Next week, perhaps... :)
    – cormullion
    Commented May 19, 2012 at 7:02
  • Use my code to fetch beta URLs if you want to. You could save each week's data in a CSV file, and read from it to compile your progress indicator. Shouldn't be too long to code.
    – CHM
    Commented May 26, 2012 at 1:05
11

Area 51 vital signs

Here are the charts made with the help from answers to this question. They are concerned with monitoring the 5 Area 51 beta proposal vital signs. The code to fetch the data is available in the parent question.

I've made modifications to the way the data is presented. The data is now sorted by value, not by "days in beta". Also, I've added a coloring scheme, "Rainbow". The names have been reordered as well, pardon me.

As of May 25 2012:


BETA

Questions per day

Percent answered

Avid Users

Answer Ratio

Visits per day


LAUNCHED

Questions per day

Percentage answered

Avid users

Answer ratio

Visits per day

4
  • Excellent and impressive work! I'm just wondering why you don't combine launched and beta graphs, perhaps using different colours for the two types. Presumably comparing the progress of betas with similar/different launched sites is the main use... But on reflection perhaps the varying scales would be a problem.
    – cormullion
    Commented Mar 25, 2012 at 14:46
  • 3
    Since you continue to update these, may I suggest that ordering the bars by length will be more informative than the current (arbitrary) order? That will make it much clearer where this site fits within the distribution of experiences of other beta sites. Also, the "percentage of answered questions" graphs are deceptive (has a large Tufte "Lie Factor") because they does not use zero as the origin (and their x-axes extend to an amazingly optimistic 120% :-). A good solution is to show the percentage of unanswered questions--but please base the bars at zero, not at some arbitrary value!
    – whuber
    Commented Apr 4, 2012 at 21:48
  • 1
    @CHM, One option would be to order as whuber suggested (it certainly is easier to read that way), but also use a color function for the bars... so something like red bars (oldest) are older than green, which are older than blue (youngest). You'll get something like this if you use "Rainbow". That way, you get to exploit the bars themselves to provide more info (right now, they're all bland) and lets one visualize in one go, the age of a site, its performance and where it stands as compared to others
    – rm -rf Mod
    Commented Apr 18, 2012 at 17:25
  • 1
    @CHM please consider updating these fine charts; thank you.
    – Mr.Wizard
    Commented May 23, 2012 at 14:51
7

This code fetches data from here. It's actually faster than the one for Area 51. There's obviously a lot of repetitive code, it could be condensed. Another night.

Suit yourselves.

seStatsURL = "http://stackexchange.com/sites?view=list#traffic";

getSEStats[main_] := 
 Module[{url, fullXML, stackCreDate, stackQuestions, stackAnswers, 
   stackPerAns, stackUsers, stackVperD, stackQperD, stackName, 
   stackFavIco, stackURL, fullStackData, stackDays},

  (*Gather XML data from SE*)

  url = Import[seStatsURL, "XMLObject"];
  fullXML = 
   Table[url[[2]][[3]][[2]][[3]][[2]][[3]][[14]][[3]][[6]][[3]][[i]], \
{i, 2, 174, 2}];

  (*SE uses epoch time (jan 1 1970)*)

  stackCreDate = 
   Table[Cases[fullXML[[i]], 
     XMLElement[
      "input", {"type" -> "hidden", "name" -> "creation-date", 
       "value" -> ___}, ___], Infinity], {i, 1, Length[fullXML]}];
  stackCreDate = 
   ToExpression[
    Table[stackCreDate[[i]][[1]][[2]][[3]][[2]], {i, 1, 
      Length[fullXML]}]];

  stackDays = 
   Table[Ceiling[
     DateDifference[stackCreDate[[i]], Round[AbsoluteTime[]]] - 
      2], {i, 1, Length[fullXML]}];

  stackQuestions = 
   Table[Cases[fullXML[[i]], 
     XMLElement[
      "input", {"type" -> "hidden", "name" -> "questions", 
       "value" -> ___}, ___], Infinity], {i, 1, Length[fullXML]}];
  stackQuestions = 
   ToExpression[
    Table[stackQuestions[[i]][[1]][[2]][[3]][[2]], {i, 
      Length[fullXML]}]];

  stackAnswers = 
   Table[Cases[fullXML[[i]], 
     XMLElement[
      "input", {"type" -> "hidden", "name" -> "answers", 
       "value" -> ___}, ___], Infinity], {i, 1, Length[fullXML]}];
  stackAnswers = 
   ToExpression[
    Table[stackAnswers[[i]][[1]][[2]][[3]][[2]], {i, 
      Length[fullXML]}]];

  stackPerAns = 
   Table[Cases[fullXML[[i]], 
     XMLElement[
      "input", {"type" -> "hidden", "name" -> "percent-answered", 
       "value" -> ___}, ___], Infinity], {i, 1, Length[fullXML]}];
  stackPerAns = 
   ToExpression[
    Table[stackPerAns[[i]][[1]][[2]][[3]][[2]], {i, Length[fullXML]}]];

  stackUsers = 
   Table[Cases[fullXML[[i]], 
     XMLElement[
      "input", {"type" -> "hidden", "name" -> "users", 
       "value" -> ___}, ___], Infinity], {i, 1, Length[fullXML]}];
  stackUsers = 
   ToExpression[
    Table[stackUsers[[i]][[1]][[2]][[3]][[2]], {i, 
      Length[fullXML]}]];

  stackVperD = 
   Table[Cases[fullXML[[i]], 
     XMLElement[
      "input", {"type" -> "hidden", "name" -> "visits-per-day", 
       "value" -> ___}, ___], Infinity], {i, 1, Length[fullXML]}];
  stackVperD = 
   ToExpression[
    Table[stackVperD[[i]][[1]][[2]][[3]][[2]], {i, 
      Length[fullXML]}]];

  stackQperD = 
   Table[Cases[fullXML[[i]], 
     XMLElement[
      "input", {"type" -> "hidden", "name" -> "questions-per-day", 
       "value" -> ___}, ___], Infinity], {i, 1, Length[fullXML]}];
  stackQperD = 
   ToExpression[
    Table[stackQperD[[i]][[1]][[2]][[3]][[2]], {i, 
      Length[fullXML]}]];

  stackName = 
   Table[Cases[fullXML[[i]], 
     XMLElement[
      "input", {"type" -> "hidden", "name" -> "name", 
       "value" -> ___}, ___], Infinity], {i, 1, Length[fullXML]}];
  stackName = 
   Table[stackName[[i]][[1]][[2]][[3]][[2]], {i, Length[fullXML]}];


  (*Get URLs*)

  stackURL = 
   Table[fullXML[[i]][[3]][[2]][[2]][[2]][[2]], {i, 1, 
     Length[fullXML]}];

  (*Get favicons*)

  stackFavIco = 
   Table[fullXML[[i]][[3]][[3]][[3]][[2]][[2]][[2]][[2]], {i, 1, 
     Length[fullXML]}];

  fullStackData = 
   Table[{stackDays[[i]] - 25567, stackQuestions[[i]], 
     stackAnswers[[i]], stackPerAns[[i]], stackUsers[[i]], 
     stackVperD[[i]], stackQperD[[i]], stackName[[i]], stackURL[[i]], 
     stackFavIco[[i]]}, {i, 1, Length[fullXML]}]
  ]

stackData = getSEStats[seStatsURL];

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .