How are we doing? (Post your favorite stats!)

Question

By popular demand, here is a place for people to post regularly-updated metrics of traffic and other indicators of how Mathematica.SE is developing.

Please post sets of analysis from different sources as (community wiki) separate answers.

Answer 1

I have been scraping information from the Area 51 page every day since launch, sometimes more than once a day. I've been saving these data into an Excel file which I can visualise in Mathematica.

As at 28 June 2012 (9:00pm AEST) - the 178-day mark - This will be my final posting of Area 51 data

Answer 2

It is also possible to get some good information from the StackExchange API, as shown in this question on the main site.

As at 10 June 2012

Number of users
registered 1428
moderator 20
unregistered 115

This shows if drive-by (single-visit) users are becoming more frequent relative to regular users.

For these histograms, I'm using the {"Log", "Knuth"} binning specification.

566 users have no badges, implying two-thirds of users have done something on the site, even if it is only to upvote or complete their bio.

Answer 3

I thought I'd have a go at making a graphic showing the progress of some Area51 sites in a single image, as a learning exercise. I did 16 sites originally (chosen at random), now 31, and tried to use the same colors as used on the separate pages (red, orange, green). It was harder than I expected, so there's still much to learn.

Graphic updated Jun 21 2012:

Code, as of today, follows. Warning: viewing this code may be unpleasant for seasoned Mathematica-users... :/

siteNames = {"Mathematica", "Money", "Biology", "Code Golf", "Signal Processing", "SmugMug", "Software testing", "Cryptography", "Startup business", "Board games", "Brewing", "Quant. Finance", "Biblical H", "Lego", "Writers", "Graphic Design",  "AudioVideo", "Poker", "Martial Arts", "Code review", "Project Mgmt", "Fit & Nutrition", "Motor Maint.", "Parenting", "Gardening", "Personal prod", "French", "Linguistics", "Cog. Sci.", "Comp. Sci", "Computational"};

siteList = {
"http://area51.stackexchange.com/proposals/37304/mathematica",
"http://area51.stackexchange.com/proposals/1721/personal-finance-and-money",
"http://area51.stackexchange.com/proposals/12502/biology",
"http://area51.stackexchange.com/proposals/4570/code-golf-programming-puzzles",
"http://area51.stackexchange.com/proposals/1691/signal-processing",
"http://area51.stackexchange.com/proposals/25069/smugmug",
"http://area51.stackexchange.com/proposals/2241/software-quality-assurance-testing",
"http://area51.stackexchange.com/proposals/15811/cryptography",
"http://area51.stackexchange.com/proposals/6243/startup-business",
"http://area51.stackexchange.com/proposals/5220/board-and-card-games",
"http://area51.stackexchange.com/proposals/1619/homebrewing-beer-wine-etc",
"http://area51.stackexchange.com/proposals/117/quantitative-finance",
"http://area51.stackexchange.com/proposals/1817/biblical-hermeneutics",
"http://area51.stackexchange.com/proposals/10919/lego",
"http://area51.stackexchange.com/proposals/1623/writers",
"http://area51.stackexchange.com/proposals/1924/graphic-design",
"http://area51.stackexchange.com/proposals/4/audio-video-production",
"http://area51.stackexchange.com/proposals/3425/poker",
"http://area51.stackexchange.com/proposals/4470/martial-arts",
"http://area51.stackexchange.com/proposals/11464/code-review",
"http://area51.stackexchange.com/proposals/10947/project-management",
"http://area51.stackexchange.com/proposals/7080/fitness-nutrition",
"http://area51.stackexchange.com/proposals/1321/motor-vehicle-maintenance-and-repair",
"http://area51.stackexchange.com/proposals/4861/parenting",
"http://area51.stackexchange.com/proposals/1369/gardening-and-landscaping",
"http://area51.stackexchange.com/proposals/4296/personal-productivity",
"http://area51.stackexchange.com/proposals/29480/french-language-usage",
"http://area51.stackexchange.com/proposals/6673/linguistics",
"http://area51.stackexchange.com/proposals/2149/cognitive-sciences",
"http://area51.stackexchange.com/proposals/35636/computer-science",
"http://area51.stackexchange.com/proposals/28815/computational-science"};

srcList=Map[Import[#,"Plaintext"] & , siteList];

(* gustavo delfino's code for scraping the site*)

getSiteData[siteSource_] := With[
  {cmt = Whitespace ~~ comment : ("Excellent" | "Needs Work" | "Okay"),
   src =  StringReplace[siteSource, "," -> ""]},
  StringCases[src,
   {percent : NumberString ~~ "%" ~~ Whitespace ~~ "answered" ~~ 
      cmt -> {"answered", percent}, 
    grade : NumberString ~~ Whitespace ~~ "questions" ~~ Whitespace ~~
       "per day" ~~ cmt -> {"questions", grade}, 
    avidUsers : NumberString ~~ Whitespace ~~ "avid users" ~~ 
      Whitespace ~~ totalUsers : NumberString ~~ Whitespace ~~ 
      "total users" ~~ cmt -> {"avidusers", {avidUsers, totalUsers}},
    answerRatio : NumberString ~~ Whitespace ~~ "answer ratio" ~~ 
      cmt -> {"answerratio", answerRatio},
    visitsPerDay : NumberString ~~ Whitespace ~~ "visits/day" ~~ 
      cmt -> {"visits", visitsPerDay}}]]

drawBox[x_, y_, text_, color_] := Module[{},
   {Opacity[0.3],
    color,
    EdgeForm[{Black, AbsoluteThickness[0.2]}], 
    Rectangle[{x, y}, {x + boxWidth, y + 1}, RoundingRadius -> 0.2],
    Opacity[1], 
    Text[Style[text, 12, Black, FontFamily -> "Helvetica", 
      FontWeight -> "Bold"], {x + boxWidth/2, y + .45}], 
    color,
    Disk[{x, y + 0.5}, 0.5]}];

(* build a stack of boxes for each site *)

siteGraphics[siteName_, siteData_] := Module[
   {data = Reverse[getSiteData[siteData]]}, 
   x = 0; y = 0; boxWidth = 5;
   results = Reap[
     Sow[(* 
      site name*)
      {Black, 
       Text[Style[siteName, 11, FontFamily -> "Helvetica", 
         FontWeight -> "Bold"], {x + boxWidth /2, y - 0.5}]}];
     Do[
      key = el[[1]];
      value = ToExpression[el[[2]]];
      Switch[
       key,
       "visits", 
       Sow[
        drawBox[x, y, value, 
         Which[value < 500, Red, value < 1500, Orange, True, Green]]],
       "answered", 
       Sow[
        drawBox[x, y, value, 
         Which[value < 80, Red, value < 90, Orange, True, Green]]],
       "avidusers", (* two values, avid then total returned in a single list *)
       {Sow[
         drawBox[x, y, value[[1]], 
          Which[value[[1]] < 100, Red, value[[1]] <= 150, Orange, 
           True, Green]]];
        y += 1.2;
        (* don't know what colors for total user values ...? *)      
          Sow[drawBox[x, y, value[[2]], 
          Which[value[[2]] < 300, Red, value[[2]] <= 600, Orange, 
           True, Green]]]},
       "answerratio", 
       Sow[
        drawBox[x, y, value, 
         Which[value < 1, Red, value <= 2.5, Orange, True, Green]]],
       "questions",(* 15 is good *)
       Sow[drawBox[x, y, value, 
         Which[value <= 5, Red, value < 15, Orange, value >= 15,  
          Green]]]
       ];
      x += 0; y += 1.2; (* next box *), 
      {el, data}] (* End Do *)
     ]; (* end Reap *)
   Graphics[{results[[2]]}] (* 
   why does Reap do that Null thing? *)
   ];

finalGraphic = 
  siteGraphics[#[[1]], #[[2]]] & /@ Transpose[{siteNames, srcList}] /. 
   RGBColor[0, 1, 0] :> Darker[Green, 0.3];

g = GraphicsGrid[Partition[finalGraphic, 7, 7, {1, 1}, { }], 
  ImageSize -> 800, Spacings -> {25, 100}]

Answer 4

Area 51 vital signs

Here are the charts made with the help from answers to this question. They are concerned with monitoring the 5 Area 51 beta proposal vital signs. The code to fetch the data is available in the parent question.

I've made modifications to the way the data is presented. The data is now sorted by value, not by "days in beta". Also, I've added a coloring scheme, "Rainbow". The names have been reordered as well, pardon me.

As of May 25 2012:

---

BETA

---

LAUNCHED

Questions per day

Percentage answered

Avid users

Answer ratio

Visits per day

Answer 5

This code fetches data from here. It's actually faster than the one for Area 51. There's obviously a lot of repetitive code, it could be condensed. Another night.

Suit yourselves.

seStatsURL = "http://stackexchange.com/sites?view=list#traffic";

getSEStats[main_] := 
 Module[{url, fullXML, stackCreDate, stackQuestions, stackAnswers, 
   stackPerAns, stackUsers, stackVperD, stackQperD, stackName, 
   stackFavIco, stackURL, fullStackData, stackDays},

  (*Gather XML data from SE*)

  url = Import[seStatsURL, "XMLObject"];
  fullXML = 
   Table[url[[2]][[3]][[2]][[3]][[2]][[3]][[14]][[3]][[6]][[3]][[i]], \
{i, 2, 174, 2}];

  (*SE uses epoch time (jan 1 1970)*)

  stackCreDate = 
   Table[Cases[fullXML[[i]], 
     XMLElement[
      "input", {"type" -> "hidden", "name" -> "creation-date", 
       "value" -> ___}, ___], Infinity], {i, 1, Length[fullXML]}];
  stackCreDate = 
   ToExpression[
    Table[stackCreDate[[i]][[1]][[2]][[3]][[2]], {i, 1, 
      Length[fullXML]}]];

  stackDays = 
   Table[Ceiling[
     DateDifference[stackCreDate[[i]], Round[AbsoluteTime[]]] - 
      2], {i, 1, Length[fullXML]}];

  stackQuestions = 
   Table[Cases[fullXML[[i]], 
     XMLElement[
      "input", {"type" -> "hidden", "name" -> "questions", 
       "value" -> ___}, ___], Infinity], {i, 1, Length[fullXML]}];
  stackQuestions = 
   ToExpression[
    Table[stackQuestions[[i]][[1]][[2]][[3]][[2]], {i, 
      Length[fullXML]}]];

  stackAnswers = 
   Table[Cases[fullXML[[i]], 
     XMLElement[
      "input", {"type" -> "hidden", "name" -> "answers", 
       "value" -> ___}, ___], Infinity], {i, 1, Length[fullXML]}];
  stackAnswers = 
   ToExpression[
    Table[stackAnswers[[i]][[1]][[2]][[3]][[2]], {i, 
      Length[fullXML]}]];

  stackPerAns = 
   Table[Cases[fullXML[[i]], 
     XMLElement[
      "input", {"type" -> "hidden", "name" -> "percent-answered", 
       "value" -> ___}, ___], Infinity], {i, 1, Length[fullXML]}];
  stackPerAns = 
   ToExpression[
    Table[stackPerAns[[i]][[1]][[2]][[3]][[2]], {i, Length[fullXML]}]];

  stackUsers = 
   Table[Cases[fullXML[[i]], 
     XMLElement[
      "input", {"type" -> "hidden", "name" -> "users", 
       "value" -> ___}, ___], Infinity], {i, 1, Length[fullXML]}];
  stackUsers = 
   ToExpression[
    Table[stackUsers[[i]][[1]][[2]][[3]][[2]], {i, 
      Length[fullXML]}]];

  stackVperD = 
   Table[Cases[fullXML[[i]], 
     XMLElement[
      "input", {"type" -> "hidden", "name" -> "visits-per-day", 
       "value" -> ___}, ___], Infinity], {i, 1, Length[fullXML]}];
  stackVperD = 
   ToExpression[
    Table[stackVperD[[i]][[1]][[2]][[3]][[2]], {i, 
      Length[fullXML]}]];

  stackQperD = 
   Table[Cases[fullXML[[i]], 
     XMLElement[
      "input", {"type" -> "hidden", "name" -> "questions-per-day", 
       "value" -> ___}, ___], Infinity], {i, 1, Length[fullXML]}];
  stackQperD = 
   ToExpression[
    Table[stackQperD[[i]][[1]][[2]][[3]][[2]], {i, 
      Length[fullXML]}]];

  stackName = 
   Table[Cases[fullXML[[i]], 
     XMLElement[
      "input", {"type" -> "hidden", "name" -> "name", 
       "value" -> ___}, ___], Infinity], {i, 1, Length[fullXML]}];
  stackName = 
   Table[stackName[[i]][[1]][[2]][[3]][[2]], {i, Length[fullXML]}];


  (*Get URLs*)

  stackURL = 
   Table[fullXML[[i]][[3]][[2]][[2]][[2]][[2]], {i, 1, 
     Length[fullXML]}];

  (*Get favicons*)

  stackFavIco = 
   Table[fullXML[[i]][[3]][[3]][[3]][[2]][[2]][[2]][[2]], {i, 1, 
     Length[fullXML]}];

  fullStackData = 
   Table[{stackDays[[i]] - 25567, stackQuestions[[i]], 
     stackAnswers[[i]], stackPerAns[[i]], stackUsers[[i]], 
     stackVperD[[i]], stackQperD[[i]], stackName[[i]], stackURL[[i]], 
     stackFavIco[[i]]}, {i, 1, Length[fullXML]}]
  ]

stackData = getSEStats[seStatsURL];