19

By popular demand, here is a place for people to post regularly-updated metrics of traffic and other indicators of how Mathematica.SE is developing.

Please post sets of analysis from different sources as (community wiki) separate answers.

  • Thanks Verbeia! I wasn't sure it is legal to openly ask this question. – István Zachar Mar 25 '12 at 10:50
  • We shouldn't obsessively update too frequently, but there is clearly interest in this stuff. – Verbeia Mar 25 '12 at 11:14
18

I have been scraping information from the Area 51 page every day since launch, sometimes more than once a day. I've been saving these data into an Excel file which I can visualise in Mathematica.

As at 28 June 2012 (9:00pm AEST) - the 178-day mark - This will be my final posting of Area 51 data

Visits per day and users with 200+ rep
Committers and followers that signed up
Questions per day and total number of users Compulsiveness ratio - visits per day/users

  • I like the suggestion in Leonid's remark here. Could you also look at 1) number of users with an accepted answer and 2) number of unique users for accepted answers, both as a function of time (days in beta)? – rm -rf Mar 31 '12 at 21:43
  • Accepted answers will require some trawling through the API. I don't think time awarded is accessible in the API. The Scholar badge tracks users with upvoted answers: Currently 225 out of 880-odd users. – Verbeia Mar 31 '12 at 21:59
  • I didn't mean time the checkmark was awarded, but rather a growth in number of users over time. Of course, some users log in after a break and accept answers to a whole bunch of questions at once, leading to spikes, but it might be a good indicator of a growth in the userbase that's answering questions. Any way, it's not a big deal, and I suggested only because you have the basic framework setup. By the way, re: trawling and accessing time awarded via API, look into the docs for timeline view :) – rm -rf Mar 31 '12 at 22:06
  • Sorry, I meant the Teacher badge earlier, of which 158 have been awarded – Verbeia Mar 31 '12 at 22:31
  • I hadn't realized our sign-on rate (follower/committer) was as high as it is. :) – rcollyer Apr 2 '12 at 17:00
  • 3
    I love the almost perfectly linear growth in total users. At roughly 10 new users per day, the whole world will be on this site in about 1.8 million years! – Simon May 6 '12 at 0:07
  • I should note that the number of Teacher badges is now 195 - a really good sign! – Verbeia May 6 '12 at 4:49
  • 2
    I can't believe I missed this. This is awesome. You're awesome. This needs to be on Area 51. – Shog9 May 22 '12 at 1:48
  • @Shog9 I have updated the data and added a new metric based on Area 51 data. I'm just about to update the other post based on the API data. – Verbeia May 23 '12 at 11:15
  • There's a (very) slight up-tick in the number of users graph at the very end. I'd like to point out, yesterday, we added 11 users when our normal rate seems to be in the 3 - 5 range. Let's see if it keeps up. – rcollyer May 25 '12 at 4:15
  • @rcollyer I'll keep an eye on it. – Verbeia May 25 '12 at 5:45
14

It is also possible to get some good information from the StackExchange API, as shown in this question on the main site.

As at 10 June 2012

Number of users
registered 1428
moderator 20
unregistered 115

Number of days since last visit Cumulative users New users daily Cumulative users who joined and never came back

This shows if drive-by (single-visit) users are becoming more frequent relative to regular users.

Ratio of single-visit users to all users

For these histograms, I'm using the {"Log", "Knuth"} binning specification.

Reputation Histogram
Reputation Histogram (excl 1 and 101 rep people)
Weekly reputation change

566 users have no badges, implying two-thirds of users have done something on the site, even if it is only to upvote or complete their bio.

Histogram of badge counts

12

I thought I'd have a go at making a graphic showing the progress of some Area51 sites in a single image, as a learning exercise. I did 16 sites originally (chosen at random), now 31, and tried to use the same colors as used on the separate pages (red, orange, green). It was harder than I expected, so there's still much to learn.

Graphic updated Jun 21 2012:

area51 stats

Code, as of today, follows. Warning: viewing this code may be unpleasant for seasoned Mathematica-users... :/

siteNames = {"Mathematica", "Money", "Biology", "Code Golf", "Signal Processing", "SmugMug", "Software testing", "Cryptography", "Startup business", "Board games", "Brewing", "Quant. Finance", "Biblical H", "Lego", "Writers", "Graphic Design",  "AudioVideo", "Poker", "Martial Arts", "Code review", "Project Mgmt", "Fit & Nutrition", "Motor Maint.", "Parenting", "Gardening", "Personal prod", "French", "Linguistics", "Cog. Sci.", "Comp. Sci", "Computational"};

siteList = {
"http://area51.stackexchange.com/proposals/37304/mathematica",
"http://area51.stackexchange.com/proposals/1721/personal-finance-and-money",
"http://area51.stackexchange.com/proposals/12502/biology",
"http://area51.stackexchange.com/proposals/4570/code-golf-programming-puzzles",
"http://area51.stackexchange.com/proposals/1691/signal-processing",
"http://area51.stackexchange.com/proposals/25069/smugmug",
"http://area51.stackexchange.com/proposals/2241/software-quality-assurance-testing",
"http://area51.stackexchange.com/proposals/15811/cryptography",
"http://area51.stackexchange.com/proposals/6243/startup-business",
"http://area51.stackexchange.com/proposals/5220/board-and-card-games",
"http://area51.stackexchange.com/proposals/1619/homebrewing-beer-wine-etc",
"http://area51.stackexchange.com/proposals/117/quantitative-finance",
"http://area51.stackexchange.com/proposals/1817/biblical-hermeneutics",
"http://area51.stackexchange.com/proposals/10919/lego",
"http://area51.stackexchange.com/proposals/1623/writers",
"http://area51.stackexchange.com/proposals/1924/graphic-design",
"http://area51.stackexchange.com/proposals/4/audio-video-production",
"http://area51.stackexchange.com/proposals/3425/poker",
"http://area51.stackexchange.com/proposals/4470/martial-arts",
"http://area51.stackexchange.com/proposals/11464/code-review",
"http://area51.stackexchange.com/proposals/10947/project-management",
"http://area51.stackexchange.com/proposals/7080/fitness-nutrition",
"http://area51.stackexchange.com/proposals/1321/motor-vehicle-maintenance-and-repair",
"http://area51.stackexchange.com/proposals/4861/parenting",
"http://area51.stackexchange.com/proposals/1369/gardening-and-landscaping",
"http://area51.stackexchange.com/proposals/4296/personal-productivity",
"http://area51.stackexchange.com/proposals/29480/french-language-usage",
"http://area51.stackexchange.com/proposals/6673/linguistics",
"http://area51.stackexchange.com/proposals/2149/cognitive-sciences",
"http://area51.stackexchange.com/proposals/35636/computer-science",
"http://area51.stackexchange.com/proposals/28815/computational-science"};

srcList=Map[Import[#,"Plaintext"] & , siteList];

(* gustavo delfino's code for scraping the site*)

getSiteData[siteSource_] := With[
  {cmt = Whitespace ~~ comment : ("Excellent" | "Needs Work" | "Okay"),
   src =  StringReplace[siteSource, "," -> ""]},
  StringCases[src,
   {percent : NumberString ~~ "%" ~~ Whitespace ~~ "answered" ~~ 
      cmt -> {"answered", percent}, 
    grade : NumberString ~~ Whitespace ~~ "questions" ~~ Whitespace ~~
       "per day" ~~ cmt -> {"questions", grade}, 
    avidUsers : NumberString ~~ Whitespace ~~ "avid users" ~~ 
      Whitespace ~~ totalUsers : NumberString ~~ Whitespace ~~ 
      "total users" ~~ cmt -> {"avidusers", {avidUsers, totalUsers}},
    answerRatio : NumberString ~~ Whitespace ~~ "answer ratio" ~~ 
      cmt -> {"answerratio", answerRatio},
    visitsPerDay : NumberString ~~ Whitespace ~~ "visits/day" ~~ 
      cmt -> {"visits", visitsPerDay}}]]

drawBox[x_, y_, text_, color_] := Module[{},
   {Opacity[0.3],
    color,
    EdgeForm[{Black, AbsoluteThickness[0.2]}], 
    Rectangle[{x, y}, {x + boxWidth, y + 1}, RoundingRadius -> 0.2],
    Opacity[1], 
    Text[Style[text, 12, Black, FontFamily -> "Helvetica", 
      FontWeight -> "Bold"], {x + boxWidth/2, y + .45}], 
    color,
    Disk[{x, y + 0.5}, 0.5]}];

(* build a stack of boxes for each site *)

siteGraphics[siteName_, siteData_] := Module[
   {data = Reverse[getSiteData[siteData]]}, 
   x = 0; y = 0; boxWidth = 5;
   results = Reap[
     Sow[(* 
      site name*)
      {Black, 
       Text[Style[siteName, 11, FontFamily -> "Helvetica", 
         FontWeight -> "Bold"], {x + boxWidth /2, y - 0.5}]}];
     Do[
      key = el[[1]];
      value = ToExpression[el[[2]]];
      Switch[
       key,
       "visits", 
       Sow[
        drawBox[x, y, value, 
         Which[value < 500, Red, value < 1500, Orange, True, Green]]],
       "answered", 
       Sow[
        drawBox[x, y, value, 
         Which[value < 80, Red, value < 90, Orange, True, Green]]],
       "avidusers", (* two values, avid then total returned in a single list *)
       {Sow[
         drawBox[x, y, value[[1]], 
          Which[value[[1]] < 100, Red, value[[1]] <= 150, Orange, 
           True, Green]]];
        y += 1.2;
        (* don't know what colors for total user values ...? *)      
          Sow[drawBox[x, y, value[[2]], 
          Which[value[[2]] < 300, Red, value[[2]] <= 600, Orange, 
           True, Green]]]},
       "answerratio", 
       Sow[
        drawBox[x, y, value, 
         Which[value < 1, Red, value <= 2.5, Orange, True, Green]]],
       "questions",(* 15 is good *)
       Sow[drawBox[x, y, value, 
         Which[value <= 5, Red, value < 15, Orange, value >= 15,  
          Green]]]
       ];
      x += 0; y += 1.2; (* next box *), 
      {el, data}] (* End Do *)
     ]; (* end Reap *)
   Graphics[{results[[2]]}] (* 
   why does Reap do that Null thing? *)
   ];

finalGraphic = 
  siteGraphics[#[[1]], #[[2]]] & /@ Transpose[{siteNames, srcList}] /. 
   RGBColor[0, 1, 0] :> Darker[Green, 0.3];

g = GraphicsGrid[Partition[finalGraphic, 7, 7, {1, 1}, { }], 
  ImageSize -> 800, Spacings -> {25, 100}]
  • Thanks, it will take a while to get that monstrosity of a code out of my head. And, no I didn't really read through it. – rcollyer May 4 '12 at 18:17
  • Thanks cormullion! How come that the neat RoundingRadius effect is not observable in the figure? – István Zachar May 11 '12 at 14:08
  • 1
    @IstvánZachar Well, mainly because I'm a klutz, and uploaded the wrong one :) – cormullion May 11 '12 at 14:48
  • 1
    The round corners of the rectangles create an illusion of dark dots inbetween them. – Szabolcs May 12 '12 at 13:16
  • @Szabolcs Ah yes - the Hermann Grid (en.wikipedia.org/wiki/Grid_illusion)! It would probably go away if I increased the vertical or horizontal margins ... Perhaps next week. :) – cormullion May 12 '12 at 16:17
  • I like the addition of the circle on top of the bar. I don't know how it would look, but have you considered coloring the circle according to the prior weeks color as an indicator of change? – rcollyer May 19 '12 at 0:48
  • @rcollyer A progress indication would be a great thing to add - though it would require a complete overhaul to store last week's numbers somehow. Next week, perhaps... :) – cormullion May 19 '12 at 7:02
  • Use my code to fetch beta URLs if you want to. You could save each week's data in a CSV file, and read from it to compile your progress indicator. Shouldn't be too long to code. – CHM May 26 '12 at 1:05
11

Area 51 vital signs

Here are the charts made with the help from answers to this question. They are concerned with monitoring the 5 Area 51 beta proposal vital signs. The code to fetch the data is available in the parent question.

I've made modifications to the way the data is presented. The data is now sorted by value, not by "days in beta". Also, I've added a coloring scheme, "Rainbow". The names have been reordered as well, pardon me.

As of May 25 2012:


BETA

Questions per day

Percent answered

Avid Users

Answer Ratio

Visits per day


LAUNCHED

Questions per day

Percentage answered

Avid users

Answer ratio

Visits per day

  • Excellent and impressive work! I'm just wondering why you don't combine launched and beta graphs, perhaps using different colours for the two types. Presumably comparing the progress of betas with similar/different launched sites is the main use... But on reflection perhaps the varying scales would be a problem. – cormullion Mar 25 '12 at 14:46
  • 3
    Since you continue to update these, may I suggest that ordering the bars by length will be more informative than the current (arbitrary) order? That will make it much clearer where this site fits within the distribution of experiences of other beta sites. Also, the "percentage of answered questions" graphs are deceptive (has a large Tufte "Lie Factor") because they does not use zero as the origin (and their x-axes extend to an amazingly optimistic 120% :-). A good solution is to show the percentage of unanswered questions--but please base the bars at zero, not at some arbitrary value! – whuber Apr 4 '12 at 21:48
  • 1
    @CHM, One option would be to order as whuber suggested (it certainly is easier to read that way), but also use a color function for the bars... so something like red bars (oldest) are older than green, which are older than blue (youngest). You'll get something like this if you use "Rainbow". That way, you get to exploit the bars themselves to provide more info (right now, they're all bland) and lets one visualize in one go, the age of a site, its performance and where it stands as compared to others – rm -rf Apr 18 '12 at 17:25
  • 1
    @CHM please consider updating these fine charts; thank you. – Mr.Wizard May 23 '12 at 14:51
7

This code fetches data from here. It's actually faster than the one for Area 51. There's obviously a lot of repetitive code, it could be condensed. Another night.

Suit yourselves.

seStatsURL = "http://stackexchange.com/sites?view=list#traffic";

getSEStats[main_] := 
 Module[{url, fullXML, stackCreDate, stackQuestions, stackAnswers, 
   stackPerAns, stackUsers, stackVperD, stackQperD, stackName, 
   stackFavIco, stackURL, fullStackData, stackDays},

  (*Gather XML data from SE*)

  url = Import[seStatsURL, "XMLObject"];
  fullXML = 
   Table[url[[2]][[3]][[2]][[3]][[2]][[3]][[14]][[3]][[6]][[3]][[i]], \
{i, 2, 174, 2}];

  (*SE uses epoch time (jan 1 1970)*)

  stackCreDate = 
   Table[Cases[fullXML[[i]], 
     XMLElement[
      "input", {"type" -> "hidden", "name" -> "creation-date", 
       "value" -> ___}, ___], Infinity], {i, 1, Length[fullXML]}];
  stackCreDate = 
   ToExpression[
    Table[stackCreDate[[i]][[1]][[2]][[3]][[2]], {i, 1, 
      Length[fullXML]}]];

  stackDays = 
   Table[Ceiling[
     DateDifference[stackCreDate[[i]], Round[AbsoluteTime[]]] - 
      2], {i, 1, Length[fullXML]}];

  stackQuestions = 
   Table[Cases[fullXML[[i]], 
     XMLElement[
      "input", {"type" -> "hidden", "name" -> "questions", 
       "value" -> ___}, ___], Infinity], {i, 1, Length[fullXML]}];
  stackQuestions = 
   ToExpression[
    Table[stackQuestions[[i]][[1]][[2]][[3]][[2]], {i, 
      Length[fullXML]}]];

  stackAnswers = 
   Table[Cases[fullXML[[i]], 
     XMLElement[
      "input", {"type" -> "hidden", "name" -> "answers", 
       "value" -> ___}, ___], Infinity], {i, 1, Length[fullXML]}];
  stackAnswers = 
   ToExpression[
    Table[stackAnswers[[i]][[1]][[2]][[3]][[2]], {i, 
      Length[fullXML]}]];

  stackPerAns = 
   Table[Cases[fullXML[[i]], 
     XMLElement[
      "input", {"type" -> "hidden", "name" -> "percent-answered", 
       "value" -> ___}, ___], Infinity], {i, 1, Length[fullXML]}];
  stackPerAns = 
   ToExpression[
    Table[stackPerAns[[i]][[1]][[2]][[3]][[2]], {i, Length[fullXML]}]];

  stackUsers = 
   Table[Cases[fullXML[[i]], 
     XMLElement[
      "input", {"type" -> "hidden", "name" -> "users", 
       "value" -> ___}, ___], Infinity], {i, 1, Length[fullXML]}];
  stackUsers = 
   ToExpression[
    Table[stackUsers[[i]][[1]][[2]][[3]][[2]], {i, 
      Length[fullXML]}]];

  stackVperD = 
   Table[Cases[fullXML[[i]], 
     XMLElement[
      "input", {"type" -> "hidden", "name" -> "visits-per-day", 
       "value" -> ___}, ___], Infinity], {i, 1, Length[fullXML]}];
  stackVperD = 
   ToExpression[
    Table[stackVperD[[i]][[1]][[2]][[3]][[2]], {i, 
      Length[fullXML]}]];

  stackQperD = 
   Table[Cases[fullXML[[i]], 
     XMLElement[
      "input", {"type" -> "hidden", "name" -> "questions-per-day", 
       "value" -> ___}, ___], Infinity], {i, 1, Length[fullXML]}];
  stackQperD = 
   ToExpression[
    Table[stackQperD[[i]][[1]][[2]][[3]][[2]], {i, 
      Length[fullXML]}]];

  stackName = 
   Table[Cases[fullXML[[i]], 
     XMLElement[
      "input", {"type" -> "hidden", "name" -> "name", 
       "value" -> ___}, ___], Infinity], {i, 1, Length[fullXML]}];
  stackName = 
   Table[stackName[[i]][[1]][[2]][[3]][[2]], {i, Length[fullXML]}];


  (*Get URLs*)

  stackURL = 
   Table[fullXML[[i]][[3]][[2]][[2]][[2]][[2]], {i, 1, 
     Length[fullXML]}];

  (*Get favicons*)

  stackFavIco = 
   Table[fullXML[[i]][[3]][[3]][[3]][[2]][[2]][[2]][[2]], {i, 1, 
     Length[fullXML]}];

  fullStackData = 
   Table[{stackDays[[i]] - 25567, stackQuestions[[i]], 
     stackAnswers[[i]], stackPerAns[[i]], stackUsers[[i]], 
     stackVperD[[i]], stackQperD[[i]], stackName[[i]], stackURL[[i]], 
     stackFavIco[[i]]}, {i, 1, Length[fullXML]}]
  ]

stackData = getSEStats[seStatsURL];

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .