35

As a statistician I focus on attempting to address questions about statistics, probability, and fitting. But I often tend to go beyond the Mathematica functional aspect and discuss/sermonize the need to meet assumptions and start with a model as opposed to just running some procedure on data.

Any suggestions as to the need to temper or encourage this urge with respect to what this site is all about?

I ask because I don't notice much of this going on with other subjects.

19
  • 27
    Speaking for myself, altho this isn't CV, I'd certainly appreciate being told if I'm abusing/misusing the statistical functionality. Keeps me honest and all that. Apr 22, 2016 at 22:13
  • 2
    I agree with J.M. I have learned a lot from your answers, e.g. your work arounds for generation of random sampling from custom distributions. The questions that are obviously about statistics or mathematics and not Mathematica are migrated. There are some hybrid questions that 'sermons' are helpful. :)
    – ubpdqn
    Apr 23, 2016 at 0:03
  • 7
    I wholeheartedly agree with the sentiments expressed above: please make us aware of the fine points in statistics and probability. Although I am supposed to use those concepts in my daily work, I have a love-hate relationship with statistics and probability, and I appreciate all the help I can get on those topics.
    – MarcoB
    Apr 23, 2016 at 0:12
  • 2
    I have the same relationship as MarcoB. I have also seen very instructive posts in physics (eg equations of motion, variational methods) and differential equations etc. I think this is enriching and increases the appropriate use of Mathematica application to the various domains.
    – ubpdqn
    Apr 23, 2016 at 1:39
  • 7
    +1, keep doing it. As nice as the probability/statistics functions are in MMA (world class, IMO), when a problem goes beyond trivial it almost always benefits from thinking before coding on the best way to utilize those functions, or work around their failings and foibles. Doing it within MMA and not advertising a third-party, non-free application? +10...
    – ciao
    Apr 23, 2016 at 7:03
  • 2
    I don't always have the time to write a careful explanation of the hypotheses and reasons underlying a Q&A, but sometimes I do. Sometimes the issue has been explained before, and I just don't have the motivation to offer yet another explanation that doesn't really say anything new. (And the +epsilon difference between it and another question means it's not an exact duplicate.)
    – Michael E2
    Apr 24, 2016 at 2:51
  • Everybody will agree with you here. However, in practice, they will, almost without exception, jump right away to coding in their answers. And they will then admire such "solutions" between themselves, praising each other for such great resourcefulness, not seeing that, in fact, they don't have a clue about theoretical basis of the MMA functionality in question. There are certain number of exceptions, but what I describe is unfortunately the current state of mentality on this site. I do expect now the worst possible reactions to this comment, which would actually be the proof that it is true.
    – VividD
    Apr 24, 2016 at 13:47
  • 2
    @VividD I am sorry that you have had a poor experience here. I am just a code monkey and I claim no theoretical basis, though I do try to help and honestly believe that many people have found my code-based answers useful. I have respect for those with a deeper understanding and much appreciation when they share it, as e.g. whuber often does. If you have ideas for how we may attract that kind of member I would be happy to hear them.
    – Mr.Wizard
    Apr 24, 2016 at 17:07
  • 2
    @VividD Even when I feel the strongest need to sermonize on an answer or comment, I've usually learned something new (and many times amazing) about Mathematica from the suggested code. In fact, having many folks not know some of the assumptions of statistical procedures has kept me in business for many, many years.
    – JimB
    Apr 25, 2016 at 0:23
  • I vote for sermon. Some of my favorite answers that I have written are exactly that type.
    – rcollyer
    Apr 25, 2016 at 2:44
  • 1
    @VividD I think you raise important points. It gives me pause for thought: wp.me/p1or7L-ER – ubpdqn 8 mins ago
    – ubpdqn
    Apr 25, 2016 at 5:49
  • 2
    The question title is a typical example of what statisticians do. Does it mean that (+1) vote is for "to sermon" and (-1) vote is for "not to sermon" ? Apr 25, 2016 at 13:45
  • 2
    @AntonAntonov. "The question title is a typical example of what statisticians do." Well, someone's got to do it. Because I sense somewhat of a green light to do so from the comments so far, I can continue with some of your instructive posts about fitting.
    – JimB
    Apr 25, 2016 at 18:16
  • 3
    @JimBaldwin I was implying that you have the mindset of the third statistician in the following joke, trying to use voting instead of shooting: "Three statisticians went duck hunting. A duck flew out and the first statistician took a shot, the shot went a foot too hight. The second statistician took his shot and the shot went a foot too low. The third statistician said, 'We got it!' " Apr 25, 2016 at 21:39
  • 3
    @VividD if you see answers with theoretical problems, please point them out to the poster. I have yet to see anyone refuse to correct or remove an answer when requested to do so; in fact this collaborative (rather than competitive) spirit seems more common on this site than others. But people can't correct what they don't know about. Apr 27, 2016 at 13:00

1 Answer 1

18

Prompted by Mr.Wizard's invitation/nudging to "pen an Answer" I decided to make some calculations to find the most important comments. It seems that "I vote to sermon. [...]" most clearly summarizes the expressed sentiments.

Definition of importance

Let us use the following definitions:

  1. The most important comments have the most important words, and
  2. the most important words are in the most important comments.

Calculation procedure

Starting with these definitions we do the following:

  1. Convert the comments into points in a linear vector space. Each word is an axis; each comment is a point.

  2. Do appropriate weighting of the comment-word associations (IDF, TFIDF, etc.).

  3. From the importance definitions we can find the eigenvectors of the representation matrix multiplied by its transpose.

  4. The most important comments will have largest coordinates in the eigenvectors.

Results

Here is an image of a Mathematica session with all the steps given above:

enter image description here

If we ignore my comment, the most important ones are :

  1. "I vote for sermon. [...]"
  2. "I have learned a lot from your answers, [...]"

Verdict

These calculations show that commenters want Jim Baldwin to make extensive statistical descriptions in this site. They also show that my comment about the question title has to be addressed.

(I am curios what kind of statistical objections Jim Baldwin might bring to this answer.)

Code

Text data

Get the question comments:

xmlObject = 
 Import["http://meta.mathematica.stackexchange.com/questions/1857/to-sermon-or-not-to-sermon/", "XMLObject"];

comments = 
 Cases[xmlObject, 
  XMLElement["span", {"class" -> "comment-copy"}, {c_}] :> c, Infinity]

Assign stop words:

stopWords = 
 ReadList["http://www.textfixer.com/resources/common-english-words.txt", Word, RecordSeparators -> ","]

Step 1 (vector space)

Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/DocumentTermMatrixConstruction.m"]

stopWords = ReadList["~/MathFiles/stop_words", String];

{cMat, cTerms} = DocumentTermMatrix[comments, {{}, stopWords}];

Magnify[cTerms, 0.7]

cMat // MatrixPlot

Step 2 (weight associations)

wcMat = WeightTerms[cMat, GlobalTermWeight["IDF", #1, #2] &, # &, #/Norm[#] &];

Step 3 (find eigenvectors of M.M^T)

nSentences = 3;

wcSMat = wcMat.Transpose[wcMat];
{vals, vecs} = Eigensystem[wcSMat, nSentences];

Find most important comments

inds = Reverse@Ordering[Abs[vecs[[1]]], -nSentences];
Transpose[{Abs[vecs[[1, inds]]], comments[[inds]]}]

(* {{0.599066, 
  "The question title is a typical example of what statisticians do. \
Does it mean that (+1) vote is for \"to sermon\" and (-1) vote is for \
\"not to sermon\" ?"}, 
    {0.563276, 
  "I vote for sermon. Some of my favorite answers that I have written \
are exactly that type."}, 
    {0.369312, 
  "+1, I'm all for sermon. Answers that give a lot of background are \
the most enjoyable to read."}} *)
13
  • 3
    Cheeky, you are. :) Apr 25, 2016 at 14:32
  • 5
    I must duly criticize this answer. Clearly, you have not learned that one does not post images, but copyable code. I'd downvote to encourage you to improve your answer, but the sheer cheek amuses me. (yes, sarcasm. Lots of it.) :)
    – rcollyer
    Apr 25, 2016 at 15:59
  • @rcollyer :) Thanks! I was hesitant to to put copyable code because I used an external package and the assignment of the comments might be lengthy -- I just wanted to provide evidence of the computations. Additionally, does the rule "post copiable code" hold also for the meta posts? Apr 25, 2016 at 16:57
  • Truthfully, I had misread your code. I thought you were grabbing the comments via a call to Import. I've looked a little at the API queries, and a simple method would have been welcome. :)
    – rcollyer
    Apr 25, 2016 at 18:34
  • @rcollyer You are completely right -- using Import is much better. I updated my answer accordingly. Apr 25, 2016 at 19:11
  • 1
    Here I thought the API would be used. Importing html as xml: bonus. I'd give you a +1, but this part of the site is like "Who's Line is it Anway?" the points don't matter. :)
    – rcollyer
    Apr 25, 2016 at 20:36
  • 2
    Ah, ha! There's a sermonable point in your code. Eigenvalues of $\mathbf{M}.\mathbf{M}^{T}$ has precision issues. Better to take the SVD of $M$ and square the singular values. :D (Yes, at this point, I'm just having fun.)
    – rcollyer
    Apr 25, 2016 at 20:41
  • 1
    @rcollyer Sigh... I was going for clarity of the code/formula given the definition of important comments. I sacrificed precision for clarity. Apr 25, 2016 at 21:29
  • @AntonAntonov :)
    – rcollyer
    Apr 26, 2016 at 2:27
  • @rcollyer, hadda link to that sermon of mine, why don't you? :P Apr 26, 2016 at 2:54
  • 1
    (On a more serious note, it's nice that statisticians are already starting to heed Golub's call to reformulate and stabilize their numerical linear algebra tools.) Apr 26, 2016 at 2:56
  • @J.M. of course I did. :)
    – rcollyer
    Apr 26, 2016 at 12:41
  • While I see that, in fact, you don't have a clue about theoretical basis of the MMA functionality in question, I praise you for such great resourcefulness... ;-}
    – ciao
    Apr 26, 2016 at 21:42

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .