35

As a statistician I focus on attempting to address questions about statistics, probability, and fitting. But I often tend to go beyond the Mathematica functional aspect and discuss/sermonize the need to meet assumptions and start with a model as opposed to just running some procedure on data.

Any suggestions as to the need to temper or encourage this urge with respect to what this site is all about?

I ask because I don't notice much of this going on with other subjects.

  • 27
    Speaking for myself, altho this isn't CV, I'd certainly appreciate being told if I'm abusing/misusing the statistical functionality. Keeps me honest and all that. – J. M. is away Apr 22 '16 at 22:13
  • 2
    I agree with J.M. I have learned a lot from your answers, e.g. your work arounds for generation of random sampling from custom distributions. The questions that are obviously about statistics or mathematics and not Mathematica are migrated. There are some hybrid questions that 'sermons' are helpful. :) – ubpdqn Apr 23 '16 at 0:03
  • 7
    I wholeheartedly agree with the sentiments expressed above: please make us aware of the fine points in statistics and probability. Although I am supposed to use those concepts in my daily work, I have a love-hate relationship with statistics and probability, and I appreciate all the help I can get on those topics. – MarcoB Apr 23 '16 at 0:12
  • 2
    I have the same relationship as MarcoB. I have also seen very instructive posts in physics (eg equations of motion, variational methods) and differential equations etc. I think this is enriching and increases the appropriate use of Mathematica application to the various domains. – ubpdqn Apr 23 '16 at 1:39
  • 7
    +1, keep doing it. As nice as the probability/statistics functions are in MMA (world class, IMO), when a problem goes beyond trivial it almost always benefits from thinking before coding on the best way to utilize those functions, or work around their failings and foibles. Doing it within MMA and not advertising a third-party, non-free application? +10... – ciao Apr 23 '16 at 7:03
  • 2
    I don't always have the time to write a careful explanation of the hypotheses and reasons underlying a Q&A, but sometimes I do. Sometimes the issue has been explained before, and I just don't have the motivation to offer yet another explanation that doesn't really say anything new. (And the +epsilon difference between it and another question means it's not an exact duplicate.) – Michael E2 Apr 24 '16 at 2:51
  • Everybody will agree with you here. However, in practice, they will, almost without exception, jump right away to coding in their answers. And they will then admire such "solutions" between themselves, praising each other for such great resourcefulness, not seeing that, in fact, they don't have a clue about theoretical basis of the MMA functionality in question. There are certain number of exceptions, but what I describe is unfortunately the current state of mentality on this site. I do expect now the worst possible reactions to this comment, which would actually be the proof that it is true. – VividD Apr 24 '16 at 13:47
  • 2
    @VividD I am sorry that you have had a poor experience here. I am just a code monkey and I claim no theoretical basis, though I do try to help and honestly believe that many people have found my code-based answers useful. I have respect for those with a deeper understanding and much appreciation when they share it, as e.g. whuber often does. If you have ideas for how we may attract that kind of member I would be happy to hear them. – Mr.Wizard Apr 24 '16 at 17:07
  • 2
    @VividD Even when I feel the strongest need to sermonize on an answer or comment, I've usually learned something new (and many times amazing) about Mathematica from the suggested code. In fact, having many folks not know some of the assumptions of statistical procedures has kept me in business for many, many years. – JimB Apr 25 '16 at 0:23
  • I vote for sermon. Some of my favorite answers that I have written are exactly that type. – rcollyer Apr 25 '16 at 2:44
  • 1
    @VividD I think you raise important points. It gives me pause for thought: wp.me/p1or7L-ER – ubpdqn 8 mins ago – ubpdqn Apr 25 '16 at 5:49
  • 2
    The question title is a typical example of what statisticians do. Does it mean that (+1) vote is for "to sermon" and (-1) vote is for "not to sermon" ? – Anton Antonov Apr 25 '16 at 13:45
  • 2
    @AntonAntonov. "The question title is a typical example of what statisticians do." Well, someone's got to do it. Because I sense somewhat of a green light to do so from the comments so far, I can continue with some of your instructive posts about fitting. – JimB Apr 25 '16 at 18:16
  • 3
    @JimBaldwin I was implying that you have the mindset of the third statistician in the following joke, trying to use voting instead of shooting: "Three statisticians went duck hunting. A duck flew out and the first statistician took a shot, the shot went a foot too hight. The second statistician took his shot and the shot went a foot too low. The third statistician said, 'We got it!' " – Anton Antonov Apr 25 '16 at 21:39
  • 3
    @VividD if you see answers with theoretical problems, please point them out to the poster. I have yet to see anyone refuse to correct or remove an answer when requested to do so; in fact this collaborative (rather than competitive) spirit seems more common on this site than others. But people can't correct what they don't know about. – Oleksandr R. Apr 27 '16 at 13:00
18

Prompted by Mr.Wizard's invitation/nudging to "pen an Answer" I decided to make some calculations to find the most important comments. It seems that "I vote to sermon. [...]" most clearly summarizes the expressed sentiments.

Definition of importance

Let us use the following definitions:

  1. The most important comments have the most important words, and
  2. the most important words are in the most important comments.

Calculation procedure

Starting with these definitions we do the following:

  1. Convert the comments into points in a linear vector space. Each word is an axis; each comment is a point.

  2. Do appropriate weighting of the comment-word associations (IDF, TFIDF, etc.).

  3. From the importance definitions we can find the eigenvectors of the representation matrix multiplied by its transpose.

  4. The most important comments will have largest coordinates in the eigenvectors.

Results

Here is an image of a Mathematica session with all the steps given above:

enter image description here

If we ignore my comment, the most important ones are :

  1. "I vote for sermon. [...]"
  2. "I have learned a lot from your answers, [...]"

Verdict

These calculations show that commenters want Jim Baldwin to make extensive statistical descriptions in this site. They also show that my comment about the question title has to be addressed.

(I am curios what kind of statistical objections Jim Baldwin might bring to this answer.)

Code

Text data

Get the question comments:

xmlObject = 
 Import["http://meta.mathematica.stackexchange.com/questions/1857/to-sermon-or-not-to-sermon/", "XMLObject"];

comments = 
 Cases[xmlObject, 
  XMLElement["span", {"class" -> "comment-copy"}, {c_}] :> c, Infinity]

Assign stop words:

stopWords = 
 ReadList["http://www.textfixer.com/resources/common-english-words.txt", Word, RecordSeparators -> ","]

Step 1 (vector space)

Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/DocumentTermMatrixConstruction.m"]

stopWords = ReadList["~/MathFiles/stop_words", String];

{cMat, cTerms} = DocumentTermMatrix[comments, {{}, stopWords}];

Magnify[cTerms, 0.7]

cMat // MatrixPlot

Step 2 (weight associations)

wcMat = WeightTerms[cMat, GlobalTermWeight["IDF", #1, #2] &, # &, #/Norm[#] &];

Step 3 (find eigenvectors of M.M^T)

nSentences = 3;

wcSMat = wcMat.Transpose[wcMat];
{vals, vecs} = Eigensystem[wcSMat, nSentences];

Find most important comments

inds = Reverse@Ordering[Abs[vecs[[1]]], -nSentences];
Transpose[{Abs[vecs[[1, inds]]], comments[[inds]]}]

(* {{0.599066, 
  "The question title is a typical example of what statisticians do. \
Does it mean that (+1) vote is for \"to sermon\" and (-1) vote is for \
\"not to sermon\" ?"}, 
    {0.563276, 
  "I vote for sermon. Some of my favorite answers that I have written \
are exactly that type."}, 
    {0.369312, 
  "+1, I'm all for sermon. Answers that give a lot of background are \
the most enjoyable to read."}} *)
  • 3
    Cheeky, you are. :) – J. M. is away Apr 25 '16 at 14:32
  • 5
    I must duly criticize this answer. Clearly, you have not learned that one does not post images, but copyable code. I'd downvote to encourage you to improve your answer, but the sheer cheek amuses me. (yes, sarcasm. Lots of it.) :) – rcollyer Apr 25 '16 at 15:59
  • @rcollyer :) Thanks! I was hesitant to to put copyable code because I used an external package and the assignment of the comments might be lengthy -- I just wanted to provide evidence of the computations. Additionally, does the rule "post copiable code" hold also for the meta posts? – Anton Antonov Apr 25 '16 at 16:57
  • Truthfully, I had misread your code. I thought you were grabbing the comments via a call to Import. I've looked a little at the API queries, and a simple method would have been welcome. :) – rcollyer Apr 25 '16 at 18:34
  • @rcollyer You are completely right -- using Import is much better. I updated my answer accordingly. – Anton Antonov Apr 25 '16 at 19:11
  • 1
    Here I thought the API would be used. Importing html as xml: bonus. I'd give you a +1, but this part of the site is like "Who's Line is it Anway?" the points don't matter. :) – rcollyer Apr 25 '16 at 20:36
  • 2
    Ah, ha! There's a sermonable point in your code. Eigenvalues of $\mathbf{M}.\mathbf{M}^{T}$ has precision issues. Better to take the SVD of $M$ and square the singular values. :D (Yes, at this point, I'm just having fun.) – rcollyer Apr 25 '16 at 20:41
  • 1
    @rcollyer Sigh... I was going for clarity of the code/formula given the definition of important comments. I sacrificed precision for clarity. – Anton Antonov Apr 25 '16 at 21:29
  • @AntonAntonov :) – rcollyer Apr 26 '16 at 2:27
  • @rcollyer, hadda link to that sermon of mine, why don't you? :P – J. M. is away Apr 26 '16 at 2:54
  • 1
    (On a more serious note, it's nice that statisticians are already starting to heed Golub's call to reformulate and stabilize their numerical linear algebra tools.) – J. M. is away Apr 26 '16 at 2:56
  • @J.M. of course I did. :) – rcollyer Apr 26 '16 at 12:41
  • While I see that, in fact, you don't have a clue about theoretical basis of the MMA functionality in question, I praise you for such great resourcefulness... ;-} – ciao Apr 26 '16 at 21:42

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .