Ok so there is clearly not a way to have SE let you know by email immediately when an answer is posted to a question, but I have written (not really super proud of this yet but it works) a grep tool where you can paste the URL to the question, the frequency that you want *Mathematica* to check, and the number of times you want *Mathematica* to keep checking. Although it aborts well on discovery of a new answer, as well as manually, it seems sort of trashy to me to not have a nicer abort button than "Evaluation->Abort Evaluation", or "Alt +". 

The grep does export then import to and from a flat file on your default local. 

Anyway, here is my code, feel free to improve:

Set `cnum` to the current number of answers (default 0).

Set `checktimer` to the frequency that you wish to probe this site (careful here, don't slam SE).

Set `stopcheck` to the number of times you you wish to probe. (all currently set to every 5 minutes for 1 hour)

    grep[file_, patt_] := 
    With[{data = Import[file, "Lines"]}, 
    Pick[Transpose[{Range[Length[data]], data}], 
    StringFreeQ[data, patt], False]]
    link = "http://meta.mathematica.stackexchange.com/questions/1293/get-notified-of-answers-to-a-question";
    cnum = 0; (*current number of posts*)
    checktimer = 300; (*check every number of seconds*)
    stopcheck = 12;(*stop the loop after stop multiplied by checktimer seconds*)

The `grep` function should seem pretty clear, I used it in the past spidering web sites (probably ripped it off somewhere long ago).

Here is the implementation:

    For[i = 1, i <= stopcheck, i++,
    Pause[checktimer];(*seconds*)
    Export["segrep.txt", Import[link, "Plaintext"]];
    If[Length[grep["segrep.txt", {"improve this answer"}]] > cnum,
    SendMail[
    "To" -> "[email protected]",
    "Subject" -> "Stackexchange!",
    "Body" -> "You have a new Mathematica answer on stackexchange!",
    "From" -> "[email protected]",
    "Server" -> "smtp.gmail.com",
    "UserName" -> "[email protected]",
    "Password" -> "your-outbound-password",(*<<<---Needs Password, else add in prompt*)
    "PortNumber" -> 587,
    "EncryptionProtocol" -> "StartTLS"];
    Abort[];
    ]
    ]

 You clearly need to put your two email addresses and password inside the quotes correctly. If you use a gmail outbound to your phone email push, then the rest will probably work as is (otherwise alter).

Change the URL, and change `checktimer` and `cnum` to some small numbers for your test. Don't slam the site.

P.S. The hardest part of this was finding the distinguishing difference between a question with n versus n+1 answers in any SE community. It turned out that grepping "improve this answer" did the trick. 

On to the final test...