7

Before going on, please read the comments here. This used to work in the user script version, but does not anymore in the version served by mathematica.se itself. Basic example:

f[x_]:=x^2

Here the x_ should be highlighted in green because it is a pattern. This does not work anymore. I could track the issue down by comparing the server-side script with my local one.

The problem seems to be, that the script was stripped before installing which should delete unwanted white-space, comments, etc but in fact did some wrong code-transformations. Below is the (reformatted) script which is now on our server. I only deleted the long list of keywords and named characters.

/*
 Mathematica highlighter https://github.com/halirutan/Mathematica-Source-Highlighting
 Licensed under the Apache License, Version 2.0
 */
'use strict';
(function () {
    var a = "\\\\\\[" + "AAcute,ABar,ACup,".split(",").join("\\]|\\\\\\[") +
        "\\]", b = [
        [PR.PR_ATTRIB_VALUE, /^(?:[a-zA-Z\$][a-zA-Z0-9\$]*\s*:)/, null]
    ];
    PR.registerLangHandler(PR.createSimpleLexer([
        [PR.PR_PLAIN, /^[\t\n\r \xA0]+/, null, "\t\n\r \u00a0"],
        [PR.PR_STRING, /^(?:"(?:[^"\\]|\\[\s\S])*(?:"|$))/, null, '"']
    ], [
        [PR.PR_COMMENT, /^\(\*[\s\S]*?\*\)/, null],
        [PR.PR_LITERAL, /^(?:(?:\d+)(?:\^\^(?:\.\w+|\w+\.\w*|\w+)(?:`(?:`?(?:\.\d+|\d+\.\d*|\d+))?)?(?:\*\^[+-]?\d+)?))/, null],
        [PR.PR_LITERAL, /^(?:(?:\.\d+|\d+\.\d*|\d+)(?:`(?:`?(?:\.\d+|\d+\.\d*|\d+))?)?(?:\*\^[+-]?\d+)?)/, null],
        ["mma_iot",
            /^(?:In|Out)\[[0-9]*\]/, null],
        ["lang-mma-usage", /^([a-zA-Z\$]+(?:`?[a-zA-Z0-9\$])*::[a-zA-Z\$][a-zA-Z0-9\$]*):?/, null],
        ["lang-mma-patterns", /^([a-zA-Z\$][a-zA-Z0-9\$]*\s*:)(?:(?:[a-zA-Z\$][a-zA-Z0-9\$]*)|(?:[^:=>~@\^\&\*\)\[\]'\?,\|])).*/, null],
        [PR.PR_ATTRIB_VALUE, /^(?:[a-zA-Z\$][a-zA-Z0-9\$]*)_+([a-zA-Z\$][a-zA-Z0-9\$]*)/, null],
        [PR.PR_ATTRIB_VALUE, /^_+([a-zA-Z\$][a-zA-Z0-9\$]*)/, null],
        [PR.PR_ATTRIB_NAME, RegExp("^(?:" + a + ")"), null],
        [PR.PR_TAG, /^(?:\[|\]|{|}|\(|\))/, null],
        [PR.PR_ATTRIB_VALUE, /^(?:#+[0-9]?)/,
            null],
        [PR.PR_KEYWORD, RegExp("^(?:AbelianGroup|Abort|AbortKernels|AbortProtect|\\$VersionNumber)\\b"),
            null],
        [PR.PR_PLAIN, /^[a-zA-Z\$][a-zA-Z0-9\$`]*/, null],
        [PR.PR_PUNCTUATION, /^(?:\+|\-|\*|\/|,|;|\.|:|@|~|=|>|<|&|\||_|`|'|\^|\?|!|%)/, null]
    ]), ["mma", "mathematica"]);
    PR.registerLangHandler(PR.createSimpleLexer([], b), ["mma-patterns"]);
    PR.registerLangHandler(PR.createSimpleLexer([], [
        ["mma_use", /^([a-zA-Z\$]+(?:`?[a-zA-Z0-9\$])*::usage)/, null],
        ["mma_msg", /^([a-zA-Z\$]+(?:`?[a-zA-Z0-9\$])*::[a-zA-Z\$][a-zA-Z0-9\$]*)/, null]
    ]), ["mma-usage"])
})();

If you want, you can compare it with the original code. The most problematic line which lead to the wrong behavior is this one:

[PR.PR_ATTRIB_VALUE, /^(?:[a-zA-Z\$][a-zA-Z0-9\$]*)_+([a-zA-Z\$][a-zA-Z0-9\$]*)/, null]

In my original code this line goes

[PR.PR_ATTRIB_VALUE, /^(?:[a-zA-Z\$]+[a-zA-Z0-9\$]*)_+([a-zA-Z\$]+[a-zA-Z0-9\$]*)*/, null],

Please note the missing * at the end. This missing star makes, that the part after the underscore(s) _+ is not optional anymore. Therefore, a pattern gets only highlighted correctly when something follows: f[x_Integer] is ok, f[x_] not and it therefore is not matched as pattern.

There are some other transformations the which were done. For instance in the lines containing

RegExp("^(?:" + a + ")")

my IDE tells me that a constructor call without new is potentially invalid.

I'm really not skilled enough to know whether or not this is important. Maybe I screwed up the regexp in the first place which lead to the wrong behavior of the code-stripper. Btw, without my ass-kicking cool java development environment IDEA I would not be able to write a single line of java-script code.

Question: Can someone help to check why the code-stripper does not work like we would expect it?

  • Sorry, cant really help here. However I noticed there are also two +'s missing. – Sjoerd C. de Vries May 8 '12 at 5:18
  • Yes, those transformations are correct afaik. The quotes have been changed form ' to " too where I have no idea why then two kinds exists, when it does not matter. I just used single quotes because I saw it in the other implementations. – halirutan May 8 '12 at 8:27
  • It's not the "code stripper" (the correct term is "minifier"). I removed that asterisk, because nested stars are extremely dangerous. I'll look at preserving your semantics without potentially catastrophic regexes when I get back from vacation. Regarding the missing new, Section 15.10.3.1 of the EcmaScript standard is pretty clear that (at least in this case) the behavior is identical. Closure has been removing that new for two years. Your IDE complaining is not a good enough reason to switch to a different minifier. – balpha May 8 '12 at 13:36
  • The second star should have been a ? anyway, since that subpattern either exists or doesn't exist; it can never repeat in a meaningful way. – Tim Stone May 8 '12 at 13:54
  • Oh, since this isn't your area of expertise, I should mention this: The word "catastrophic" isn't meant to criticize you; it refers to catastrophic backtracking, which is a serious issue to be aware of in regular expressions (it may be a DOS vector). That's why nested (in particular, greedy) repeaters should be avoided wherever possible. – balpha May 8 '12 at 14:54
  • @TimStone Exactly, except that the inner and outer patterns are very similar, which favors catastrophic backtracking. I haven't tried constructing a bad input for this, and it may actually be safe, but I like to err on the side of not risking it. Reasoning about non-trivial regular expressions is bound to miss edge cases. – balpha May 8 '12 at 14:58
  • @balpha Oh, absolutely. I'll try taking a more in-depth look at the expressions later and see if there's a safer solution to propose, in case I can save you some trouble. Too early in the morning for mucking about with regular expressions right now. :) – Tim Stone May 8 '12 at 15:04
  • @balpha I looked up some code-minifiers yesterday and I saw, that they never did such transformations and I then assumed, that you cleaned the code. @TimeStone is completely right, the star should be a ?. Thanks for looking into this guys. – halirutan May 8 '12 at 17:06
  • 'catastrophic backtracking' is interesting in the context of Mathematica as well (since pattern matching is a big part of Mathematica). In Mathematica we don't typically have very complex patterns though. – Szabolcs May 15 '12 at 7:43
6

There are many ways I'd like to fix this, but the way that Prettify does tokenization causes some issues that make my "clever" approaches less than useful. Instead, the easiest fix seems to be to just break the pattern into three explicit parts and rely on the execution order to get the desired result:

[PR.PR_ATTRIB_VALUE, /^[a-zA-Z\$][a-zA-Z0-9\$]*_+[a-zA-Z\$][a-zA-Z0-9\$]*/, null],
[PR.PR_ATTRIB_VALUE, /^[a-zA-Z\$][a-zA-Z0-9\$]*_+/, null],
[PR.PR_ATTRIB_VALUE, /^_+[a-zA-Z\$][a-zA-Z0-9\$]*/, null]

This eliminates the nested repetition that balpha was concerned about, and should restore the original functionality.

  • Sounds reasonable. – halirutan May 14 '12 at 11:21
  • You probably know the inner workings of prettify better than I do, so I trust your judgement on this. Added to the next build. – balpha May 18 '12 at 8:35
  • @balpha Can you let us know when this goes live, so we can see if the problem is gone? – Szabolcs May 19 '12 at 9:23
  • @Szabolcs It's live now. – balpha May 21 '12 at 19:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .