Syntax highlighter treats patterns that end with an underscore incorrectly

Question

Before going on, please read the comments here. This used to work in the user script version, but does not anymore in the version served by mathematica.se itself. Basic example:

f[x_]:=x^2

Here the x_ should be highlighted in green because it is a pattern. This does not work anymore. I could track the issue down by comparing the server-side script with my local one.

The problem seems to be, that the script was stripped before installing which should delete unwanted white-space, comments, etc but in fact did some wrong code-transformations. Below is the (reformatted) script which is now on our server. I only deleted the long list of keywords and named characters.

/*
 Mathematica highlighter https://github.com/halirutan/Mathematica-Source-Highlighting
 Licensed under the Apache License, Version 2.0
 */
'use strict';
(function () {
    var a = "\\\\\\[" + "AAcute,ABar,ACup,".split(",").join("\\]|\\\\\\[") +
        "\\]", b = [
        [PR.PR_ATTRIB_VALUE, /^(?:[a-zA-Z\$][a-zA-Z0-9\$]*\s*:)/, null]
    ];
    PR.registerLangHandler(PR.createSimpleLexer([
        [PR.PR_PLAIN, /^[\t\n\r \xA0]+/, null, "\t\n\r \u00a0"],
        [PR.PR_STRING, /^(?:"(?:[^"\\]|\\[\s\S])*(?:"|$))/, null, '"']
    ], [
        [PR.PR_COMMENT, /^\(\*[\s\S]*?\*\)/, null],
        [PR.PR_LITERAL, /^(?:(?:\d+)(?:\^\^(?:\.\w+|\w+\.\w*|\w+)(?:`(?:`?(?:\.\d+|\d+\.\d*|\d+))?)?(?:\*\^[+-]?\d+)?))/, null],
        [PR.PR_LITERAL, /^(?:(?:\.\d+|\d+\.\d*|\d+)(?:`(?:`?(?:\.\d+|\d+\.\d*|\d+))?)?(?:\*\^[+-]?\d+)?)/, null],
        ["mma_iot",
            /^(?:In|Out)\[[0-9]*\]/, null],
        ["lang-mma-usage", /^([a-zA-Z\$]+(?:`?[a-zA-Z0-9\$])*::[a-zA-Z\$][a-zA-Z0-9\$]*):?/, null],
        ["lang-mma-patterns", /^([a-zA-Z\$][a-zA-Z0-9\$]*\s*:)(?:(?:[a-zA-Z\$][a-zA-Z0-9\$]*)|(?:[^:=>~@\^\&\*\)\[\]'\?,\|])).*/, null],
        [PR.PR_ATTRIB_VALUE, /^(?:[a-zA-Z\$][a-zA-Z0-9\$]*)_+([a-zA-Z\$][a-zA-Z0-9\$]*)/, null],
        [PR.PR_ATTRIB_VALUE, /^_+([a-zA-Z\$][a-zA-Z0-9\$]*)/, null],
        [PR.PR_ATTRIB_NAME, RegExp("^(?:" + a + ")"), null],
        [PR.PR_TAG, /^(?:\[|\]|{|}|\(|\))/, null],
        [PR.PR_ATTRIB_VALUE, /^(?:#+[0-9]?)/,
            null],
        [PR.PR_KEYWORD, RegExp("^(?:AbelianGroup|Abort|AbortKernels|AbortProtect|\\$VersionNumber)\\b"),
            null],
        [PR.PR_PLAIN, /^[a-zA-Z\$][a-zA-Z0-9\$`]*/, null],
        [PR.PR_PUNCTUATION, /^(?:\+|\-|\*|\/|,|;|\.|:|@|~|=|>|<|&|\||_|`|'|\^|\?|!|%)/, null]
    ]), ["mma", "mathematica"]);
    PR.registerLangHandler(PR.createSimpleLexer([], b), ["mma-patterns"]);
    PR.registerLangHandler(PR.createSimpleLexer([], [
        ["mma_use", /^([a-zA-Z\$]+(?:`?[a-zA-Z0-9\$])*::usage)/, null],
        ["mma_msg", /^([a-zA-Z\$]+(?:`?[a-zA-Z0-9\$])*::[a-zA-Z\$][a-zA-Z0-9\$]*)/, null]
    ]), ["mma-usage"])
})();

If you want, you can compare it with the original code. The most problematic line which lead to the wrong behavior is this one:

[PR.PR_ATTRIB_VALUE, /^(?:[a-zA-Z\$][a-zA-Z0-9\$]*)_+([a-zA-Z\$][a-zA-Z0-9\$]*)/, null]

In my original code this line goes

[PR.PR_ATTRIB_VALUE, /^(?:[a-zA-Z\$]+[a-zA-Z0-9\$]*)_+([a-zA-Z\$]+[a-zA-Z0-9\$]*)*/, null],

Please note the missing * at the end. This missing star makes, that the part after the underscore(s) _+ is not optional anymore. Therefore, a pattern gets only highlighted correctly when something follows: f[x_Integer] is ok, f[x_] not and it therefore is not matched as pattern.

There are some other transformations the which were done. For instance in the lines containing

RegExp("^(?:" + a + ")")

my IDE tells me that a constructor call without new is potentially invalid.

I'm really not skilled enough to know whether or not this is important. Maybe I screwed up the regexp in the first place which lead to the wrong behavior of the code-stripper. Btw, without my ass-kicking cool java development environment IDEA I would not be able to write a single line of java-script code.

Question: Can someone help to check why the code-stripper does not work like we would expect it?

Answer 1

There are many ways I'd like to fix this, but the way that Prettify does tokenization causes some issues that make my "clever" approaches less than useful. Instead, the easiest fix seems to be to just break the pattern into three explicit parts and rely on the execution order to get the desired result:

[PR.PR_ATTRIB_VALUE, /^[a-zA-Z\$][a-zA-Z0-9\$]*_+[a-zA-Z\$][a-zA-Z0-9\$]*/, null],
[PR.PR_ATTRIB_VALUE, /^[a-zA-Z\$][a-zA-Z0-9\$]*_+/, null],
[PR.PR_ATTRIB_VALUE, /^_+[a-zA-Z\$][a-zA-Z0-9\$]*/, null]

This eliminates the nested repetition that balpha was concerned about, and should restore the original functionality.