Syntax highlighter problem reports (please test!!)

Question

Thanks to @halirutan and @TimStone, the syntax highlighter is now considered to be finished. We just need some testing before we can start talking about integrating it into the site.

This is a collectively edited collection of problems reports.

Click here to install the syntax highlighter userscript!

Please test the highlighter, and report any problems you find here! This will eventually become part of the site.

How to install:

• On Chrome, just click the link above to install
• On Firefox, first install Greasemonkey, then just click the link above
• For other browsers, Google "installing userscript in [Browsername]", and after you managed to install the script, update this post with instructions.

You can use my post below as a guideline on how to report problem.

Answer 1

Using version, bb43c21bacad8fca9e42a885c24906e8366fba36, on this question reveals a few inconsistencies with highlighting patterns in the function definition. Consider:

As you can see, only the latin alphabet is highlighted, and neither the greek nor the numeric characters are.

Answer 2

[I don't know whether to post this, or whether this is the right place - couldn't find any mention of it - but I'm sure you'll tell me either way... :)]

@halirutan's cool script works fine in a Safari Extension on MacOS X (Safari version 5.0 and over, probably).

Assuming you know how to build Safari Extensions ... all you have to do is the following, in Safari Extension Builder:

1 Set the domain pattern:

2 Set the script:

3 Optionally, add any style modifications to a CSS file. I quite like orangey-red keywords and less heavy parentheses, so:

/* keyword */
.kwd
{
color: #dd3300  !important;
}

/* tag */
.tag
{
color: #999 !important;
font-weight: lighter !important;
}

and add the CSS file in Safari Extension Builder:

5 Finally, load the extension. You'll see all Mathematica code highlighted:

(I haven't tested it all yet. Later, perhaps... :)

Thanks for making this possible!

Answer 3

Problem: Some system symbols starting with a $ sign, such as $TemporaryDirectory, are rendered in blue. They should be black.

• version: bb43c21bacad8fca9e42a885c24906e8366fba36 (retrieved from here)

• browser: tested in Firefox 10 and Chrome

• sample code demonstrating the problem:

  FileNameSplit[$TemporaryDirectory]
  

• screenshot (test using the editor's preview):

  

Answer 4

Problem with greek letters in patterns

This issue is based on the fact that the match for valid variables and identifiers is restricted to ASCII characters:

 [a-zA-Z\$]+[a-zA-Z0-9\$`]

This matches all identifiers not starting with a number or a context separator but can otherwise contain letters, numbers, $ and `. Since this regex is used very often, it should be reasonable short. It is used not only to match separated identifiers, but to help match identifiers in patterns with _ and :, in usage and other messages. The matching of usage messages already needs three instances of this regex.

We have a similar issue with the named-characters:

Since I wanted to have this green too, I tried to include the named-characters in the match of patterns. The performance dropped noticeable, which lead to finally leave this out from the match.

Since the identifier-regex is used so often I would really like to restrict it to ASCII characters and, while being aware of this, accept that there are sometimes wrong colorings when people insist to post code like in this question.

Comment on comment

Is it possible to write the pattern regex as /\S+/, per this site? Would that be sufficient, in this case to capture both [Gamma] and α_ alongside the standard ASCII characters?

Yes, this would be sufficient to match both. On[msg::sarcasm] And even more, it would hapily match the whole left side of _ up to the next whitespace Off[msg::sarcasm]. Exactly there lies the dilemma, because what we try to do is to parse an expression without parsing, which means we are only guessing.

Let me explain this in more detail, because maybe I'm wrong and there is a way to do it. Without the fancy argument-pattern-matching, the thing before the underscore is often a literal, the underscore is an operator and the stuff right from the underscore is often a mix of braces, literals, keywords and operators like e.g. in blub_?MatrixQ.

To implement, that this is matched as green argument pattern, the match must take place before all the other stuff, which means the rule for the argument-pattern-thingy is tested really early. Other rules which have to be tried very early in the process are for instance strings and comments, because a comment runs over everything until it sees the closing *).

Assume our matcher just started a new line and sees the following code:

VeryLongFunction[\[Gamma]_?NumericQ]

It's clearly not a string or a comment, but what happens if it would try the rule /\S+_/ saying "match one or more non-whitespace-characters until you find an underscore"? Right, it matches everything from the start. That's why you have to restrict the regular expression as much as possible. Even when one only includes the brackets and the backslash, this would make a match beyond the function-brackets possible.

In the case above, there would be a solution, as already pointed out. You could take the list of all named-characters and test every thing from \[Alpha]_ to \[Zeta]_ separately, which is far too slow.

To say it clearly: The weak points of the highlighter are the rules, that try to guess the context. These are the argument-patterns _ and :, context expressions like Developer`PackedArrayQ, the::usages and the mess::ages. They work pretty well in practice, but it is always possible to crash them.

Problem with system variables rendered in blue. (Fixed)

That was a mistake while I exported the list of keywords. This can easily be fixed.

Test Zone

As I see now, on meta.mathematica the code highlighting is not turned on. So if you want to test the code below, you could paste it into a new "Ask Question" on mathematica.stackexchange and watch the preview

@Szabolcs was so kind to create some test-cases for the pattern-highlighting which uncovered some bugs. These should be highlighted correctly now.

• Please if you find unusual highlighting do not wait to ask me!
• Note we do not make a difference between a pattern and a default value since this would include more super-power than we have available in the lexical scanner.

Valid patterns:

a:b
a_Integer:b
a_:b
a:123
a:(12)
a:{12}
a:`b   
a:!x
a:#
a:$
a:%
a:_
a:\[Alpha]
a:\    
a:-2  
a:+2  
a:.2  
a:;;2 
a:<<file
a:"string"
a: "string" 

Invalid patterns:

a::b

a:=b

a:>b

Illegal syntax (which is now not highlighted as pattern):

a:~
a:@
a:^
a:&
a:*
a:)
a:[
a:]
a:'
a:?
a:,
a:|

Other stuff (which I will not take care of)

(a):b  (* edge case, I suggest not bothering with it *)

Answer 5

I used angle brackets (i.e. AngleBracket) here and I noticed that they are colored blue rather than black. I think all bracketing characters should be colored black.