Syntax Highlighting Improvements

[ehuss]

I've done a survey of syntax issues of some things I think can be highlighted better. Syntax highlighting can be subjective so it's not always clear how things should work. I'm thinking it would be preferred to try to address issues with individual PRs rather than trying to do massive changes, however, that might cause a lot of merge conflicts and may be a pain for @jasonwilliams to review (thoughts?). @dten would you be willing to help, or at least review changes or make suggestions?

• 1. type-any-identifiers not referenced.

• 2. Global const and static values don't show up in goto symbol.

  const VERSION: u32 = 1;
  static NEXT_ID: AtomicUsize = ATOMIC_USIZE_INIT;

  Const identifier should include entity.name.constant. (and possibly variable.other.constant.?) and add entity.name.constant to RustSymbols.

  I'm not sure what scope static identifiers should have. Most other syntaxes don't seem to have special support.

• 3. Turbofish calls don't get colored the same as regular calls.
  They get assigned meta.path.rust instead of support.function.rust.

  foo::<Vec<_>>();
  // vs
  foo();

• 4. mut highlights weird when part of the type of a trait impl.

  impl<A> Thing for &'a mut A {}

  I think it should be storage.modifier, not part of the name.

• 5. Function and closure parameters don't highlight irrefutable patterns as regular parameters. I think it looks weird.

  let c = |(foo, bar)| {};
  fn f((a, b): (i32, i32)) {}

• 6. Newlines in closure arguments breaks highlighting.

  let c = |foo,
           bar| {};

• 7. Comments in strange places doesn't always work.

  let c = |foo,  // weird, but should work
           bar| {};

• 8. Macro calls should not break highlighting on unusual tokens. It would be nice to retain some Rust-like expression handling within macros since that is the common case, however, it shouldn't break down when given invalid syntax since macros can contain almost anything. I hit this most frequently with serde's macro where the struct keyword breaks highlighting until the next block closes.

  forward_to_deserialize_any! {
      f32 f64 char str bytes
      byte_buf unit unit_struct
      struct enum identifier ignored_any
  }

• 9. Add support for dyn trait object type keyword. See stabilize dyn Trait in Rust 2015 rust#49218 for some valid/invalid examples.

• 10. Hex codes above 0x7f are invalid in string escapes.

  let s = "\xff"; // should be invalid

• 11. Unicode escape sequences can include underscores.

  let s = '\u{10_FFFF}';;

• 12. Tuple indexes show up with float highlighting. I don't think the dot should be highlighted (like a field access expression), but I'm uncertain if the number should be highlighted (probably not?).

  t.0;

• 13. super and self when used as a path highlight differently, although they are generally the same thing. I think they should be the same (although self should probably stay as variable.language.rust when used as a method parameter/variable).

  pub(super)
  pub(self)
  super::foo
  self::foo
  use std::collections::hash_map::{self, HashMap};
  use super::*;

• 14. Support attributes for type or lifetime parameters. (48848)

  unsafe impl<#[may_dangle] T: ?Sized> Drop for Box<T> { }

• 15. where clauses tend to avoid highlighting much, leaving large blocks of unhighlighted text. Here are some examples, but there are probably many more.

  fn f<F>(func: F) -> usize
      where F: Fn(usize) -> usize {}
  fn f<L, R>(lhs: L, rhs: R)
      where L: IntoIterator<Item=(&'a i32, &'a i32)>,
            R: IntoIterator<Item=(&'a i32, &'a i32)>, {}
  // Bounds in generic parameters seem better.
  fn f<F: Fn(usize) -> usize>(func: f) {}
  fn f<L: IntoIterator<Item=(&'a i32, &'a i32)>>(lhs: L) {}

• 16. for as a trait bound does not highlight as a keyword in a few situations.

  fn f<F: for<'c> Fn(&'c mut Self)>(&mut self, f: F) {}
  // `unsafe` doesn't correctly highlight here, either.
  fn f(a: for<'a, 'b> unsafe fn() -> String) {}

• 17. Visibility modifiers on struct/union fields doesn't seem to support anything besides plain pub.

  struct S {
      // `crate` should highlight here.
      pub(crate) f1: i32,
  }

• 18. enums don't highlight any generics.

  enum E<'asdf> {}
  enum C<T> where T: Copy {}
  // many other examples

• 19. Trait definitions don't highlight bounds or type parameters.

  trait Foo<'a>: Sized {}
  trait IntoCow<'a, B: ?Sized> where B: ToOwned {}

• 20. Opt-out trait impls highlight weird.

  impl !Send for MyStruct {}

• 21. Type qualified paths highlight as less-than/greater-than operators when they shouldn't.

  let some_constructor = Some::<i32>;
  let push_integer = Vec::<i32>::push;
  let slice_reverse = <[i32]>::reverse;

• 22. break/continue labels are highlighted as lifetimes, but probably should be entity.name.label.

  'outer: loop {
      break 'outer;
  }

• 23. (2018) Anonymous lifetimes

  Foo<'_>

• 24. match-beginning-vert (RFC #1925)

  match foo {
      | A | B | C | D => (),
  }

• 25. Macro blocks need to support semicolons between matches.

  macro_rules! c {
      ($a:ident) => (
      );
      // Everything after the previous semicolon is broken.
      ($a:ident = $e:expr) => (
      )
  }

• 26. Lifetime bounds on a trait do not highlight as a lifetime.

   trait Foo: 'static {}

• 27. Lifetime parameter in enum does not highlight as a lifetime.

  enum Foo<'a> {}

• 28. Support unreserved keywords in 1.28. pure, sizeof, alignof, offsetof. And proc from 1.27.

• 29. Add lifetime specifier to macro_rules! (1.28)

  macro_rules! m { ($lt:lifetime) => {} }

• 30. Raw const pointer in function parameter is not highlighted correctly.

   fn f(a: *const u8) {}

• 31. Raw pointer types still aren't quite right. *const and *mut should be the same. I think mut should be storage.type in this case. Also, it looks like *const is getting confused with const FOO, so the type immediately after *const is wrong.

   let p: *const T;
   let p: *mut u8;

• 32. Closures fail to highlight in some positions. In this example the second closure fails.

  foo_or_else(|x,y| x+y, |a,b| a*b);

• 33. type aliases do not properly handle generic arguments.

  type Foo<i32> = Bar<i32>;

Grammar references:

• https://doc.rust-lang.org/nightly/reference/
• https://brauliobz.github.io/rust-reference/grammar.html
• https://github.com/rust-lang/rust/tree/master/src/grammar
• https://github.com/jorendorff/rust-grammar/
• https://github.com/nikomatsakis/rustypop
• Canonical Rust grammar distinct from parser (tracking issue for RFC #1331) rust#30942

Currently tested on Rust Enhanced 2.11.0.

[TheIronBorn]

Wow! Nice work. Thanks for putting this together. Now to solve them

[dten]

Really nice list. Quite a range on difficulty.

[jasonwilliams]

Awesome work putting this together!
And yes you’re right, individual PRs would be preferable so they’re easy to review

[jasonwilliams]

@ehuss it might be an idea to create some issues which are "good first contribution" bugs and we can mark them as easy

[ehuss]

I added numbers to the list. If anyone wants to work on any, be sure to comment here which ones. I'll probably start working through the list sometime in the upcoming week.

[TheIronBorn]

I'll take on # 20

[TheIronBorn]

Turns out # 20 could be as simple as adding

!?

to impl-definition

[dten]

got another one

fn is_base64_char(c: u8) -> bool {
    match c {
        | b'A'...b'Z' | b'a'...b'z' | b'0'...b'9' | b'/' | b'+' | b'=' => true,
        _ => false,
    }
}

the first | is valid recently but it tries to highlight as a closure

[TheIronBorn]

Another:

fn foo(x: [T; 1 << 6])

[Lucretiel]

Request (feel free to tell me if this is more appropriate as a separate issue):

31: support Hashed Syntax Highlighting for identifiers (https://www.sublimetext.com/docs/3/color_schemes.html#hashed_syntax_highlighting). Essentially you define a range of colors, and sublime will uniquely assign a color to each identifier, making them easier to distinguish from each other while still being collectively distinct from other syntax highlighting. There's an example here: https://www.sublimetext.com/blog/articles/sublime-text-3-point-1

[ehuss]

@Lucretiel I believe the hashed color highlighting is part of your color scheme, not the syntax definition.

[Lucretiel]

Oh, my apologies! Will check it out.

…

On Fri, Dec 7, 2018, 5:36 PM ehuss ***@***.*** wrote: @Lucretiel <https://github.com/Lucretiel> I believe the hashed color highlighting is part of your color scheme, not the syntax definition. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#284 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABvSrbSAbjhnRtIaDwTMh4g0qO49oNvMks5u2xeCgaJpZM4UOuYW> .

[Boscop]

I noticed, const generic args cause the closing > to be highlighted like an operator instead of "closing >":

[s0lst1ce]

I find that dyn being highlighted as a type when used in a result weird. Shouldn't it always be recognized as a keyword? The following is an example:

Result<(), Box<dyn Error>>

[marcospb19]

32. Closures fail to highlight in some positions. In this example the second closure fails.

Here's an example where only the first one gets highlighted.

[Boscop]

The trait alias syntax trait Foo = Bar; is also not working. It causes following lines to not be highlighted correctly:

[ehuss]

@Boscop Unstable/unfinished nightly features are tracked in #333.

In general, it is more helpful to file new issues for syntax issues, as they will likely be lost in the sea of comments. This issue was just to help organize problems found during a top-down review; I don't particularly want to keep adding to it.