better printing and syntax for strange symbols

[StefanKarpinski]

We do a lousy job of printing symbols that aren't valid identifiers:

julia> symbol("foo bar ")
:foo bar

This is relatively easy to fix by printing symbols that aren't valid identifiers like this:

julia> symbol("foo bar ")
symbol("foo bar ")

Speaking of which, do we have an easy function for checking if a symbol is a valid identifier? That's become and increasingly tricky thing to decide.

We could also support a better input/output syntax for weird symbols: :"foo bar ". Using this syntax is ok since there's no point in quoting a string literal like that – it has the same effect as just writing "foo bar " currently, so is effectively wasted syntax.

This issue was spawned by these conversations:

• insistence on valid identifiers for explicitly given column names JuliaData/DataFrames.jl#617
• why  JuliaData/DataFrames.jl#618.

[johnmyleswhite]

My failed attempt at writing code to identify identifiers is here: https://github.com/JuliaStats/DataFrames.jl/blob/master/src/other/utils.jl#L17

[jakebolewski]

All the code to do this should be part of the Lexer / Tokenizer in JuliaParser.jl.

[StefanKarpinski]

This is remarkably hard to even figure out how to change. The display code is such a tangled web these days; it really needs some serious cleaning.

[StefanKarpinski]

cc: @JeffBezanson, @johnmyleswhite

[JeffBezanson]

The lisp approach would be to make :"a" proper lexical syntax for a symbol, so that

   :"#" = 0

would actually initialize a variable called # in your program. However I'm not sure we want that. In particular, such symbols are hard to quote. You'd have to write :(:"#"), and I suspect most real uses of this would want quoted symbols.

Therefore I'd propose a custom string literal, s"#" or sym"#", short for symbol("#"). This has the advantage of not needing parser changes and not further overloading colon.

[jakebolewski]

I like the s"#" string literal syntax, seems nice and compact.

[StefanKarpinski]

I think I'd prefer sym"#" since this strikes me as rare in everyday usage, but I guess the brevity in printing ASTs might be good. But this issue is not really about having syntax for strange identifiers but for strange symbol literals. I.e. we don't need to be allowed to write :"foo bar" = 1 any more than we can write :foo = 1 – i.e. these should both result in "syntax: invalid assignment location".

[johnmyleswhite]

I agree: I don't think one should allow crazy identifiers, just allow construction of symbols that are not valid identifiers.

[stevengj]

In general, we need to export a number of things from the Scheme parser: a predicate for testing whether a string is a valid identifier, the list of binary operators and their precedence, and probably some other things.

That said, I'm extremely skeptical that this should block 0.3.