Operator overloading and custom operators

[zolomatok]

This is a feature request rather than an issue. Originally I wanted to inquire about macro support, but to be honest what I'm mostly looking for is operator overloading and custom operators.

Swift has a wonderful feature where you can overload prefix, infix and postfix operators or create new ones to your heart's content and it's the best. It has the potential to make the code very succinct so such feature I believe is such an obvious fit in Civet and CoffeeScript.

For example, for a long time, Swift had an awfully verbose regex matching API, but I could just define the ~= operator on String and I could express with two characters that otherwise would have taken 4 lines to do:

url ~= ".*/account/.*/login"

Or how about using operators for matrices? Makes so much sense.
Or how about some reactive code where you could just add together multiple observables to get a combined signal?
Or how about a language that uses := as a readonly shorthand? 😉

Mathematical operators is one of the best known and most succinct language we have as humans and I feel like it's such a waste to use them only for numbers.

I would really really love a Swift-like solution. 🚀

[zolomatok]

I originally wanted to request macros and the issue was titled Of Macros and Men 😄
But I'm mostly just looking for operators. Although, replacing console.log with log would also be a
very welcome capability.

[STRd6]

@zolomatok to replace console.log with log you can do log := console.log or {log} := console currently.

I think some kind of operator customizing could be useful. It's a big feature and will take a bit of thought. One big difference is that Coffee/ES/JS is duck typed so it's not as easy to attach an operator to a type or a class. It might be more possible with TypeScript integration.

There's already a little bit of similar features with a in b becoming b.indexOf(a) >= 0 in Coffee and x[a...b] becoming x.slice(a, b). Though those are built in to the language and are not customizable (yet). Feel free to keep adding ideas and suggestions to this issue and let's see where it goes!

Thanks for the feedback!

[zolomatok]

@STRd6

to replace console.log with log you can do log := console.log or {log} := console currently.

ah, you're completely right, of course, thanks, awesome!

I think some kind of operator customizing could be useful.

It was an absolute joy to learning about op overloading when I was learning Swift. Operators lend so much legibility to the code compared to whatever handwaving one needs to do in leu of them. Not only are they infinitely less verbose than calling functions on objects, they provide much better semantics. I remember try a reactive library in some language where combining observables went something like (pseudocode):

first.combine(second).listen -> ... but of course since the order of operations make no difference, you could also do
second.combine(first).listen -> ...

and this kind of method calling suggests a kind of hierarchy between first and second where there is none. It's much more semantically correct, way more succinct and takes less time to visually parse if we just do:

(first + second).listen -> ...

This kind of friendly, concise syntax is right in line with Civet and Coffee I think.

One big difference is that Coffee/ES/JS is duck typed so it's not as easy to attach an operator to a type or a class.

Could you please elaborate on why that makes it harder? I don't know that much about the intricacies of compilation.

[edemaine]

Could you please elaborate on why that makes it harder? I don't know that much about the intricacies of compilation.

At compilation time, we generally have no idea whether first and second (say) are numbers or strings, so + should compile to +, or a special object that defines its own + operator, so + should compile to calling that function. So without some clever use of the TypeScript type system or something, we'd need to compile it to a run-time check:

(first + second).listen -> ...

might become

(first.__plus__ ? first.__plus__(second) : first + second).listen(function () { ... })

So there would be runtime overhead for this, but I think it would be great to add as a togglable option via "civet operatorOverloading" at the top of the file. And maybe there are some cases where we can avoid the runtime check, e.g., when we know that the type of first requires a __plus__ property.

(By contrast, in statically typed languages, the exact type of first is known at compile time, so there's no need for such runtime checks.)

[zolomatok]

Ah, makes sense, thanks @edemaine !

[STRd6]

This is definitely in scope for Civet and I think I have a decent idea of the specification of how things could work.

This will reconcile and improve the CoffeeScript/JS in fiasco.

Here's some examples that I thought about for discussion.

operator in<T>(a: T, b: Array<T>): boolean {
  b.includes(a)
}
operator in<T>(a: T, b: Set<T>): boolean {
  b.has(a)
}

declare const a: Array<number>
declare const s: Set<number>
declare const o: Object
declare const x: number

x in a // => a.includes(x)
x in s // => s.has(x)
x in o // => x in o

// Add a scalar to each element of an array
operator +(a: Array<number>, b: number): Array<number> {
  a.map &+b
}
// Append an item to an array, returning the array (like Ruby)
operator <<<T>(a: Array<T>, b: T): Array<T> {
  (a.push(b), a)
}

// tuple add
operator +(a: [number, number], b: [number, number]): [number, number] {
  [a[0] + b[0], a[1] + b[1]]
}

// unary overload
operator &(a: string): <T extends {[key: string]: any}>(x: T) => T[string] {
  (function(x) { return x[a] })
}

Overloads will currently only be defined per file. They will be checked bottom to top, the first matching one will apply so generic overloads can be declared above more specific overloads.

Overloads cannot change precedence, associativity, or add new operators that don't exist in JS.

Notes

---

• https://github.com/tc39/proposal-operator-overloading
• Why not operator overloading in Typescript? microsoft/TypeScript#6936
• Operator Overloads microsoft/TypeScript#5407
• Suggestion: Infix operators/functions microsoft/TypeScript#2319

[zolomatok]

This is definitely in scope for Civet and I think I have a decent idea of the specification of how things could work.

luv it!! 🌈

Overloads cannot [...] add new operators that don't exist in JS.
operator <<<T>(a: Array<T>, b: T): Array<T> {

Does << exist in JS?

[edemaine]

operator+ is a nice idea for notation! I'm having C++ flashbacks. 😅

The obvious sensitivity here is, if you're relying on the type system, you might not call the correct operator if you don't have types set correctly. But I guess you'd likely get the wrong type output as a result (e.g. vanilla + usually returns number | string) so you'd hopefully detect such errors at type checking time.

The other obvious big thing is that Civet would need to actually talk to TypeScript during compilation phase in order to know types while transpiling. Currently, only the language server does this, so it's a big structural change to Civet. But it does seem exciting what it could enable... On the other hand, it makes me wonder: should these features be added to TS before Civet?

One important detail is how to import operators from other files. The TC39 spec has a weird with operators from but I imagine something like

import {operator+, operator<<} from './foo.civet'

(or the same without import) would make sense? I assume import {operator+} imports all operator+s in the file, and it's impossible to import just a specific operator+. Incidentally, I wonder whether this should be a new TC39 spec? Ah, no, it only makes sense in TS.

Overloading

One unusual thing here is that we're overloading functions. In TS, function prototype overloading already exists. But it's unusual (even in TS) to have multiple function bodies for different prototypes. I agree that it makes special sense for operators, because there's only one + but lots of different types of things to add. But it does make me wonder whether it would make sense to add function overloading more generally... (and possibly as a first step toward operator overloading?) For example:

function combine(a: number, b: number): number {
  return a * b;
}
function combine(a: string, b: string): string{
  return a + ' ' + b;
}
combine(3, 3)  // 9
combine('hello', 'world')  // 'hello world'

Compilation would turn this into two functions, combine1 and combine2, and call the appropriate one. This kind of highlights how challenging this is going to be across multiple files with imports and exports. Even if we don't have function overloading, operators will have all the same issues.

Identifier Operators

I really like the idea of overloading in. It makes me wonder whether any identifier could be used as a binary operator, presumably preventing it from being used as a unary operator (as all functions are by default). For example:

operator union<T>(a: Set<T>, b: Set<T>): Set<T> {
  return new Set<T>(...a, ...b);
}
declare const a: Set, b: Set
const c = a union b
// now invalid: union a, b

I assume operator declarations are always global to the file so should only be done at the top level (e.g. not within a function — not even sure what that would mean), just like exports. And they can optionally be exported.

Does << exist in JS?

Yes, it's shift left from C.

[edemaine]

Combining my earlier ideas of runtime type checking with Daniel's latest, here's a possibility for operator redefining (not overloading) that might be an interesting first step toward operator overloading (which IMO is as hard as function overloading, a significant task):

// Redefines the meaning of + in this file.  Untyped version.
operator+(a, b) {
  if Array.isArray(a) and Array.isArray(b)
    return [...a, ...b]
  else
    return a + b
}

// TS types for above
operator+(a: Array, b: Array): Array
operator+(a: number, b: number): number
operator+(a: string, b: number): number
operator+(a: number, b: string): number
operator+(a: string, b: string): string

The idea here is that there can be only one operator+ in any file (imported or locally defined), and that replaces +. The user is responsible for dispatching the correct operator at runtime, but at least the syntax is redefinable. And you could provide your own in, or perhaps other identifiers like union.

If there are multiple libraries that offer their own operator+, they could perhaps be combined via import...as:

import {Vector, operator+ as vectorAdd} from './vector.civet'
import {Magic, operator+ as magicAdd} from './magic.civet'
export operator+(a, b) {
  if a instanceof Vector or b instanceof Vector
    return vectorAdd(a, b)
  else if a instanceof Magic or b instanceof Magic
    return magicAdd(a, b)
}

I'm not claiming this is convenient, but it's a much easier first target on the way to full operator overloading, and could already be quite useful, especially for projects that only want to redefine each operator once (e.g. they just want Vectors).

[STRd6]

I think untyped, per file redefinitions would still be most of the work to implement but lack a lot of the benefits of mixing types and operators (multiply a point through a matrix, add a scalar to a point). We wouldn't necessarily need to hook into the entire TS system for the initial version. A very basic implementation would match literal string to string for types and not know anything about TS at all (just might need to cast explicitly if your type is more complex)

I also think the implementation of operators would be closer to macros than to functions so overloading the function type wouldn't quite make sense. My hunch is that it is a direct transformation on the array of binary op and expression tokens. Basically starting at the highest precedence operators, checking for operators defined on the explicit exact string match type, rewriting the tokens substituting in the "return type" string and then repeating at the next layer up.

For Identifier Operators I'd rather add additional symbol operators first. I'm still on the fence but adding custom operators by string names could make sense then and and or and isnt would be instances of this type of extension.

Additional note: comparison operators should still chain:

operator <=(a: Set<any>, b:Set<any>) {
  Array.from(a).every(x => b.has(x))
}

declare const s1: Set<any>, s2: Set<any>, s3: Set<any>

s1 <= s2 <= s3 // Array.from(s1).every(x => s2.has(x)) && Array.from(s2).every(x => s3.has(x))

Another fun one to consider:

operator +(...objs: Object[]) {
  Object.assign(...objs)
}

declare const a: Object, b: Object, c: Object

x := a + b + c // const x = Object.assign(a, b, c)

[edemaine]

Just to update this issue, Civet now has a simple form of custom infix operators (no overloading). Check out the docs. It's designed to be compatible with the future according to the above proposals for e.g. operator +.

[zolomatok]

Thanks for the heads up @edemaine, this is pretty freaking awesome! 🚀

[al6x]

One big difference is that Coffee/ES/JS is duck typed so it's not as easy to attach an operator to a type or a class. It might be more possible with TypeScript integration.

TypeScript integration is an ideal solution, but hard to implement. There's also hacky, but much simpler solution - alter JS runtime. a) Convert every operator to function call, and b) add those functions to built-in JS objects. Like 1 + 2 in Civet will be 1..plus(2) in Compiled JS. And define Number.prototype.plus = (x, y) => x + y. This is hacky and considered no-no in JS, but it may worth to try because a) it's possible make it optional with flag and b) it's easy to try and see if it indeed provides great benefit and decide to invest in that approach more, possibly evolving it into safer approach with TS types.

Actually, to do that there's no need to even modify Civent, it could be implemented as post-Civet JS transformation, with babel transformer code.civet --CivetCompiler-> JS --BabelTransformer-> JS. But the problem is - types, I don't know how to support TS types that way.

[zolomatok]

@edemaine

Just my annual, gentle probing into the topic of arithmetic operator overloading and the defining of custom ones. :)
Is it still in the project's scope and aspirations?
If so, has there been maybe some adjacent work done that brought them closer to being implemented?

[zolomatok]

Thank you for the thorough update @edemaine ! 🙏
Great to see things are humming along! Happy to see the other proposal!

[zolomatok]

Civet is the most pleasant language to use ever imho I love it so much 👍

[bbrk24]

There's a recent proposal (#1382) to extend this support to characters that are not valid for identifiers, to define custom operators like +. or m+ or something like that. This is relative easy to implement so will probably get done relatively soon. It would probably also let you globally redefine + so that you could do your own function dispatching depending on the types of arguments (including e.g. check for __add__ methods), but that's going to be on the slower side.

How would this work? The new definition of + would have to call the original JS builtin + somehow if the arguments were numbers or bigints, and making it so that operators can't be recursive would almost certainly break existing code (or at least be surprising).

[STRd6]

How would this work? The new definition of + would have to call the original JS builtin + somehow if the arguments were numbers or bigints, and making it so that operators can't be recursive would almost certainly break existing code (or at least be surprising).

It could be possible with type annotations. Like +<Vec2, Vec2> could be different and take precedence over+<any, any>

There's a long term goal where we hook into TS more fully but in the meantime it could just be for explicit exact match annotations.

[edemaine]

Good point that defining a custom operator (+) would shadow JS's +. There's a hacky workaround: copy the old operator before declaring the new one.

jsAdd := (+)
operator (+)(a, b)
  switch a
    <? "number"
      jsAdd a, b
    //...

Slightly less ugly version:

operator (JS+) := (+)
operator (+)(a, b)
  switch a
    <? "number"
      a JS+ b
    //...

Especially in the latter case, I think we could reasonably compile a JS+ b to a + b instead of JSplus(a, b), so there's no loss in efficiency. This seems pretty reasonable...

Alternatively, we could add some sort of escape hatch that gives access to the original operator. Something like CoffeeScript's ` could be one approach, or maybe there's a special notation like original:+ or JS:+.