Strong mode const semantics don't allow for compile time canonicalization

[leafpetersen]

Moving this here from dart-archive/dev_compiler#503 since it's really a language issue rather than a DDC issue. See discussion there for some background and examples.

Because Dart 1.0 is defined to allow compile time canonicalization of constant expressions, const expressions are not allowed to be instantiated with type variables. This code is therefore invalid:

class A<T> {
  const A() : x = const <T>[];
  final List<T> x;
}

In Dart 1.0 the type parameter can be left off, since List<dynamic> can masquerade as List<T> for any T.

class A<T> {
  const A() : x = const [];
  final List<T> x;
}

In strong mode, a List<dynamic> cannot be used as a List<T>. Strong mode currently allows a type parameter to be filled in via inference for const objects such as these in order to continue to type as much existing code as possible. DDC implements this via runtime canonicalization. As we move towards unifying strong mode across platforms, we need to decide what to do here going forward. I see two immediate options:

1. Allow generic type parameters (inferred or explicit) to be passed to const objects, canonicalize at runtime (current strong mode/DDC)
2. Forbid generic type parameters entirely (inferred or explicit) from being passed to const objects. This means that const objects allocated inside of generic classes/methods can never use the generic type. This permits compile time canonicalization.

Are there other options?

[leafpetersen]

cc @lrhn @floitschG @eernstg @munificent

[lrhn]

There is the in-between option of allowing type variables in potentially compile-time constant expressions (and also allow other constructor parameters as variables), but not in general.

In this case we have:

const A() : x  = const <T>[];  // implicit or explicit <T>

This is similar to the classical code:

const A(x) : x = const [x];

in that it uses a constructor parameter in a compile-time constant expression. Neither is allowed by the current specification, both could be allowed without breaking anything (I think, but as usual, I might just be wrong).
The point with const-invoked const constructors is that the values of parameters are known at compile-time. That goes for type parameters as well. We currently disallow this (maybe, spec isn't entirely clear) because the const [x] expression must be a compile time constant expression, not just a potentially compile-time constant expression.

It's still not runtime canonicalization - the constructor only works when const invoked, you can't do new A<T>() at all. That's pretty annoying, and why would prefer if there was a way to get a new operator when the const constructor is invoked with new and const when it's used as const:

const A(T x) : auto <T>[x];  // new if "new A", const if "const A".

(Allowing new to be omitted would make this easier).

[munificent]

Are there other options?

Remove const? It adds a ton of complexity to the language, for dubious benefit. From looking at early docs and email, it seems there was some hope that const would be helpful for "immutable values", but people at the time didn't seem to realize it's only useful for the very narrow subset of immutable values that you happen to know at compile time.

I'd rather have features to make all immutable values fast, even ones known only runtime. C# does this with structs, and we can probably do something along the same lines.

[eernstg]

We have to be very explicit about the underlying dynamic semantics. There's a semantics (1) where a type argument can be inferred and then reified, and a semantics (2) where omitted type arguments get supplied implicitly as <dynamic...>. Semantics (2) is relatively unambiguously known as 'spec mode', and semantics (1) could be called 'strong mode semantics' (although 'strong mode' is also used to denote a certain set of static checks, paired up with spec mode execution).

With strong mode semantics we need to take inferred elements of programs just as seriously as explicit ones. This means that Leaf's second example should be considered to be an abbreviation:

class A<T> {
  const A() : x = const [];  // Exactly the same thing as `const <T>[]`.
  final List<T> x;
}

Every invocation of the const constructor above will pass a type argument (possibly inferred), because every instance of A, including const ones, will have an actual type argument corresponding to T. This means that the type parameter T on the right hand side is similar to a value argument, e.g.,

class B {
  const B(int i) : x = i; // Not using `this.x`, to make the similarity obvious.
  final int x;
}

The reason why const <T>[] is disallowed in the spec is not that this is inherently not a compile-time value: it's a perfectly well-defined compile-time value starting from the ultimate const-invoked constructor (and yes, Lasse's remark about the need to allow const constructors which have no valid new invocations apply: the const constructor in A admits only const-invocations). But const <T>[] is still not a 'potentially constant' expression, because of the constraints on a listLiteral.

This means that the actual conflict here is between the (lacking) generality of const and the "natural" expressions available in Dart: "We can write this, so why don't you just do it!!"

So const could be made more general, or it could be dropped altogether. We discussed generalizations several times; I think it would be nice to generalize const to give it a more "natural" set of limitations, which may or may not be considered to be a simplification, depending on your taste. ;-)

We should remember, though, that "compile-time-ness" is essential in one case, namely for metadata annotations. So if we want to have metadata then we need some way to specify compile time values.

It's worth noting that Java has an entire micro-language just for this purpose (https://docs.oracle.com/javase/specs/jls/se8/html/jls-9.html#jls-9.7). I actually think that a relatively manageable mechanism like const in Dart is more well-integrated in the language than the annotations sublanguage in Java.

But it would be really nice if we could exploit the existing notion of 'potentially constant' expressions to obtain good and smooth support for the construction of objects (and object trees) which are explicitly marked as immutable, and maintained as such by the language.

[lrhn]

We should remember, though, that "compile-time-ness" is essential in one case, namely for metadata annotations. So if we want to have metadata then we need some way to specify compile time values.

That's probably going to be guaranteed by anything we chose to do because the annotation expressions are not in scope of any runtime parameter.

[eernstg]

(indeed, but I was thinking about the proposal to drop const entirely)

[munificent]

We should remember, though, that "compile-time-ness" is essential in one case, namely for metadata annotations. So if we want to have metadata then we need some way to specify compile time values.

Yup, this is a good point. My hunch is that we could simplify the set of expressions allowed in metadata annotations without too much loss in usefulness.

It's worth noting that Java has an entire micro-language just for this purpose (https://docs.oracle.com/javase/specs/jls/se8/html/jls-9.html#jls-9.7).

C# has something similar:

24.1.3 Attribute parameter types

The types of positional and named parameters for an attribute class are limited to the attribute parameter types, which are:

• One of the following types: bool, byte, char, double, float, int, long, short, string.
• The type object.
• The type System.Type.
• An enum type, provided it has public accessibility and the types in which it is nested (if any) also have public accessibility.
• Single-dimensional arrays of the above types.

And:

An expression E is an attribute-argument-expression if all of the following statements are true:

• The type of E is an attribute parameter type (§24.1.3).
• At compile-time, the value of E can be resolved to one of the following:
  • A constant value.
  • A typeof-expression (§14.5.11) specifying a non-generic type, a closed constructed type (§25.5.2), or an unbound generic type (§25.5).
  • A one-dimensional array of attribute-argument-expressions.

It's quite limited—basically literals of primitive types and single arrays of such—and yet in my years using them I can't recall it feeling like much of a restriction. That leads me to think const in Dart is overpowered for the metadata use case.

I actually think that a relatively manageable mechanism like const in Dart is more well-integrated in the language than the annotations sublanguage in Java.

Yeah, I agree it's more integrated—it shows up in things like default values, you can declare named constants, etc.—but it adds a lot of complexity to get that, and I don't see that we get that much value from it in return.

It almost lets you do real compile-time programming, which would be useful for many things, but isn't powerful enough for that. It ends up leading you down a path where you think you can make something const, but will likely end up hitting the wall later. (Recall, for example, how RegExp used to have a const constructor, which then later had to be changed to non-const.)

And we pay a high price for that just-slightly-more-powerful-than-Java-and-C#-ness: forwarding constructors, canonicalization, constructor initialization lists, const, const constructors, "potentially const", rules around inheritance and mixins of classes with const constructors, etc.

I think we could come up with something that had a better power-to-weight ratio than that.

[eernstg]

So default values should be evaluated in static class scope and otherwise not be different from any expression that might appear in the body of a static method? That would be somewhat more powerful, and presumably benign. And we wouldn't need constness for that. ;)

[lrhn]

One problem that const values avoid is evaluation order dependencies.
If a default value is evaluated in a static scope and otherwise not different from a static function body (or, say, the initializer expression of a static variable), then the value may refer to variables, and the actual value used may depend on when the expression is evaluated - and whether it's evaluated more than once.

class C {
  static _ctr = 0;
  foo([int x = _ctr++]) => x;
  bar([int x = _ctr++]) => x;
}
main() {
  print([C.foo(), C.bar(), C.foo()]);  // prints 0, 1, (0 or 2) ?
}

I still think that a syntax for side-effect free and evaluation order independent values is useful.
(But then, I like const in Dart).

[munificent]

My cavalier answer is that we should get rid of default values too. :) I do see their usefulness, but they break forwarding in such ugly ways, pollute interfaces in confusing ways, and don't play nice with non-nullability. Maybe they are a necessary evil, but I'm not convinced they carry their weight either.

In my own personal code, I almost never use them. (And, when I do, more often than not, I end up having to remove it later in order to get forwarding to work again.)

I would expect your program to print [0, 1, 2]. Dart is an imperative language with side effects, so I think the intuitive desugaring for a default value is that it gets run right before the body of the function. Python's approach of only running a default value expression once is a well-known pitfall.

Don't get me wrong, I see the value in having side-effect free code, but it does feel kind of bolted on to a language that is otherwise deeply imperative and mutable. If we want to start caring more about being side-effect free, I think we should have a more comprehensive immutability story. Limiting your side-effect free language to only be able to compute on values known at compile time is just too limited to be worth the language complexity, I think.

[lrhn]

My cavalier answer is that we should get rid of default values too.

That would mean that optional parameters cannot be non-nullable. Which does make a kind of sense.
With the ??= operator, it's pretty easy to do the default value in code, and it removes the requirement that the default value is constant.
I also dislike that the default value becomes part of the signature so that it affects subclasses - I'd prefer that it was an implementation detail of the method (only difference is that then you have to document it in prose instead of just in code).

You managed to convince me in the time it took me to write a response :)

[munificent]

With the ??= operator, it's pretty easy to do the default value in code, and it removes the requirement that the default value is constant.

Yup!

I also dislike that the default value becomes part of the signature so that it affects subclasses

Yes, and interfaces. Having to document it is kind of a chore, but I still would prefer that over the fact that right now subclasses have to actually repeat the default value. That violates DRY for me, which I care a lot about.

[eernstg]

About the idea that a default value could be similar to an expression in static scope:

Sure, there would be a potential for side effects and order dependencies, but Dart code in general has that everywhere, and we generally trust programmers to be able to handle it.

I think this is a typical trade-off between emphasizing constraints and security vs. emphasizing flexibility and expressive power. Dart typically sets out offering a lot of flexibility, and then there are ways to add constraints to that. So could we do that?

Yes: We should be able to support things like "this function has no side-effects" (@pure?), especially for local and top-level functions where overriding is a non-issue. We could even allow for a method to declare itself @pure, and then complain in overriding definitions if they can't be shown to satisfy that. We would then restore some core guarantees provided by the current default value semantics, even in the context where we have a lot more flexibility and expressive power.

[eernstg]

About broken forwarding with default values:

How much would it help to have hints for the cases where not all named arguments are passed? There could be a @forward on the forwarding method, telling tools that it is intended to cover them all.

In the general case it's hardly reasonable to require extra ceremony whenever an optional parameter is omitted, but we could of course add mandatory named parameters (whose semantics for dynamic (statically uncheckable) invocations would be like a default parameter whose default value throws .. ah, darn, then we need default parameters with a dynamic semantics, but I won't argue against that ;-).

[bwilkerson]

... mandatory named parameters ...

FYI: See the annotation required in the meta package (pkg/meta in the sdk repo).

[eernstg]

About default values polluting interfaces:

I think that they are part of the interfaces, because it is semantically significant exactly which value will be chosen for the given parameter when nothing is specified at call sites, and it would be a serious violation of encapsulation if the caller had to know the exact type of the receiver in order to be able to specify an explicit argument which would have the same effect.

But this means that it should be a (fatally smelling) hint to give a different default value for the same parameter in an overriding method declaration.

The interface is now bigger, because it includes the default values, but I don't think that there is any complexity there which is particularly confusing: It takes up more brain space (and more source code) because it does more.

You could even claim that

  void foo({int someName: 5}) {
    ..
  }

is so much more informative than

  void foo({int someName}) {
    someName ??= 5;
    ..
  }

that the former should be preferred, because it is relevant to the caller what an omitted parameter means.

[eernstg]

Ah, sorry: Didn't know about required. Thanks!

[bwilkerson]

... it should be a (fatally smelling) hint to give a different default value for the same parameter in an overriding method declaration.

Actually, it's a warning specified by the specification in section 10.1:

It is a static warning if an instance method m1 overrides an instance member m2, the signature of m2 explicitly specifies a default value for a formal parameter p and the signature of m1 implies a different default value for p.

[eernstg]

Ah, missed that one, too! Thanks for the clarification.

[lrhn]

The solution is to infer a <Null>[] in a const context. A <Null>[] can be used as any list type, like <dynamic>[] could in Dart 1, and since the list is empty, there are no elements to contradict the type.

You still can't use type variables in a const expression.