Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot use experimental language features in @experimental code #13392

Closed
odersky opened this issue Aug 26, 2021 · 48 comments · Fixed by #13394
Closed

Cannot use experimental language features in @experimental code #13392

odersky opened this issue Aug 26, 2021 · 48 comments · Fixed by #13394
Milestone

Comments

@odersky
Copy link
Contributor

odersky commented Aug 26, 2021

Compiler version

3.1.0

Minimized example

import language.experimental.erasedDefinitions
import annotation.experimental

@experimental
erased class CanThrow[-E <: Exception]

Output

Error:  2 |import language.experimental.erasedDefinitions
Error:    |       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error:    |Experimental features may only be used with a nightly or snapshot version of the compiler

Expectation

Should compile.

Otherwise, we cannot use experimental language features in @experimental code.

I think what happens is that we check the compiler version when we analyze the import statement. We should wait with it until we need to check whether the language import is enabled. If all these tests are from experimental scopes, this should be permitted.

@nicolasstucki
Copy link
Contributor

It is expected to have a compilation error on import language.experimental.erasedDefinitions if we tell to the compiler to fail on the use of experimental features with -Yno-experimental.

@odersky
Copy link
Contributor Author

odersky commented Aug 26, 2021

Yes, but I think in terms of language features that's wrong. We should check on use, not import.

@odersky
Copy link
Contributor Author

odersky commented Aug 26, 2021

Can we wait with enforcing @experimental until 3.2? I don't think it's baked yet. The problem shown by CanThrow is not very special and is in fact quite typical. @experimental has already the reputation of being quixotic. If we cannot make it 100% usable we should not impose it now.

@nicolasstucki
Copy link
Contributor

The current version is conservative, if we relax it we will need to add custom logic for each experimental feature on a case-by-case basis.

  • namedTypeArguments should fail on import
  • genericNumberLiterals could be checked at use site (not sure how)
  • erasedDefinitions could be checked at the definition site. An @experimental erased def ... would be accepted.
  • fewerBraces should fail on import

@nicolasstucki
Copy link
Contributor

Can we wait with enforcing @experimental until 3.2?

This is too dangerous in general. For CanThrow we can just not enforce it for the standard library. We would need to be extra careful to not leak experimental features that are not in an @expreimental definition.

@nicolasstucki
Copy link
Contributor

Oh no, but then we would not be able to bootstrap on a stable compiler. The simples change seems to be to special case erasedDefinitions now.

@odersky
Copy link
Contributor Author

odersky commented Aug 26, 2021

Yes, I think at least so far erasedDefinitions is special since it affects allowed modifiers. For all other features we could have the experimental import inside the @experimental scope.

@odersky
Copy link
Contributor Author

odersky commented Aug 26, 2021

But in fact that will probably not work either. Since the parser does not know about experimental contexts.

@odersky
Copy link
Contributor Author

odersky commented Aug 26, 2021

This is too dangerous in general.

Why is it dangerous? That's what I am not getting.

@nicolasstucki
Copy link
Contributor

The idea would be to not fail on the import of erasedDefinitions in any situation. Then, after typing, check that all erased definitions are in an experimental scope.

@odersky
Copy link
Contributor Author

odersky commented Aug 26, 2021

I just had another idea. We could delay the experimental warning on experimental imports to typer. Then, allow an experimental import if one of the following is true:

  • the import is in an experimental scope
  • the import is in a scope where all definitions are marked @experimental and there are no expressions

@nicolasstucki
Copy link
Contributor

That might be a bit more complex. My version only requires the delay of the erasedDefinitions import. This might be the first step before moving all the checks after parser.

@odersky
Copy link
Contributor Author

odersky commented Aug 26, 2021

Let me try it anyway. I prefer not to need special treatments and exceptions. It's very awkward documenting them, to start with.

@nicolasstucki
Copy link
Contributor

@odersky in #13394 I have a draft of the checks needed for the use sites of erased. This is enough to support the CanThrow use case. The rest of the import checks can change could be implemented on top of this.

nicolasstucki added a commit to dotty-staging/dotty that referenced this issue Aug 26, 2021
nicolasstucki added a commit to dotty-staging/dotty that referenced this issue Aug 26, 2021
nicolasstucki added a commit to dotty-staging/dotty that referenced this issue Aug 26, 2021
@nicolasstucki nicolasstucki added this to the 3.1.0 milestone Aug 27, 2021
@nicolasstucki
Copy link
Contributor

Overview of the current solutions

There are two PRs containing different parts of the solution (#13394 and #13396). The solutions in those PRs are half-complementary half-overlapping.

There are 3 new rules that can be extracted from those PRs

Rule (1)

Experimental imports can be used in experimental scopes (in #13396 but could be added to #13394)

This rule is not controversial. It allows experimental imports in any experimental scope such as in a nightly build of the compiler or within an experimental definition.

Example

Assuming that we are on a stable release of the compiler or added the -Yno-experimental compiler flag.

import annotation.experimental

@experimental
object Foo:
  import language.experimental.erasedDefinitions // ok, Foo is @experimental
  def foo = ...
  erased def bar = ...

object Bar:
  import language.experimental.erasedDefinitions // error
  def foo = ...
  erased def bar = ...

This rule alone does not allow the use of erased on top-level definition.

This rule can be complemented by rules (2) or (3).

Rule (2)

If an experimental import is in a scope where all definitions are marked @experimental and there are no expressions then that import is allowed

This rule aims to allow experimental erased top-level definitions in a file.

Example

Assuming that we are on a stable release of the compiler or added the -Yno-experimental compiler flag.

import language.experimental.erasedDefinitions // ok, because we only have experimental definitions
import annotation.experimental

@experimental erased class Foo
import language.experimental.erasedDefinitions // error: Experimental features may only be used with a nightly or snapshot version of the compiler. Note: the scope enclosing the import is not considered experimental because it contains the non-experimental class Bar.
import annotation.experimental

@experimental erased class Foo

class Bar
import annotation.experimental

class Foo:
  import language.experimental.erasedDefinitions // ok, because all statements are experimental
  @experimental erased def foo = 1
import annotation.experimental

class Foo:
  import language.experimental.erasedDefinitions // error: Experimental features may only be used with a nightly or snapshot version of the compiler. Note: the scope enclosing the import is not considered experimental because it contains the non-experimental method bar.
  @experimental erased def foo = 1
  def bar = 1

This allows any experimental features to be used a top-level in a source file if all definitions are experimental. It also allows the use of experimental in several other contexts (hard to describe which).

Currently, it suffers from UX problems that may confuse the user or make them think there is a bug where there is non. There might also be some bugs in the implementation that make this judgment harder. Here are a few examples that will cause confusion or any users:

Problematic cases

Assuming that we are on a stable release of the compiler or added the -Yno-experimental compiler flag.

Adding a new object will cause a compilation error

- import language.experimental.erasedDefinitions
+ import language.experimental.erasedDefinitions // error: Experimental features may only be used with a nightly or snapshot version of the compiler. Note: the scope enclosing the import is not considered experimental because it contains the non-experimental class Bar.
import annotation.experimental

@experimental erased class Foo

+ object Bar

but the following does work

import language.experimental.erasedDefinitions
import annotation.experimental

@experimental erased class Foo

+ object Foo

Adding some stable member breaks code

import annotation.experimental
class Foo:
-  import language.experimental.erasedDefinitions
+  import language.experimental.erasedDefinitions // error: Experimental features may only be used with a nightly or snapshot version of the compiler. Note: the scope enclosing the import is not considered experimental because it contains the non-experimental method bar.
  @experimental erased def foo = 1
+ def bar = 1

Strangely the following also breaks even though there is already had a non experimental constructor.

import annotation.experimental
class Foo:
-  import language.experimental.erasedDefinitions
+  import language.experimental.erasedDefinitions // error: Experimental features may only be used with a nightly or snapshot version of the compiler. Note: the scope enclosing the import is not considered experimental because it contains the non-experimental constructor Foo.
  @experimental erased def foo = 1
+ def this(i: Int) = this()

And adding code inside the constructor that was already allowed breaks the code

import annotation.experimental
class Foo:
-  import language.experimental.erasedDefinitions
+  import language.experimental.erasedDefinitions // error: Experimental features may only be used with a nightly or snapshot version of the compiler. Note: the scope enclosing the import is not considered experimental because it contains the non-experimental expression println(1).
  @experimental erased def foo = 1
+ println(1)

It behaves strangely in term scopes such as iin a method bodies

import annotation.experimental
def foo: Unit = {
-  import language.experimental.erasedDefinitions
+  import language.experimental.erasedDefinitions // error: Experimental features may only be used with a nightly or snapshot version of the compiler. Note: the scope enclosing the import is not considered experimental because it contains the non-experimental expression println().
  @experimental erased def foo = 1
  println()
+ println()
}

Why does it fail or why didn't it?

Interaction with @experimental definition and @experimental erased experimental differs for no good reason.

import annotation.experimental

class Foo:
 // import language.experimental.erasedDefinitions
 def foo = 1
+ @experimental def bar = 1 // ok

class Bar:
+ import language.experimental.erasedDefinitions // error: Experimental features may only be used with a nightly or snapshot version of the compiler. Note: the scope enclosing the import is not considered experimental because it contains the non-experimental method foo.
 def foo = 1
+ @experimental erased def bar = 1 // not ok

It is impossible to define erased sealed classes.

import language.experimental.erasedDefinitions // error: ....
import annotation.experimental
sealed trait Foo
@experimental erased class Foo extends Foo

I've hit more strange interactions and limitations, but I do not have the time to list them all. Therefore I stop with this small subset of issues that users will encounter and report.

It is unclear if it is really useful to be able to import a language feature inside a definition but not make that definition experimental (i.e. use rule 1). If we would not support this use case we would only need a rule that allow us to import experimental definition at the top level of a file. This is equivalent to forcing all top level definitions of a file to be experimental. We could instead create a simpler targeted mechanism to achieve this purpose.

Rule (3)

erased definition can be used in experimental scopes

This rule special cases the imports of erasedDefinition by allowing it in any scope (in the same way we already do with experimental.macro). Then it checks that all erased definitions are in an experimental scope.

Example

Assuming that we are on a stable release of the compiler or added the -Yno-experimental compiler flag.

import language.experimental.erasedDefinitions // ok
import annotation.experimental

object Foo:
  def foo = ... // ok
  @experimental erased def bar = ... // ok
  erased def baz = ... // error: `erased` definitions can only be used in an experimental scope. Consider making `baz` experimental.

@experimental erased class Bar // ok
@experimental erased def baz = ... // ok

Like rule (3), it also supports top-level erased definitions. Unlike rule (3), this allows stable definitions to be defined along with @experimental erased definitions.

When we stabilize the erased language feature we will be able to drop this rule.

Conclusion

To fix #13392 we need (2) or (3). (1) can be added to either to enhance the user experience. All rules complement each other and overlap in some use-cases.

If we use rule (3) we will be able to remove it once erased is stable.
It will always be safe to change this rule along the way as it only affects experimental definitions. We can add (1) in 3.1.x and test it on its own a bit more.
This approach makes erased definitions simpler to use in all experimental scopes, making it more useful for users.

If we use rule (2) we will have to support it from here on even if it is broken as changes to it might affect non-experimental parts of sources.
If we need to change this rule we might break a valid non-experimental user code.
We must add (1) at the same time.
This feature was designed to solve the problem with the CanThrow but does not seem to take into account other users.

I would favor (3) for 3.1.0 and then add (2) for 3.1.x. This solution cannot lead to broken non-experimental features and allow users to define their own experimental erased definitions.

@odersky
Copy link
Contributor Author

odersky commented Aug 27, 2021

(3) means that we cannot use any of the other experimental language imports in a stable compiler. For instance, I cannot write

import language.experimental.fewerBraces

@experimental
class Test(xs: List[Int]: 
  xs.map x =>
    x + 1

and try that out in a standard Scala 3 distribution. I don't find that acceptable.

This leaves (2), or else we turn experimental checking off until 3.2.

@nicolasstucki
Copy link
Contributor

We need (1) to allow experimental language features to be used in stable compiler. Extra use case can be added later.

@experimental
class Test(xs: List[Int]: 
  import language.experimental.fewerBraces
  xs.map x =>
    x + 1

This works for any of the experimental language features we have except for erasedDefinitions.

In it's current state (2) looks like a special case for us to be able to compile some code while it make it hard for users to use the feature. The current version has a rule that is already complex and also leaves some underspecified cases (or has many bugs). For the sake of the users we should not have this one and find a good way to achieve the same.

Rule happens to work for the experimental language features we have now. There is the possibility that at some point we will want to add a new experimental feature that is incompatible with this rule. This would be a blocker for that feature.

On the other hand (3) was designed to be easily usable by anyone that wants to try the erasedDefinitions feature. It is not an exception for us but it also allows us to compile the use case we care about. It also us keeps the experimental scopes rules simple.

@odersky
Copy link
Contributor Author

odersky commented Aug 30, 2021

In it's current state (2) looks like a special case for us to be able to compile some code while it make it hard for users to use the feature. The current version has a rule that is already complex and also leaves some underspecified cases (or has many bugs). For the sake of the users we should not have this one and find a good way to achieve the same.

I disagree and I think you have to be more specific to make that allegation. Also, any underspecification simply does not matter! We are not talking a type soundness theorem here but a pragmatic feature that should prevent the common misuses and allow the comon use cases. That is all.

The solution to put fewerBraces inside each experimental class is workable but weird since it violates the common assumption that language imports go at the top. Someone who would try it out the first would always write it at the top
and then be surprised why it does not work, and most likely be disgusted by the proposed fix.

I don't think we will make progress discussing this further. Let's get both fixes in and call it a day.

@nicolasstucki
Copy link
Contributor

Rule (4): an alternative to rule (3) that would work for users as well

The idea is to strip the strange corner cases out of (3) and solve specifically the problem it tries to solve. Also, remove inference of experimental to avoid bad UX.

  • (4) Contents of experimental source file is in experimental scope.

This implies that all top classes must be marked as experimental as in normal experimental rules. We might also require top-level methods and values to be experimental.

It is intended to be used with rule (1) to allow all the use cases in the tests of #13396

The question then is how to state that a source file is experimental. Here are some options:

  • Special import: import scala.language.experimental.source
  • Special file name: Foo.experimental.scala
  • Annotate the package clause in some way package @experimental foo { } (not sure if possible)

@odersky
Copy link
Contributor Author

odersky commented Aug 30, 2021

I believe any of these conventions is too complicated and not necessary. it's still in the spirit that we should do our utmost to prevent usability rather than allow common use cases without hassles.

@odersky
Copy link
Contributor Author

odersky commented Aug 30, 2021

I agree it would be good to have tests but I personally have spent way too much time on this already. So somebody else should add tests. And I insist that #13396 will go into 3.1, or experimental checking as a whole goes out.

@nicolasstucki
Copy link
Contributor

If we get #13396 we will have a flood of issues from users. I already have 5+ issues that I could immediately report. This would not be good from the user's perspective.

To disable experimental checks we would need to make @experimental experimental again. It would take a couple of days in the best of cases to implement as it is non-trivial at this point. This seems to also not allow us to re-bootstrap on a release compiler, this would block us again.

On the other hand, we have #13394 which does not touch non-experimental features, has proper tests, is useful for users, and is ready.

nicolasstucki added a commit to dotty-staging/dotty that referenced this issue Aug 30, 2021
@nicolasstucki
Copy link
Contributor

We agreed that more precise version of (2) that only allows top-level definitions is the solution.

  • (5) experimental language imports are allowed if all non-synthetic top-level statements are @experimental definitions

olsdavis pushed a commit to olsdavis/dotty that referenced this issue Apr 4, 2022
@pshirshov
Copy link
Contributor

I don't know why this had been done. Seems like the enforcement rules make it's practically impossible to write libraries relying on "experimental" features, because all the client code needs to be annotated with this.

@SethTisue
Copy link
Member

Seems like the enforcement rules make it's practically impossible to write libraries relying on "experimental" features

sounds intentional to me... otherwise the concept of experimental has no teeth

@mdedetrich
Copy link

mdedetrich commented Aug 23, 2023

I guess I am a bit late to the "experimental" train, but am I correct in my understanding is that its not possible to avoid to the usage of the @experimental annotation even with compiler flags?

If so its somewhat annoying, I am actually currently developing a macro annotation that is designed portable across Scala 2.12, Scala 2.13 and Scala 3.3 (i.e. the whole point of the macro annotation is that you have a single source that is cross compiled against those Scala versions) and since macro annotations are considered experimental in Scala 3 this basically means that users of that macro annotation would have to mark @experimental everywhere just because of Scala 3

@pshirshov
Copy link
Contributor

In most cases it is possible to work around with various tricks, in worst case by relying on java reflection.

@mdedetrich
Copy link

mdedetrich commented Aug 23, 2023

In most cases it is possible to work around with various tricks, in worst case by relying on java reflection.

Is there a reference (i.e. code sample) for these tricks?

Also in case its not clear, I am not so worried about using @experimental in the definition of the macro, i.e.

import scala.annotation.MacroAnnotation
import scala.quoted.Quotes
import scala.annotation.experimental

@experimental class getInline extends MacroAnnotation

Tne problem is that it appears to pollute/propogate the call site of the macro, i.e. to use @getInline you have to do

import scala.annotation.experimental

object Test {
  @experimental @getInline final def myVal = 5
}

Which is what I meant beforehand about the annoying part.

@odersky
Copy link
Contributor Author

odersky commented Aug 24, 2023

The whole point of @experimental is that it is transitive. You cannot have a definition that's @experimental and have the users not be @experimental. I also find that mechanism a bit heavy-handed since we don't want to sprinkle @experimental on all user code.

But there is an alternative: we can also access experimental definition in a snapshot compiler release. This would mean you can cross compile using @experimental Scala 3 features, but the Scala 3 compiler would have to be a snapshot compiler.

Question to @Kordyjan , @nicolasstucki , @smarter: What is currently the best way to specify snapshot release? Are we happy with that, or would we like to have something simpler or more robust?

@mdedetrich
Copy link

mdedetrich commented Aug 24, 2023

The whole point of @experimental is that it is transitive. You cannot have a definition that's @experimental and have the users not be @experimental. I also find that mechanism a bit heavy-handed since we don't want to sprinkle @experimental on all user code.

I would question the transitivity feature specifically for macro annotations here because unless I am missing something, unlike other experimental features (i.e. direct style) whether the macro annotation is experimental or not is not relevant for users (i.e. callers) of the macro. Macros just transform code so even if the implementation of how macro annotations is designed is marked as experimental thats a problem for the macro author, i.e. if the design of macro annotations changes then the author has to deal with it but the user doesn't have to do anything, i.e. for a user they will always just do @myMacroAnnotation. This is contrast to direct style, where if it would hypothetically change then the user is indeed severely impacted so I can see the merit of transitive checking of @experimental of other features. For this reason I would argue that rather than forcing @experimental on the MacroAnnotation annotation itself I would rather put it on the transform method.

I mean I guess there are ways around this, i.e. putting @experimental at the top level (such as a top level object) does appear to work but that also leaves other annoyances, i.e. it gives the impression that the entire top level object is experimental when in fact its just methods with the macro annotation but its better than nothing.

But there is an alternative: we can also access experimental definition in a snapshot compiler release. This would mean you can cross compile using @experimental Scala 3 features, but the Scala 3 compiler would have to be a snapshot compiler.

At least for the project I am talking about (Pekko), using a snapshot compiler would very likely not cut it although I can see how this is a good general improvement for other cases.

Note that there is a scala contributors thread with more context on the problem I am dealing with https://contributors.scala-lang.org/t/make-scala-2-inline-annotation-work-as-inline-keyword-for-scala-3-with-source-3-0-migration/6286

@mdedetrich
Copy link

I mean I guess there are ways around this, i.e. putting @experimental at the top level (such as a top level object) does appear to work but that also leaves other annoyances, i.e. it gives the impression that the entire top level object is experimental when in fact its just methods with the macro annotation but its better than nothing.

I quickly tried this suggestion and while it does work for Scala 3, it doesn't work for Scala 2 because the scala.annotation.experimental doesn't exist although that can be worked around with a stub annotation.

@mdedetrich
Copy link

@odersky Should I make a discussion thread on Scala contributors on the more general point of whether @experimental makes sense for macro annotations specifically?

@nicolasstucki
Copy link
Contributor

What is currently the best way to specify snapshot release?

Using a nightly release is probably the simplest.

@odersky
Copy link
Contributor Author

odersky commented Aug 24, 2023

Using a nightly release is probably the simplest.

It would be good to see concrete incantations to achieve that, with consideration how it would work in the compiler, REPL, and the IDE.

@odersky
Copy link
Contributor Author

odersky commented Aug 24, 2023

@odersky Should I make a discussion thread on Scala contributors on the more general point of whether @experimental makes sense for macro annotations specifically?

I think for macro annotations @experimental is currently unavoidable until they are fully stabilized. The question how to deal with @experimental in general could be interesting in a contributors thread.

EDIT: One thing we probably need is to backport @experimental to the Scala 3 library. It's really high time we get that moving again, it was frozen for too long and this state is holding everyone back.

@mdedetrich
Copy link

I think for macro annotations @experimental is currently unavoidable until they are fully stabilized.

True, but to clarify I am not suggesting making macro annotations fully stable. I am making a distinction between users of the macro and the implementation/design of the macro and my argument is that the former is stable (unless you are thinking of hypothetically removing macro annotations completely in the future at some point) where as the latter is unstable.

Put differently, it really doesn't matter how macro annotations are designed/implemented, for a user the code will always be @myMacroAnnotation, which is in stark contrast to other experimental features.

The question how to deal with @experimental in general could be interesting in a contributors thread.

Okay ill make a thread, better to talk about it there rather than derailing this issue thread.

@nicolasstucki
Copy link
Contributor

For this reason I would argue that rather than forcing @experimental on the MacroAnnotation annotation itself I would rather put it on the transform method.

At this point the MacroAnnotation trait is not part of the "stable" Scala library, as such it must be marked with @experimental. Note that this trait could disappear at any point, for example if we decide to change its name or move it to another package.

@odersky
Copy link
Contributor Author

odersky commented Aug 24, 2023

True, but to clarify I am not suggesting making macro annotations fully stable. I am making a distinction between users of the macro and the implementation/design of the macro and my argument is that the former is stable (unless you are thinking of hypothetically removing macro annotations completely in the future at some point) where as the latter is unstable.

Until they are stabilized there's a theoretical chance that they might be removed or changed in a way that some functionality is no longer expressible. I don't expect it, but there's no formal guarantee it won't happen. So yes, we need users to be aware of that.

@nicolasstucki
Copy link
Contributor

Put differently, it really doesn't matter how macro annotations are designed/implemented, for a user the code will always be @myMacroAnnotation, which is in stark contrast to other experimental features.

The fact that there is a reference to @myMacroAnnotation in the code implies that there is a dependency/call into an API that is experimental. There the user of that annotation cannot rely on it working in a later version.

@mdedetrich
Copy link

At this point the MacroAnnotation trait is not part of the "stable" Scala library, as such it must be marked with @experimental. Note that this trait could disappear at any point, for example if we decide to change its name or move it to another package.

This to me is more of a technical implementation detail because MacroAnnotation is relevant for authors of Macro, not users.

Until they are stabilized there's a theoretical chance that they might be removed or changed in a way that some functionality is no longer expressible. I don't expect it, but there's no formal guarantee it won't happen. So yes, we need users to be aware of that.

Understood

The fact that there is a reference to @myMacroAnnotation in the code implies that there is a dependency/call into an API that is experimental. There the user of that annotation cannot rely on it working in a later version.

Aside from the hypothetical macro annotations can be removed at some future point, would this case happen if users of the macro stick with a specific Scala 3 version? If not, then aside from the severe changes @odersky alluded to I don't see how it would effect users, i.e. if they upgrade their Scala version to 3.4 then that would bring in a new artifact which has a new implementation against a stable macro annotation API (not sure if this is the case, genuine question).

@nicolasstucki
Copy link
Contributor

Aside from the hypothetical macro annotations can be removed at some future point, would this case happen if users of the macro stick with a specific Scala 3 version?

If they use the same patch version (3.x.y) it will be OK, but not necessarily if they use the same minor release (3.x). @expeimental definitions can be added/removed/modified in any patch version.

@xerial
Copy link
Contributor

xerial commented Sep 5, 2023

mechanism a bit heavy-handed since we don't want to sprinkle @experimental on all user code.
#13392 (comment)

In Scala 3.3.1, this issue becomes more prominent. Any usage of the @experimental macro requires users to add the @experimental annotation to their code. Currently, there is no way to split the responsibility between library creators and users. Ideally, if the library creators use @experimental methods, they should be notified and take responsibility for fixing the code for upcoming Scala 3.x releases. In the current design, however, both library creators and users are required to annotate all experimental code usages, including transitive ones.

Especially, @pshirshov (izumi-reflect) and I (airframe-surface) have created libraries based on Scala 3 macros. I believe we share the same concerns and troubles in dealing with this experimental mechanism. It is unfortunate that, even though some features are experimental, they are good enough to be used in production. Changing or dropping the API design in the future is acceptable, but the only currently available workaround is to depend on the snapshot compiler version, which will create an additional burden for library maintainers.

@SethTisue
Copy link
Member

SethTisue commented Sep 5, 2023

If there are specific experimental features or APIs that the community considers high priority to be stabilized (not be experimental anymore), I think we should have specific tickets about those features. I doubt that any general expression of dissatisfaction about experimental stuff in general is going to have any effect — especially not on this ticket, as it's already been requested to move general discussion to https://contributors.scala-lang.org.

Ideally, if the library creators use @experimental methods, they should be notified and take responsibility for fixing the code for upcoming Scala 3.x releases

I'm curious: could or should the open community build have caught this...? That might be a subject for its own ticket or discussion thread, also.

@mdedetrich
Copy link

mdedetrich commented Sep 5, 2023

In Scala 3.3.1, this issue becomes more prominent. Any usage of the @experimental macro requires users to add the @experimental annotation to their code.

Is this new to Scala 3.3.1? I was experiencing this already in 3.3.0, at least with the prototype macro I am writing in https://github.com/mdedetrich/get-inline the users calling that macro annotation also had to add @experimental (as can be seen at https://github.com/mdedetrich/get-inline/blob/main/src/test/scala-3/org/mdedetrich/Test.scala#L6)

especially not on this ticket, as it's already been requested to move general discussion to https://contributors.scala-lang.org/.

I should get around to doing this, kinda fell of the table.

@xerial
Copy link
Contributor

xerial commented Sep 5, 2023

@mdedetrich Yes. Since 3.3.1-RC1. Another user also reported the behavior change:

I noticed in scala 3.3.1-RC1 that if I used an experimental feature I had to add the @experimental annotation to both the macro method (that returns an Expr) and the inline def that returns the true type I care about. in scala 3.3.0 it was only necessary on the macro method.
https://discord.com/channels/632150470000902164/875868146949554207/1121597037213925396

@xerial
Copy link
Contributor

xerial commented Sep 5, 2023

@SethTisue Yes. ClassDef.apply, Symbol.newClass are essential to produce some type of code, which was impossible to generate in Scala 3.1 #14124. Those are marked as experimental for some reason, probably testing the usability or feasibility:
https://github.com/lampepfl/dotty/blob/6dc3737e5f83416801e61131b0444bff27af581a/tests/run-tasty-inspector/stdlibExperimentalDefinitions.scala#L78-L80

Libraries (izumi, zio, airframe) are already using these reflect methods, so the community build will be able to track the usage.

Yeah. I should have opened another community thread instead of messing up this ticket.

@mdedetrich
Copy link

mdedetrich commented Sep 5, 2023

As requested, created a thread at https://contributors.scala-lang.org/t/behaviour-of-experimental-in-scala-3/6309 . @xerial feel free to post/add there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
6 participants