Test plan for "ref assemblies" (7.1)

[gafter]

This is the test plan for "ref assemblies".

See also

• Language Feature Status
• The refout feature branch
• PR comparing the refout feature branch to master
• [Work items] Refining ref assemblies
• Specification
• Produce ref assemblies from command-line and msbuild (initial implementation)
• Make the command-line compilers able to generate reference assemblies (old specification)

---

Golden Path Tests

• Unless otherwise specified, the following tests should exist for both VB and C#
• Should produce both an output assembly and a ref assembly using both /out and /refout
• Should produce only a ref assembly using /refonly
• The ref assembly produced should be deterministic when * not used in AssemblyVersionAttribute
• When /refonly is used, we should not be sensitive to semantic error in [see RefAssembly_InvariantToSomeChanges]
  • Method bodies (including operators)
  • Properties
  • Field initializers
  • Constructor bodies including initializers
  • Property Accessor bodies
  • Event Accessor bodies
  • Similarly, all these cases when there is an InternalVisibleToAttribute
• When /refout and not /refonly, we should not produce a ref assembly if there are any errors [see RefOutWithError]
• Tuple names in APIs are visible through the ref assembly. (See RefAssemblyClient_EmitTupleNames)
• Dynamic in APIs is visible through the ref assembly (C# only). (See RefAssemblyClient_EmitDynamic)
• out parameters in APIs are visible through the ref assembly (C# only). (See RefAssemblyClient_EmitOut)
• Modules are visible through the ref assembly (VB only)
• Variance is visible through the ref assembly (See RefAssemblyClient_EmitVariance)
• Constant and enum values visible through the ref assembly (See RefAssemblyClient_EmitConst and RefAssemblyClient_EmitEnum)
• Parameter default values visible through the ref assembly (See RefAssemblyClient_EmitOptionalArguments. But no way to verify the default value of parameter)
• Parameter names used in overrides visible through the ref assembly. (See RefAssemblyClient_EmitArgumentNames)
• Params parameters visible through the ref assembly. (See RefAssemblyClient_EmitParams)
• Extension methods accessed through a ref assembly act as extension methods. (See RefAssemblyClient_EmitExtension)
• When /refonly and /doc are used, XML docs produced for included APIs (See RefOnly And RefOut in commandline tests)
• When /refonly and /doc are used, XML docs not produced for excluded APIs (out of scope for 7.1)
• When /refout used and resource are specified, they are not in the ref assembly
• When /refonly used and resource are specified, they are included in the ref assembly
• When method bodies, comments, and spacing are changed, the ref assembly is invariant.
• The order of members appearing in a ref assembly is the same as in the main assembly (See RefAssembly_VerifyTypesAndMembers)
• Changing the order of methods in a class causes the order to be changed in the ref assembly
• The generated assembly excludes compiler-generated "implementation details" type (See RefAssembly_VerifyTypesAndMembers, which has anonymous type)

Inclusion and Exclusion

• Most private members of classes should be excluded from the ref assembly
  • Fields excluded when private (and included otherwise)
• Most private members of structs should be excluded from the ref assembly
  • Fields should be included (until we refine this policy)
• For both classes and structs
  • Nested types should be included no matter the access (See RefAssemblyClient_EmitNestedTypes)
  • Explicit interface implementations should be included (not sure how to verify)
  • Methods, properties, events, operators, indexers, constructors excluded only when private (I don't think I have a test for operators or indexers though)
  • Private accessors of properties should be excluded (See RefAssembly_VerifyTypesAndMembers)
  • Attributes in all locations included (until we refine this policy)
• In the future, we can change private to internal and exclude more if there are no InternalsVisibleTo.
• Generic type parameter constraints on included API elements are included.

Miscellaneous

• It is an error to use /addmodule with either /refout or /refonly.
• Produce ReferenceAssemblyAttribute in the ref assembly with either /refout or /refonly.
• /refonly and /debug together are an error (There is a test for this, but the test asserts that it is NOT an error. Perhaps change spec? See /refout and /debug together should be an error #19708)
• Ensure error scenarios around structs are preserved when used from a ref assembly
  • A struct with any non-empty field is not empty for DA purposes (See RefAssemblyClient_StructWithPrivateIntField)
  • A generic struct with a generic private field may cause circularity errors at a use site, e.g. struct S1<T> { private T t } in the ref assembly used this way struct S2 { S1<S2> x; } (See RefAssemblyClient_StructWithPrivateGenericField)
• Are there winmd-specific scenarios?
• Can F# compile against ref assemblies?
• Can C++ compile against ref assemblies?
• There should be an error if the main assembly and ref assembly have different names.
• Test F5 scenario in IDE

Build Scenarios

• The implementation includes MSBuild support not covered by this test plan. However, here are some basics:
  • Test build and rebuild scenarios.
  • An up-to-date ref assembly should cause the build of a dependent assembly to be skipped
  • An up-to-date ref assembly should not prevent deployment of a dependent project when the associated DLL has changed.
• Does C# scripting run against the proper assembly when ref assemblies are used in the build?

[jasonmalinowski]

Some things to think about for the test plan:

• Testing build, and rebuild scenarios, and making sure the binaries are correctly deployed in all projects. The fast-up-to-date check could interfere with things.
• Testing references between C# and VB project references, and ensuring the IDE is working correctly across those. (Do refactorings work, for example?)
• Testing references to and from C++ projects, both regular DLL kinds and also .winmd producing kinds.
• Testing across the .NET Core / legacy project system boundary.
• Object browser?
• XAML designer.
• WinForms designer with controls being consumed across-projects.
• Live Unit Testing. (broken)

[tmat]

Also references that use AssemblyVersion(*)

[rainersigwald]

@jcouv's initial testing has revealed a couple of interesting project set-up scenarios:

• Legacy csproj that references System.Runtime facades
• New-SDK csproj that references System.Runtime facades through NuGet packages

(I tested the latter during development but broke the former.)

[jcouv]

Another scenario from Jason:

I’m not sure exactly when the legacy project system reevaluates that item group, so you’ll probably want to test that it does it right after adding a new project reference and also does it if you rename the output name of a project you’re depending on or change it’s output path.

[gafter]

Went over the test plan today with @jcouv and checked off all we could.

[jcouv]

Issues so far:

• P2P reference behave as file reference (filed issue)
  • Go To Definition goes to metadata view, instead of source
  • If I add an API to a library, I need to re-build the library and the client before the API is visible to the client
• Live Unit Testing gives "Build completed (failed)." (filed issue)
• When renaming a library project, then building the test project a couple of time, I would expect things would be up-to-date (but they are not). This does not repro if the test project is just a client project. So I think this is a pre-existing problem that Paul mentioned.

Other things I tested:

• Big red button (setting CompileUsingReferenceAssemblies to false) in IDE
• FUTD check across old/new project systems

[gafter]

I'm closing this. @jcouv, do you want issues for additional proposed tests?