Make System.Formats.Asn1 library public #36729

bartonjs · 2020-05-20T00:01:14Z

GitHub's compare UI suggests that as one giant squash the existing and new files are too different to count as a move+edit, so this PR is marked as No Squash so the move gets preserved into the file histories.

The first commit is simply a move for the existing AsnReader partials (renamed to AsnDecoder partials), the AsnWriter partials, and the related types (Asn1Tag, UniversalTagNumber, etc), and the tests into their appropriate destinations under System.Formats.Asn1.
The second commit is the infrastructure work to make a minimal package build (the test.csproj has most of the file-includes commented out because the ref.cs is empty at this stage)
The third commit makes all of the changes from the API review, enables tests, and adds some more tests to get code coverage back up to all blocks that lack a Debug.Fail.
- Note that at this point the System.Formats.Asn1 package builds fully... but since all of the original files were deleted in the first commit the repository-as-a-whole does not.
- Everything was done in such a way to minimize edits to anything that actually did the work of reading or writing bytes.
The fourth commit changes the existing libraries using the shared source version to use the public package.
- asn.xslt changed to account for new method names, lack of IDisposable on the writer, and exception type changes (CryptographicException => AsnContentException or ArgumentException).
- An internal, shared-source, AsnValueReader was created to make the transition easier. We can wean off of it over time, or decide we're fine with that as an internal state manager. It only defines methods that were needed.
- Some code was changed to use the fact that AsnWriter.Push* is IDisposable, but it wasn't consistent. (Largely, it was done in inverse proportion to how much code nearby needed to change... more code => less likely to change => more likely to finish the overall change)
- This commit pulls System.Formats.Asn1 into the shared framework. We can alter this by moving (back) to shared source, ilmerge, or other build-time trickery to remove the package dependency; but it can be done as a low-noise followup.

AsnReader partials are named AsnDecoder to better match the final name for the core logic they contain.

ghost · 2020-05-20T00:01:16Z

Tagging subscribers to this area: @bartonjs, @vcsjones, @krwq
Notify danmosemsft if you want to be subscribed.

Dotnet-GitSync-Bot · 2020-05-20T00:01:31Z

Note regarding the new-api-needs-documentation label:

This serves as a reminder for when your PR is modifying a ref *.cs file and adding/modifying public APIs, to please make sure the API implementation in the src *.cs file is documented with triple slash comments, so the PR reviewers can sign off that change.

src/libraries/Common/src/System/Security/Cryptography/Asn1Reader/AsnValueReader.cs

src/libraries/System.Formats.Asn1/Directory.Build.props

src/libraries/System.Formats.Asn1/pkg/System.Formats.Asn1.pkgproj

src/libraries/System.Formats.Asn1/ref/System.Formats.Asn1.csproj

src/libraries/System.Formats.Asn1/src/System.Formats.Asn1.csproj

safern · 2020-05-20T00:39:20Z

src/libraries/System.Formats.Asn1/ref/System.Formats.Asn1.csproj

+<Project Sdk="Microsoft.NET.Sdk">
+  <PropertyGroup>
+    <IsPartialFacadeAssembly Condition="'$(TargetsNetFx)' == 'true'">true</IsPartialFacadeAssembly>
+    <TargetFrameworks>netstandard2.0</TargetFrameworks>


if you want the facades to come into play when this is used in netframework, we should cross target this for netfx.

cc: @joperezr

That is correct. That said, I still have to merge all of the netstandard facades work into dotnet/runtime, so I don't mind taking care of this one as well once I do that so as to not introduce m ore changes into this PR.

safern · 2020-05-20T00:39:31Z

src/libraries/System.Formats.Asn1/src/System.Formats.Asn1.csproj

+    <IncludeDllSafeSearchPathAttribute>true</IncludeDllSafeSearchPathAttribute>
+    <NoWarn>$(NoWarn);CS1574;CS3016;CA5379;CA5384</NoWarn>
+    <Nullable>enable</Nullable>
+    <TargetFrameworks>netstandard2.0</TargetFrameworks>


This one as well I guess? @joperezr

Same thing, you technically do want them but I'm fine with adding them later if you don't want to introduce more changes into your PR.

src/libraries/System.Formats.Asn1/src/System.Formats.Asn1.csproj

src/libraries/System.Formats.Asn1/ref/System.Formats.Asn1.csproj

src/libraries/System.Formats.Asn1/src/Resources/Strings.resx

src/libraries/System.Formats.Asn1/tests/System.Formats.Asn1.Tests.csproj

src/libraries/System.Formats.Asn1/src/System/Formats/Asn1/AsnCharacterStringEncodings.cs

src/libraries/System.Formats.Asn1/src/System/Formats/Asn1/AsnContentException.cs

src/libraries/System.Formats.Asn1/src/System/Formats/Asn1/AsnDecoder.Enumerated.cs

src/libraries/System.Formats.Asn1/src/System/Formats/Asn1/AsnDecoder.GeneralizedTime.cs

stephentoub · 2020-05-20T14:00:27Z

src/libraries/System.Formats.Asn1/src/System/Formats/Asn1/AsnDecoder.GeneralizedTime.cs

                    }
                }
                else
                {
                    Debug.Fail($"Unhandled value '{octet:X2}' in {nameof(FracState)}");
-                    throw new CryptographicException();
+                    throw new AsnContentException();


Why both the Debug.Fail and the throw?

Yeah, I don't think this Debug.Fail should be here. In general, invalid data from the client is "expected" in that we intend for it to go through graceful exception handling code paths. It doesn't violate invariants in our code.

The fail is appropriate in that the only things which should transition into FracState are seeing a . or a , and both were already handled (an invalid character would have been processed at the point it decided if it was going to the Frac or Offset or Z states). This could be folded with the comma case as an else (as an assert that it's a comma) if there's a strong desire to do so.

stephentoub · 2020-05-20T14:42:32Z

Large PR :) Did a bunch of spot checking. Generally looks good.

src/libraries/System.Formats.Asn1/ref/System.Formats.Asn1.csproj

src/libraries/System.Formats.Asn1/pkg/System.Formats.Asn1.pkgproj

src/libraries/System.Formats.Asn1/src/System.Formats.Asn1.csproj

src/libraries/System.Formats.Asn1/Directory.Build.props

joperezr · 2020-05-20T21:19:08Z

All infrastructure code looks good to me. Just that comment regarding removing refs from package.

src/libraries/System.Formats.Asn1/ref/System.Formats.Asn1.cs

src/libraries/System.Formats.Asn1/src/System/Formats/Asn1/AsnContentException.cs

src/libraries/Common/src/System/Security/Cryptography/Asn1/Pbkdf2SaltChoice.xml.cs

src/libraries/System.Formats.Asn1/src/System/Formats/Asn1/AsnDecoder.Integer.cs

GrabYourPitchforks · 2020-06-04T21:59:20Z

src/libraries/System.Formats.Asn1/src/System/Formats/Asn1/AsnDecoder.Integer.cs

+            {
+                ushort bigEndianValue = BinaryPrimitives.ReadUInt16BigEndian(contents);
+                const ushort RedundancyMask = 0b1111_1111_1000_0000;
+                ushort masked = (ushort)(bigEndianValue & RedundancyMask);


Let me know offline if this is a hot path and we can work to bit-twiddle this in a more optimized fashion.

GrabYourPitchforks · 2020-06-04T22:04:07Z

src/libraries/System.Formats.Asn1/src/System/Formats/Asn1/AsnWriter.cs

@@ -22,8 +22,11 @@ public sealed partial class AsnWriter
        private Stack<StackFrame>? _nestingStack;

        /// <summary>
-        ///   The <see cref="AsnEncodingRules"/> in use by this writer.
+        ///   Gets the encoding rules in use by this writer.


There are some other methods in this file (e.g., Reverse on line 596) that won't be needed once targeting netstandard2.1+ or net5.0+. Are we keeping track of these anywhere?

GrabYourPitchforks · 2020-06-04T22:10:42Z

src/libraries/System.Formats.Asn1/src/System/Formats/Asn1/AsnDecoder.Integer.cs

+            }
+
+            bool isNegative = (contents[0] & 0x80) != 0;
+            long accum = isNegative ? -1 : 0;


Nit: long accum = (sbyte)contents[0];, which will sign-extend the 8-bit contents[0] value to 64 bits. Then we can drop line 449 entirely, and on line 452 start the for loop at i = 1.

src/libraries/System.Formats.Asn1/src/System/Formats/Asn1/AsnDecoder.NamedBitList.cs

src/libraries/System.Formats.Asn1/src/System/Formats/Asn1/AsnDecoder.GeneralizedTime.cs

GrabYourPitchforks · 2020-06-04T22:41:14Z

src/libraries/System.Formats.Asn1/src/System.Formats.Asn1.csproj

@@ -0,0 +1,56 @@
+<Project Sdk="Microsoft.NET.Sdk">


I'm not quite sure how we would do this, but presumably we want this entire project to opt out of the [SkipLocalsInit] behavior that our libraries projects normally get.

Related: #35527

We should also consider having all of the arithmetic in this project checked by default, but the team collectively might find that too big a hammer to wield.

Once #37541 is merged, looks like we'll be able to use <ShouldSkipLocalsInit>false</ShouldSkipLocalsInit> in the .csproj to opt out of this optimization.

opt out of the [SkipLocalsInit] behavior that our libraries projects normally get

For my education, why is that?

GrabYourPitchforks · 2020-06-04T22:51:58Z

src/libraries/Common/src/System/Security/Cryptography/Asn1/asn.xslt

 using System.Runtime.InteropServices;
-using System.Security.Cryptography;
-using System.Security.Cryptography.Asn1;

 namespace <xsl:value-of select="@namespace" />
 {


These structs are marked Sequential. Since they're not expected to be passed via p/invoke, would it make sense instead to annotate them as Auto so that the runtime can optimize them as it sees fit?

This same question applies to other structs, such as Asn1Tag and AsnValueReader. You'd need to consult with somebody more familiar with the runtime + interop to see what an appropriate annotation would be for the structs defined here.

src/libraries/System.Formats.Asn1/src/System/Formats/Asn1/Asn1Tag.cs

src/libraries/System.Formats.Asn1/src/System/Formats/Asn1/SetOfValueComparer.cs

src/libraries/System.Formats.Asn1/src/System.Formats.Asn1.csproj

src/libraries/System.Formats.Asn1/src/System/Formats/Asn1/AsnWriter.cs

src/libraries/System.Formats.Asn1/src/System/Formats/Asn1/AsnWriter.Text.cs

src/libraries/System.Formats.Asn1/src/System/Formats/Asn1/AsnDecoder.BitString.cs

GrabYourPitchforks · 2020-06-05T21:35:45Z

src/libraries/System.Security.Cryptography.Algorithms/src/System/Security/Cryptography/RSA.cs

-                    RSAKeyFormatHelper.ReadRsaPublicKey(firstValue, ignored, out RSAParameters rsaParameters);
+                fixed (byte* ptr = &MemoryMarshal.GetReference(source))
+                {
+                    using (MemoryManager<byte> manager = new PointerMemoryManager<byte>(ptr, localRead))


Future improvement: we should consider getting rid of these call sites if we can. It appears that the only reason the method accepts a ROM<byte> instead of a ROS<byte> is that the memory instance is eventually plumbed through as rebind, but in cases like this where there's no ROM<byte> field on the struct the parameter remains unused. The end effect is that the caller is forced to use unsafe code to wrap a ROM<byte> around a ROS<byte>, when the span itself was all that was needed in the first place.

src/libraries/System.Formats.Asn1/src/System/Formats/Asn1/AsnDecoder.Enumerated.cs

src/libraries/System.Formats.Asn1/src/System/Formats/Asn1/AsnDecoder.OctetString.cs

src/libraries/System.Formats.Asn1/src/System/Formats/Asn1/AsnCharacterStringEncodings.cs

src/libraries/System.Formats.Asn1/src/System/Formats/Asn1/AsnWriter.cs

...m.Security.Cryptography.Pkcs/src/Internal/Cryptography/Pal/Windows/PkcsPalWindows.Encrypt.cs

Some additional tests were written to increase code coverage after the changes

GrabYourPitchforks · 2020-06-09T04:41:06Z

🎉🥳

bartonjs · 2020-06-09T16:04:39Z

Looks like I accidentally hit squash and merge anyways. About half of the moved files ended up looking like moves... so... could have been worse.

Sigh.

Move current ASN src and tests into new tree with no modifications

47bd7a1

AsnReader partials are named AsnDecoder to better match the final name for the core logic they contain.

bartonjs added NO-SQUASH The PR should not be squashed area-System.Security labels May 20, 2020

bartonjs added this to the 5.0 milestone May 20, 2020

bartonjs requested review from GrabYourPitchforks, stephentoub and safern May 20, 2020 00:01

Dotnet-GitSync-Bot added the new-api-needs-documentation label May 20, 2020

bartonjs commented May 20, 2020

View reviewed changes

src/libraries/Common/src/System/Security/Cryptography/Asn1Reader/AsnValueReader.cs Show resolved Hide resolved

safern reviewed May 20, 2020

View reviewed changes

src/libraries/System.Formats.Asn1/Directory.Build.props Show resolved Hide resolved

safern reviewed May 20, 2020

View reviewed changes

src/libraries/System.Formats.Asn1/pkg/System.Formats.Asn1.pkgproj Show resolved Hide resolved

safern reviewed May 20, 2020

View reviewed changes

src/libraries/System.Formats.Asn1/ref/System.Formats.Asn1.csproj Outdated Show resolved Hide resolved

safern reviewed May 20, 2020

View reviewed changes

src/libraries/System.Formats.Asn1/src/System.Formats.Asn1.csproj Outdated Show resolved Hide resolved

safern reviewed May 20, 2020

View reviewed changes

src/libraries/System.Formats.Asn1/src/System.Formats.Asn1.csproj Outdated Show resolved Hide resolved

safern reviewed May 20, 2020

View reviewed changes

src/libraries/System.Formats.Asn1/ref/System.Formats.Asn1.csproj Outdated Show resolved Hide resolved