-
Notifications
You must be signed in to change notification settings - Fork 4.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
.NET should expose a generalized ASN.1 reader/writer API #22610
Comments
I'm interested to see what you come up with so I can compare it to my fast forward-only ASN.1 reader. |
Pretty excited to hear this being considered. I was working on a TSP client a while back and wrote my own generic parser for it. There's naturally BouncyCastle but the API looks quite overcomplicated and alloc-heavy. Let's just say the task is very very far from trivial and since it's usually security related there's a very low margin for error. Sometimes you may want BER+DER in the same input stream or mixed implicit+explicit tagging. Not sure about the attribute/serializer approach. I ditched this pretty early on as it was too complex to implement and not flexible enough. Using an XmlReader-like approach (with peek instead of ReadNext and some sequence scoping) worked quite well though. I could then avoid reflection entirely and add app-specific logic to the deserialization like inline-decoding the context-specific content of an EncapsulatedContentInfo. |
Yeah, the idea is that someone should be able to, for example, look at https://tools.ietf.org/html/rfc5652#section-9.1 and quickly (and correctly) be able to write a structured reader to read the AuthenticatedData object structure. (And, one supposes, write it as well). While it's possible to do that with a reader-style API it's really easy to forget to check for unknown values, which could be there as part of a length-extension (or other) payload attack. |
If it is helps with the consideration and planning at all, I consider this feature important enough to me that I am willing to spend a significant amount of time helping, if you're open to it. |
@vcsjones The core feature work is on the feature-set that I've given up my management chain. There's a chance that we won't want to mark the API as public during the 2.1 timeframe; it depends on how polished we think it is. My official suggestions were that we'd need to start on this first, since it's useful in a number of things I'd like us to be doing, and when we're happy with the API flip it to public. I also decided that the Reader-style API is important in addition to the template/serialization API, for different reasons... so maybe what happens for 2.1 is a good primitive reader, but the serialization part isn't ready. And, of course, there's a chance that all "my" funding gets cut and nothing at all happens here. But I was told I can mark my proposals as being in the 2.1 milestone 😄. |
@bartonjs estimates 1 week remaining |
Awesome! Reader only or does that also include serialization? |
@ebekker The "one week remaining" is for the reader+writer and the serializer+deserializer; all as internal API only. For becoming public it looks like:
The "one week remaining" means that I think there are between 20 and 40 bartonjs-hours remaining to move the "We are here" down one-and-a-half points, so a reader+writer and a serializer+deserializer will be checked in, non-public. There's still a fair bit of things to do before "Make it public", and that means that it's probably not going to be public in the 2.1 release. But it will be in it, getting tested by being the internals powering a couple of things. |
Wow, thanks for the update. |
I was hoping to use this ASN.1 reader/writer in .NET 4.x projects. @bartonjs do I understand it correctly that it will available only for .NET Core? |
@jariq When the ASN.1 types become public API they will become public only for netcoreapp; and then would appear in .NET Framework at some later point; but it won't be applicable retroactively to existing .NET Framework versions. The code for the reader/writer/serializer/deserializer should work on .NET Framework (since it was built using only netstandard references) if you want to copy it into a project locally. |
FWIW, NuGet seems to have had to implement quite a bit of ASN.1 parsing/writing as part of getting signatures working: https://github.com/NuGet/NuGet.Client/tree/dev/src/NuGet.Core/NuGet.Packaging/Signing/Signatures They even have their own DerEncoder. It should would be nice to get something public here so we can all stop re-implementing this :) |
Why does this have to be the case? Can they be added to a new standalone For ASN.1 parsing, that seems to be the kind of thing that can be standalone. |
Sort of. They just copied the first draft of this.
No, because then the platform can't use it. (It's particularly a problem for the ability to go back to .NET Framework) |
Can this be done like the ValueTuple nuget? Separate package for older platforms, built-in for newer platforms. |
@bartonjs can I get a "It works on baronjs machine" seal of "it should work but I'm not gonna give you any guarantees\support" approval, so that I can copy the code into my own project? :) I'd rather use yours then use my own "just write those bytes to disk should be fine" pem reader\writer code :D |
@bartonjs so I was copying the code and noticed this, (yes I couldn't wait) |
In .NET Core 2.1 these classes are powering SignedCms, and the Unix implementation of EnvelopedCms.
Huh, yep. Good thing that's a design-time exception, since it means no one will get a field name as an exception message with CMS 😄. |
@bartonjs I must say that the current AsnWriter is making it a breeze to manually write PKCS files |
@KLuuKer, are you just copying all the code locally and changing all the internal classes to public? Just looking to get an idea of the best approach to experiment with this myself. I would like to support exporting private keys (RSA, EC) using as much in-the-box components (or what will hopefully become) as possible. |
@ebekker I straight up copy the code into a "IBorrowedThisGetRidOfItWhenPublicReleased" folder and remove the calls to the internal you could mark the classes as public, I am making helper methods in that project anyway so I left them internal don't forget to enable languageversion 7.2 |
@bartonjs so I am running into I think a security check from https://github.com/dotnet/corefx/blob/07e9caf00ea0f1893d4c25a5ee287000903fbbe2/src/Common/src/System/Security/Cryptography/AsnWriter.cs#L390 because I am trying to write this value
its the first two bytes of a RSAParameter after bruteforce writing encodes of |
That's an illegal encoding for |
So the real question is who is supposed to fix this? the consumer or the writer? same goes for when reading it back in again, do I need to check for it again before I put it back in my RSAParameter? |
The responsibility of translating between the two formats would belong in the code which speaks both formats (in this case, the code you are writing). Inserting an extra 0x00 when writing isn't always correct (because it can violate the "minimum number of bytes" rule), and removing it during read isn't always correct (because the properties other than Modulus and Exponent have specific required lengths). |
@bartonjs I hate to ask this but do you have some code that does this correctly? I have tried some stuff and that works when I import it back in using my own code var pem = "-----BEGIN RSA PRIVATE KEY-----\n" + Convert.ToBase64String(der) + "-----END RSA PRIVATE KEY-----";
var pr = new PemReader(new StringReader(pem));
var keyPair = (AsymmetricCipherKeyPair)pr.ReadObject();
var rsaParams = DotNetUtilities.ToRSAParameters((RsaPrivateCrtKeyParameters)keyPair.Private); this is how I export mine var asnWriter = new AsnWriter(AsnEncodingRules.DER);
asnWriter.PushSequence();
asnWriter.WriteInteger(0);
asnWriter.WriteInteger(parameters.Modulus);
asnWriter.WriteInteger(parameters.Exponent);
asnWriter.WriteInteger(FixValues(parameters.D));
asnWriter.WriteInteger(FixValues(parameters.P));
asnWriter.WriteInteger(FixValues(parameters.Q));
asnWriter.WriteInteger(FixValues(parameters.DP));
asnWriter.WriteInteger(FixValues(parameters.DQ));
asnWriter.WriteInteger(FixValues(parameters.InverseQ));
asnWriter.PopSequence();
return asnWriter.Encode(); p.s. FixValues is my testing helper to add and remove 0x00 from the byte array |
We'll also most probably need ASN.1 parsing and writing soon. Our use case is not related to x.509 certs, but different data which happens to be serialized in ASN.1. We're aiming for a stable toolchain, if possible including a compiler. So far, we found https://github.com/drasil/BinaryNotes (with its NetStandard fork https://github.com/matizk144/BinaryNotes), which looks promising, but the Compiler is in Java, and the project itself had it's last updates 2017, so it's questionable whether it's still maintained. |
I went through the API and tried to switch public long ReadInt64();
public long ReadInt64(Asn1Tag expectedTag);
public ulong ReadUInt64();
public ulong ReadUInt64(Asn1Tag expectedTag);
public int ReadInt32();
public int ReadInt32(Asn1Tag expectedTag);
public uint ReadUInt32();
public uint ReadUInt32(Asn1Tag expectedTag);
public short ReadInt16();
public short ReadInt16(Asn1Tag expectedTag);
public ushort ReadUInt16();
public ushort ReadUInt16(Asn1Tag expectedTag);
public sbyte ReadInt8();
public sbyte ReadInt8(Asn1Tag expectedTag);
public byte ReadUInt8();
public byte ReadUInt8(Asn1Tag expectedTag); They would act exactly as their Rationale is that all the current usages of the |
Hmm - those could be implemented as Extension Methods, right? |
Yes, they could be implemented as extension methods, but the API is not public yet and there's basically no reason not to add them as first class methods. You could argue that the parameter-less |
The Update: After thinking about it more it's really just an insufficient definition to use
It is used all over the place in CoreFX and it is assigned the normalized serial number (eg. |
There's not a way to say "this is a BigInteger and it must not be negative", but that's just an application problem... same as if a field is So, anywhere that I used |
Stumbled upon this thread and wanted to mention that I have created a DER / ASN.1 encoder/decoder package which can be used with .NET Framework: https://github.com/huysentruitw/pem-utils |
Is there any specific reason why the Asn1Attribute(s) cannot target properties as well as fields? I was thinking of submitting a PR that adds property support because it seems to fit the pattern of serializer attributes throughout the rest of .NET Core. |
Because of point (2) this isn't something that I'd want in the ASN deserializer, since it potentially introduces deserialization vulnerabilities into components which are supposed to make trust decisions. The structures involved in ASN (de)serialization should be minimalistic, and probably not exposed as public API types. |
Good point and makes sense. Thank you.
|
@KLuuKer If you don't mind me asking, how did you resolve |
I'm currently working on an LDAP library (RFC 4511) using version 35320e0 (straight copied as @KLuuKer suggested).
Please let me know if feedback would be helpful at this point or if someone else tried this before. |
@zivillian I'd be happy if you give some feedback. I converted all the internal DER/ASN code to use the new APIs in order to provide some validation of the API design. I'm generally happy with it now except for few issues that are tracked here or in other issues. |
First of all, thanks for your great work. I'm not sure wether this issue is the correct place to track all my possibly unrelated issues and have created a new repo at zivillian/ldap.net. Until now, I've found the following:
Please let me know if those are already tracked somewhere else, or if I should open another issue over here. I may add some more while I'm trying to implement the remaining data types. |
Another pain point is that I would not be able to implement the protocol without a debugger. Whenever I have an error in my type definitions I get a I will definitely look into the code generator mentioned in 28683, but this may not help others. I know that the exception might be security related, but maybe there's a way to set a static property or detect a debugger to get more helpful exceptions during debugging or in unit tests (just skip the catch in https://github.com/dotnet/corefx/blob/35320e086b5f159d74737558f4e3806c53dfaf8c/src/Common/src/System/Security/Cryptography/Asn1V2.Serializer.cs#L367-L375). |
Definitely look at the code generator, it solves the issue with useless exceptions. I will look at the other issues in detail. |
The original exception should be accessible in the InnerException...
Am 18.09.2018 um 21:16 schrieb zivillian <[email protected]<mailto:[email protected]>>:
Another pain point is that I would not be able to implement the protocol without a debugger. Whenever I have an error in my type definitions I get a CryptographicException which is rooted at ASN1 corrupted data.
I will definitely look into the code generator mentioned in 28683, but this may not help others.
I know that the exception might be security related, but maybe there's a way to set a static property or detect a debugger to get more helpful exceptions during debugging or in unit tests (just skip the catch in https://github.com/dotnet/corefx/blob/35320e086b5f159d74737558f4e3806c53dfaf8c/src/Common/src/System/Security/Cryptography/Asn1V2.Serializer.cs#L367-L375).
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub<https://github.com/dotnet/corefx/issues/21833#issuecomment-422513418>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAQnJIsI8ucDSOoRPuliUxvHURGA1KW7ks5ucUZ7gaJpZM4OMk-H>.
|
After trying to migrate to the XML generator I found a blocking issue (zivillian/ldap.net#5) regarding the tag class. With attributes it is possible to specify the tagClass and tagValue: [ExpectedTag(TagClass.Application, 10)]
[OctetString]
public ReadOnlyMemory<byte>? DelRequest; as specified in RFC 4511 - 4.8
I wasn't able to find a way to set the tagClass in xml, since there is no attribute. Using <asn:OctetString name="DelRequest" implicitTag="10"/> results in a if (tag.HasSameClassAndValue(new Asn1Tag(TagClass.ContextSpecific, 10)))
{
if (reader.TryGetPrimitiveOctetStringBytes(new Asn1Tag(TagClass.ContextSpecific, 10), out ReadOnlyMemory<byte> tmpDelRequest))
{
decoded.DelRequest = tmpDelRequest;
}
else
{
decoded.DelRequest = reader.ReadOctetString(new Asn1Tag(TagClass.ContextSpecific, 10));
}
}
else
{
throw new CryptographicException();
} |
@zivillian You could add an attribute in your copy of asn.xsd and process it in asn.xslt to change the class https://github.com/dotnet/corefx/blob/master/src/Common/src/System/Security/Cryptography/Asn1/asn.xslt#L882. We simply haven't had a need for it yet (nothing that we've built used the APPLICATION or PRIVATE classes) |
@bartonjs thanks for your hint, works like a charm zivillian/ldap.net#5 (comment) |
@bartonjs What are the odds of getting these types made public for .NET Core 3? |
@onovotny Public: not great ("temporal mechanics"). But I've closed the gap a bit by copying the reader and writer (and support types) to corefxlab. The API there is different than the one here, because it got an extra pass of me looking at the shape, and starting to write down how I'd present it for API review, and then me making all the fixes that I'd bring up during that meeting if it wasn't my feature :). (The good old "let it sit for a couple months and see if you still like it" strategy) |
I understand that Application/Private/ContextSpecific tag classes might be out of scope here, but it would be great to not leave them out absolutely, as they are widely used in smart cards industry, for example, including banking (see EMV specifications). And it's all mostly related to cryptography in those areas too. The major complexities I see here are:
What if we actually try to break the two layers mentioned before apart in the code and introduce one more to control how the actual Asn.1 elements are serialized, say This way we give users the ability to extend this implementation with their own converters for their Application/Private tag classes. Or even provide extension libraries to deserialize TLV/ASN.1 data into POCO. |
@lil-Toady I'm not sure what the context of your comment is...
The The
I don't follow.
I'm not sure how this would help write to I'm also not sure where you're suggesting it would be / how it would be utilized. The interpretation I'm coming up with is codifying the Encode/Decode contract from what public abstract class AsnSerializer<T>
{
public T Decode(AsnReader reader) { throw null; }
public void Decode(AsnReader reader, out T value) { throw null; }
protected abstract void DecodeCore(AsnReader reader, out T value) { throw null; }
public void Encode(in T value, AsnWriter writer) { throw null; }
protected abstract void EncodeCore(in T value, AsnWriter writer) { throw null; }
} but that still seems to require explicit calls to I'm definitely not understanding what going down to something called |
There's now a spec/feature proposal at dotnet/designs#93. |
Fixed by #36729. |
A lot of features in cryptography and X.509 certificates rely on the ability to read or write ASN.1-structured data. Not only would a good API for ASN.1 reading/writing simplify some of our existing code, it's very handy when someone wants to try adding support for something we don't have inbox, like reading a new certificate extension.
There are a couple of different aspects to the problem:
1) What do inputs/outputs look like?
One of the harder parts to remember to get right is rejecting unexpected data. For example, from https://tools.ietf.org/html/rfc3280#section-4.1.1.2:
The
AlgorithmIdentifier
structure defines two, and only two fields. If a third field is present the data is invalid. (In our current internal implementation this means you needif (reader.HasData) throw new CryptographicException();
at the end of reading it).This means that we really don't want to have to resort to a ReadNext() type API, since it's easy to do it poorly.
One strawman is to build on the model of existing serializers, so one would so something like
The fact that the first example drew a need for a base class might be telling that it needs to be a Sequential-layout class instead of a struct. Or maybe ASN-ANY has to be a byte[] that gets reinterpreted by the caller (but the serializer validates that it was legally encoded).
But sequential ordering on the type seems like a requirement.
C-based libraries generally seem to solve this problem with macro-based template languages. So maybe a template is the way to go. But, maybe not.
Needs to consider:
2) What encodings are supported?
ITU-T X.690 defines 3 different encoding rulesets:
Generally speaking; BER defines all the possibilities for representing data, CER describes a set of restrictions on BER, and DER describes even more restrictions. Occasionally CER and DER give conflicting recommendations, though.
Almost all things in cryptography use DER, making DER a minimum bar. But, since CMS / S/MIME / PKCS#7 says that it's designed to allow BER indefinite length encodings, there are clearly cases where readers need to be more flexible. (Windows and OpenSSL both write all CMS objects with DER; macOS uses indefinite length encodings)
3) What does an API look like?
Largely this comes out of the inputs/outputs section, but encodings need to be taken into account.
If there's a base class then there could be an OnDeserialize()-style validation callback per hydrated member. If it's a convention that ends up being reflection, which isn't super fun. So maybe it's just up to the user... but that's somewhat annoying for embedded structures. Particularly through OPTIONAL.
It probably makes sense to try to design the API, and use it internally, and then eventually expose it (essentially letting internal consumption be the proving ground).
The text was updated successfully, but these errors were encountered: