Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[release/7.0] Properly handle filesizes larger than 8 Gb #76756

Merged
merged 4 commits into from
Oct 10, 2022

Conversation

github-actions[bot]
Copy link
Contributor

@github-actions github-actions bot commented Oct 7, 2022

Backport of #76707 to release/7.0

/cc @jozkee

Customer Impact

  • Right now, with any format, an entry greater than 2GB will cause an overflow exception to be thrown
    • The fix addresses this and we can now handle sizes up to what the spec allows, which is 11 octal digits
  • Right now, with V7 and Ustar, an entry greater than 8GB will be silently truncated
    • The fix will make these scenarios throw exceptions since those formats don't support >8GB entries
  • Right now, with other formats, an entry greater than 8GB will also be silently truncated
    • The fix will make these entries get written correctly with "unlimited" sizes
  • PAX is the only entry format that allows entry sizes greater than 8GB (working around the metadata limit)
    • PAX is our default entry format
    • The fix ensures PAX entries can exceed that 8GB limit
  • Impact to dotnet/sdk-container-builds
    • These builds are currently using the GNU format (Microsoft.NET.Build.Containers/Layer.cs) which is subject to the 8GB limit (and the current 2GB size parsing bug)
    • With this fix, dotnet/sdk-container-builds would only be subject to the 8GB limit
    • With this fix, dotnet/sdk-container-builds could switch to PAX and support entries greater than 8GB

Testing

  • Tested roundtripping with 8 Gb files (i.e. tar field size = 8,589,934,591) on all formats.
  • Tested with size 8 Gb + 1 and confirmed formats V7, Ustar, Gnu correctly throw since that wouldn't fit in the 11 octal bytes of the tar field size.
  • Tested roundtripping with 8 Gb + 1 on Pax format, since pax allows to write sizes past the limit as extended properties.

Risk

New feature, no regression involved. There's risk on the newly added tests since those create 8 Gb files, but they are marked as outerloop and parallelization is disabled.

IMPORTANT: Is this backport for a servicing release? If so and this change touches code that ships in a NuGet package, please make certain that you have added any necessary package authoring and gotten it explicitly reviewed.

@jozkee jozkee added this to the 7.0.0 milestone Oct 7, 2022
@jozkee jozkee added the Servicing-consider Issue for next servicing release review label Oct 7, 2022
@ghost
Copy link

ghost commented Oct 7, 2022

Tagging subscribers to this area: @dotnet/area-system-io
See info in area-owners.md if you want to be subscribed.

Issue Details

Backport of #76707 to release/7.0

/cc @jozkee

Customer Impact

Testing

Risk

IMPORTANT: Is this backport for a servicing release? If so and this change touches code that ships in a NuGet package, please make certain that you have added any necessary package authoring and gotten it explicitly reviewed.

Author: github-actions[bot]
Assignees: -
Labels:

area-System.IO

Milestone: -

@carlossanlop carlossanlop added Servicing-approved Approved for servicing release and removed Servicing-consider Issue for next servicing release review labels Oct 7, 2022
@carlossanlop
Copy link
Member

Approved by Tactics via email.

@jozkee

This comment was marked as off-topic.

@github-actions

This comment was marked as off-topic.

@jozkee
Copy link
Member

jozkee commented Oct 7, 2022

/azp run runtime-libraries-coreclr outerloop

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@carlossanlop
Copy link
Member

CI failures unrelated. Approved and signed off. Ready to merge. :shipit:

@carlossanlop carlossanlop merged commit de6b9a3 into release/7.0 Oct 10, 2022
@carlossanlop carlossanlop deleted the backport/pr-76707-to-release/7.0 branch October 10, 2022 15:58
@ghost ghost locked as resolved and limited conversation to collaborators Nov 9, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-System.Formats.Tar Servicing-approved Approved for servicing release
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants