Issue with extracting zip files in local AWS /tmp folder #1138
-
|
Hi everyone, For this I am using this code : zip file has been successfully created in /tmp of aws lambda. Reading zipfile. This line OpenRead can't be executed and throws BadRequest message. |
Beta Was this translation helpful? Give feedback.
Replies: 5 comments 1 reply
-
|
Hi @bveeni, Good afternoon. Looks like you are using 3rd party package
You could also check documentation for 3rd party package Thanks, |
Beta Was this translation helpful? Give feedback.
-
|
Hi Ashish,
Thanks for getting back to me.
This dto from "dto.ImageZipFile.CopyToAsync(fileStream);" is just the
class obj with ImageZipFile as a member of type IFormFile. Here I am just
copying streams to it, which is send as a part of MultipartFormDataContent
via HttpRequestMessage from other console application.
Having tried on mentioned points, it seems like my zip file can't be read
using 3rd party tool ZipFile, whereas when I tried opening this zip file
using code
System.IO.File.OpenRead(zipFilePath);
it doesn't throw error.
With
var archive = ZipFile.OpenRead(zipFilePath);
It throws error with exception with message "Offset to Central Directory
cannot be held in an Int64".
But with local running app, this code works fine and zip file is extracted
properly with ZipFile.Just doesn't work for app running on Lambda.
Is it like, we are not able to use third party library like ZipFile?
Any guidance?
Binita Shrestha
Web Developer
Odessi Group Pty Ltd
…On Sat, Apr 9, 2022 at 9:52 AM Ashish Dhingra ***@***.***> wrote:
Hi @bveeni <https://github.com/bveeni>,
Good afternoon.
Looks like you are using 3rd party package ZipFile to read the ZIP file.
Just to be clear the local /tmp folder is not specific to AWS Lambda, it
can be used by any application/tool. Also, it appears that you are copying
the ZIP file using some application specific
dto.ImageZipFile.CopyToAsync(). So the issue is no where related to
Lambda or related tooling. Few things you could try to troubleshoot:
- After the ZIP file is copied, could you verify that you are able to
read it other 3rd party tool. This is just to verify the ZIP file is not
corrupt.
- Does ZIP file has proper permissions attributes?
- Does your application which is executing ZipFile.OpenRead() command
has permissions to read from /tmp folder?
You could also check documentation for 3rd party package ZipFile to see
if they expose some sort of verbose logging for troubleshooting the issue.
Thanks,
Ashish
—
Reply to this email directly, view it on GitHub
<#1138 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ALM6TT2JGBSHHAB7QUNDRBDVEDBDHANCNFSM5S4FSQEA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Hi Ashish,
I tried other way to check to if the zip file created on /tmp is valid or
not and it gives me invalid file.
For this, I used one valid zip file to send to /tmp directory of lambda and
that zip file from /tmp directory is then uploaded in S3 bucket.
I downloaded this zip file on my pc to check if it is extractable or not.
When I tried to extract it, it throws error as shown in image.
Same thing following in local development gives no error and can be
extracted .
I am quite confused why file created on /tmp directory is invalid and it
happens only for file in lambda /tmp.
[image: image.png]
Any suggestions would be appreciated.
Thanks
Binita Shrestha
Web Developer
Odessi Group Pty Ltd
On Sat, Apr 9, 2022 at 1:15 PM binita shrestha ***@***.***>
wrote:
… Hi Ashish,
Thanks for getting back to me.
This dto from "dto.ImageZipFile.CopyToAsync(fileStream);" is just the
class obj with ImageZipFile as a member of type IFormFile. Here I am just
copying streams to it, which is send as a part of MultipartFormDataContent
via HttpRequestMessage from other console application.
Having tried on mentioned points, it seems like my zip file can't be read
using 3rd party tool ZipFile, whereas when I tried opening this zip file
using code
System.IO.File.OpenRead(zipFilePath);
it doesn't throw error.
With
var archive = ZipFile.OpenRead(zipFilePath);
It throws error with exception with message "Offset to Central Directory
cannot be held in an Int64".
But with local running app, this code works fine and zip file is extracted
properly with ZipFile.Just doesn't work for app running on Lambda.
Is it like, we are not able to use third party library like ZipFile?
Any guidance?
Binita Shrestha
Web Developer
Odessi Group Pty Ltd
On Sat, Apr 9, 2022 at 9:52 AM Ashish Dhingra ***@***.***>
wrote:
> Hi @bveeni <https://github.com/bveeni>,
>
> Good afternoon.
>
> Looks like you are using 3rd party package ZipFile to read the ZIP file.
> Just to be clear the local /tmp folder is not specific to AWS Lambda, it
> can be used by any application/tool. Also, it appears that you are copying
> the ZIP file using some application specific
> dto.ImageZipFile.CopyToAsync(). So the issue is no where related to
> Lambda or related tooling. Few things you could try to troubleshoot:
>
> - After the ZIP file is copied, could you verify that you are able to
> read it other 3rd party tool. This is just to verify the ZIP file is not
> corrupt.
> - Does ZIP file has proper permissions attributes?
> - Does your application which is executing ZipFile.OpenRead() command
> has permissions to read from /tmp folder?
>
> You could also check documentation for 3rd party package ZipFile to see
> if they expose some sort of verbose logging for troubleshooting the issue.
>
> Thanks,
> Ashish
>
> —
> Reply to this email directly, view it on GitHub
> <#1138 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ALM6TT2JGBSHHAB7QUNDRBDVEDBDHANCNFSM5S4FSQEA>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
|
Beta Was this translation helpful? Give feedback.
-
|
@bveeni Before I go ahead and try to recreate the issue, please refer |
Beta Was this translation helpful? Give feedback.
-
|
Hi Ashish,
I tried uploading single image file which is around 457 kb to S3 via lambda
api. The file is uploaded successfully but is corrupted as it can't be
opened and also, I found the file size is increased almost two times, when
it hit to lambda api. I do found similar issue discussion and tried some
solution like adding entry image/png to binary media type to lambda api but
it couldn't resolve the issue.
Is there any other additional configuration to be made for lambda api for
solving this corruption issue?
Binita Shrestha
Web Developer
Odessi Group Pty Ltd
…On Tue, Apr 12, 2022 at 3:17 AM Ashish Dhingra ***@***.***> wrote:
@bveeni <https://github.com/bveeni> Before I go ahead and try to recreate
the issue, please refer What if I need scratch space on disk for my AWS
Lambda function? at https://aws.amazon.com/lambda/faqs/. Each function
has access to 512MB of storage at no additional cost. If your ZIP file and
the extracted size is above this size, then you could get all sorts of
errors.
—
Reply to this email directly, view it on GitHub
<#1138 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ALM6TTY7UBASP4JBVPDXOWTVERNBNANCNFSM5S4FSQEA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Hi @bveeni, Unfortunately, I'm unable to reproduce the issue. For demonstration purposes, used the below code. using Amazon;
using Amazon.Lambda.Core;
using Amazon.S3;
using Amazon.S3.Model;
using System;
using System.IO;
using System.IO.Compression;
using System.Threading.Tasks;
// Assembly attribute to enable the Lambda function's JSON input to be converted into a .NET class.
[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]
namespace LambdaS3DownloadUploadTmp
{
public class Function
{
string bucketName = "<<valid_bucket_name>>";
string imageFileKey = "<<valid_key_for_imagefile>>";
string zipFileKey = "<<valid_key_for_zipfile>>";
RegionEndpoint regionEndpoint = RegionEndpoint.USEast2; // Should be set to proper region endpoint for S3 bucket.
string tmpfolder = Path.GetTempPath(); // "/tmp";
public void FunctionHandler(ILambdaContext context)
{
AmazonS3Client amazonS3Client = new AmazonS3Client(regionEndpoint);
Console.WriteLine("Temp directory path is '{0}'", tmpfolder);
ReadAndSaveObjectDataAsync(amazonS3Client, bucketName, imageFileKey).Wait();
ReadAndSaveObjectDataAsync(amazonS3Client, bucketName, zipFileKey).Wait();
}
async Task ReadAndSaveObjectDataAsync(AmazonS3Client client, string bucketName, string keyName)
{
try
{
string outputKey = Path.GetFileNameWithoutExtension(keyName) + "_" + Guid.NewGuid().ToString() + (Path.GetExtension(keyName) ?? string.Empty);
string outputTmpFile = Path.Combine(tmpfolder, outputKey);
bool isZipFile = false;
GetObjectRequest request = new GetObjectRequest
{
BucketName = bucketName,
Key = keyName
};
using (GetObjectResponse getObjectResponse = await client.GetObjectAsync(request))
using (Stream responseStream = getObjectResponse.ResponseStream)
using (FileStream fileStream = File.OpenWrite(Path.Combine(tmpfolder, outputKey)))
{
byte[] buffer = new byte[2048];
int numBytesRead = 0;
long remainingBytes = responseStream.Length;
string contentType = getObjectResponse.Headers["Content-Type"];
isZipFile = (contentType == "application/zip");
Console.WriteLine("GetObjectResponse: Object Key {0}, Content Type: {1}", keyName, contentType);
Console.WriteLine("BEGIN Writing output file '{0}'", outputTmpFile);
do
{
numBytesRead = responseStream.Read(buffer, 0, (int)Math.Min(buffer.Length, remainingBytes));
if (numBytesRead > 0)
{
fileStream.Write(buffer, 0, numBytesRead);
}
remainingBytes -= numBytesRead;
}
while (remainingBytes > 0);
}
Console.WriteLine("Writing output file '{0}' COMPLETE", outputTmpFile);
if (isZipFile)
{
Console.WriteLine("Opening ZIP file '{0}' for read.", outputTmpFile);
using (var archiveZipFile = ZipFile.OpenRead(outputTmpFile))
{
string outputDirectory = Path.Combine(tmpfolder, outputKey + "_extract");
if (!Directory.Exists(outputDirectory)) Directory.CreateDirectory(outputDirectory);
Console.WriteLine("Extracting ZIP file '{0}' for to directory '{1}'.", outputTmpFile, outputDirectory);
archiveZipFile.ExtractToDirectory(outputDirectory);
}
}
Console.WriteLine("BEGIN Uploading file '{0}' to bucket '{1}'", outputTmpFile, bucketName);
PutObjectResponse putObjectResponse = await client.PutObjectAsync(new PutObjectRequest() { BucketName = bucketName, FilePath = outputTmpFile, Key = outputKey });
Console.WriteLine("Uploading file '{0}' to bucket '{1}' COMPLETE", outputTmpFile, bucketName);
}
catch (Exception e)
{
Console.WriteLine("Unknown error. Message:'{0}'", e.Message);
}
}
}
}.csproj <Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>netcoreapp3.1</TargetFramework>
<GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
<AWSProjectType>Lambda</AWSProjectType>
<!-- This property makes the build directory similar to a publish directory and helps the AWS .NET Lambda Mock Test Tool find project dependencies. -->
<CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Amazon.Lambda.Core" Version="2.1.0" />
<PackageReference Include="Amazon.Lambda.Serialization.SystemTextJson" Version="2.3.0" />
<PackageReference Include="AWSSDK.S3" Version="3.7.8.18" />
</ItemGroup>
</Project>Tested it locally using Mock Lambda Test tool. Thereafter, published the project to Lambda and attached policy Please validate the logic you are using to read/write to streams and handling your ZIP file. I'm not sure what more elaborated guidance could be provided here. The note NOTE: The above code should only be used for demonstration purposes. AWS support is not responsible for any side effects or data loss as a result of using above code. All production code should follow best practices and should be tested thoroughly first in staging/test environment. |
Beta Was this translation helpful? Give feedback.
Hi @bveeni,
Unfortunately, I'm unable to reproduce the issue. For demonstration purposes, used the below code.
Function.cs