503 error (api error Unavailable: The service is unavailable) when trying to create VpcEndpoint on new VpcEndpointService #2870
Comments
Tyzanol
added
bug
|
Hi @Tyzanol , If I understand correctly the issue is that In services like EC2 you might encounter propagation delays, so even though the endpoint service itself might be ready, other transient resources that are needed to provision the service are not so it gives you a false positive of Can you please enable the request and response debug logs and share the logs here? Here is how to enable logging: cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithClientLogMode(aws.LogRequestWithBody|aws.LogResponseWithBody))I want to make sure there's nothing obviously wrong with the request, but following this I'll probably ask you to cut an internal ticket via the AWS console. That way you'll get assigned a Technical Account Manager that can drive this case internally on your behalf and investigate this with the service team. Thanks, |
RanVaknin
added
response-requested
|
This issue has not received a response in 1 week. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled. |
github-actions
bot
added
the
closing-soon
|
Hi @RanVaknin , EndpointDetails: RequestID: dc734f1c-d080-45c1-8b95-fe3f8c705b41 Let me know if you need any more details. |
github-actions
bot
removed
closing-soon
|
Hi @Tyzanol , I have redacted the resource information you provided since this is sensitive information that should only be shared with the service team internally. As I mentioned, this will likely get investigated by the service team itself, and not the SDK team. The SDK is a client, so we don't have access to logs / service side resources. I asked to enable the logger in order to inspect the request and response logs so I can make sure the SDK doesn't send / receives corrupted data, or fails to serialize / deserializes it. Thus ruling out SDK specific behavior. To me, the logical thing would have been to use Once you provide the logs I'll have a bit more clarity about the problem at hand. (please make sure to redact any sensitive info from your logs like Credentials and Account information. Thanks, |
|
Hi @RanVaknin, SDK 2023/08/30 14:19:00 DEBUG Request Action=DescribeVpcEndpointServiceConfigurations&Filter.1.Name=tag%3AName&Filter.1.Value.1=ugw-michal-rl-aws-c&Version=2016-11-15 7e1 63657c0a-abbc-45a9-806b-3c292dbb03e2 false eu-central-1a eu-central-1b eu-central-1c vpce-svc-006ccfab24f741253.eu-central-1.vpce.amazonaws.com false arn:aws:elasticloadbalancing:eu-central-1:678183734201:loadbalancer/net/ugw-michal-rl-aws-c/64dd9dbad4434c92 vpce-svc-006ccfab24f741253 com.amazonaws.vpce.eu-central-1.vpce-svc-006ccfab24f741253 Available Interface ipv4 Project ugw-michal Landscape rl-aws-c Name ugw-michal-rl-aws-c 0SDK 2023/08/30 14:19:01 DEBUG Request Action=ModifyVpcEndpointServicePermissions&AddAllowedPrincipals.1=arn%3Aaws%3Aiam%3A%3A678183734201%3Arole%2FAWS-IAM&ServiceId=vpce-svc-006ccfab24f741253&Version=2016-11-15 26d ec906309-48c5-4aee-aabb-6a6d0b6c4355 arn:aws:iam::678183734201:role/AWS-IAM Role vpce-svc-006ccfab24f741253 vpce-svc-perm-0f34e1a4328b005a5 true 0{"level":"info","ts":"2023-08-30T14:19:01.398+0300","caller":"ratelimit/awsSetup.go:392","msg":"vpc endpoint service ID: com.amazonaws.vpce.eu-central-1.vpce-svc-006ccfab24f741253"} Action=DescribeVpcs&Filter.1.Name=tag%3AName&Filter.1.Value.1=shoot--ugw-michal--rl-aws-e&Version=2016-11-15 Action=DescribeSubnets&Filter.1.Name=tag%3AName&Filter.1.Value.1=shoot--ugw-michal--rl-aws-e-nodes-z0&Version=2016-11-15 Action=DescribeSecurityGroups&Filter.1.Name=tag%3AName&Filter.1.Value.1=ugw-michal-rl-aws-c-rl-aws-e&Version=2016-11-15 Action=DescribeVpcEndpoints&Filter.1.Name=tag%3AName&Filter.1.Value.1=ugw-michal-rl-aws-c-rl-aws-e&Version=2016-11-15 ec 4a369b9e-2927-4d65-9eb6-84cb6e87911a 0{"level":"info","ts":"2023-08-30T14:19:02.825+0300","caller":"ratelimit/awsSetup.go:332","msg":"start creating Endpoint resource"} Action=CreateVpcEndpoint&SecurityGroupId.1=sg-0820367be7c04a8bb&ServiceName=com.amazonaws.vpce.eu-central-1.vpce-svc-006ccfab24f741253&SubnetId.1=subnet-0ece697f9d5c58eb6&TagSpecification.1.ResourceType=vpc-endpoint&TagSpecification.1.Tag.1.Key=Name&TagSpecification.1.Tag.1.Value=ugw-michal-rl-aws-c-rl-aws-e&TagSpecification.1.Tag.2.Key=Landscape&TagSpecification.1.Tag.2.Value=rl-aws-c&Version=2016-11-15&VpcEndpointType=Interface&VpcId=vpc-0cca0eb40f93494a2 f7
SDK 2023/08/30 14:19:04 DEBUG Request Action=CreateVpcEndpoint&SecurityGroupId.1=sg-0820367be7c04a8bb&ServiceName=com.amazonaws.vpce.eu-central-1.vpce-svc-006ccfab24f741253&SubnetId.1=subnet-0ece697f9d5c58eb6&TagSpecification.1.ResourceType=vpc-endpoint&TagSpecification.1.Tag.1.Key=Name&TagSpecification.1.Tag.1.Value=ugw-michal-rl-aws-c-rl-aws-e&TagSpecification.1.Tag.2.Key=Landscape&TagSpecification.1.Tag.2.Value=rl-aws-c&Version=2016-11-15&VpcEndpointType=Interface&VpcId=vpc-0cca0eb40f93494a2 f7
SDK 2023/08/30 14:19:08 DEBUG Request Action=CreateVpcEndpoint&SecurityGroupId.1=sg-0820367be7c04a8bb&ServiceName=com.amazonaws.vpce.eu-central-1.vpce-svc-006ccfab24f741253&SubnetId.1=subnet-0ece697f9d5c58eb6&TagSpecification.1.ResourceType=vpc-endpoint&TagSpecification.1.Tag.1.Key=Name&TagSpecification.1.Tag.1.Value=ugw-michal-rl-aws-c-rl-aws-e&TagSpecification.1.Tag.2.Key=Landscape&TagSpecification.1.Tag.2.Value=rl-aws-c&Version=2016-11-15&VpcEndpointType=Interface&VpcId=vpc-0cca0eb40f93494a2 f7
|
|
Hi @RanVaknin, Raya. |
|
Hi @rayakoren, I'm not sure why are you running into this. package main
import (
"context"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/service/ec2"
"log"
)
func main() {
cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithClientLogMode(aws.LogRequestWithBody|aws.LogResponseWithBody))
if err != nil {
log.Fatalf("unable to load SDK config, %v", err)
}
client := ec2.NewFromConfig(cfg)
vpcResp, err := client.CreateVpc(context.TODO(), &ec2.CreateVpcInput{
CidrBlock: aws.String("11.0.0.0/16"),
})
if err != nil {
log.Fatalf("failed to create VPC, %v", err)
}
_, err = client.CreateSubnet(context.TODO(), &ec2.CreateSubnetInput{
CidrBlock: aws.String("11.0.1.0/24"),
VpcId: vpcResp.Vpc.VpcId,
})
if err != nil {
log.Fatalf("failed to create subnet, %v", err)
}
sgResp, err := client.CreateSecurityGroup(context.TODO(), &ec2.CreateSecurityGroupInput{
GroupName: aws.String("MySecurityGroup2"),
Description: aws.String("My security group2"),
VpcId: vpcResp.Vpc.VpcId,
})
if err != nil {
log.Fatalf("failed to create security group, %v", err)
}
_, err = client.CreateVpcEndpoint(context.TODO(), &ec2.CreateVpcEndpointInput{
VpcId: vpcResp.Vpc.VpcId,
ServiceName: aws.String("com.amazonaws.us-east-1.s3"),
VpcEndpointType: "Interface",
SecurityGroupIds: []string{*sgResp.GroupId},
})
if err != nil {
log.Fatalf("failed to create VPC endpoint, %v", err)
}
}My logs: SDK 2023/10/03 09:28:37 DEBUG Request
POST / HTTP/1.1
Host: ec2.us-east-1.amazonaws.com
User-Agent: aws-sdk-go-v2/1.21.0 os/macos lang/go#1.19.1 md/GOOS#darwin md/GOARCH#arm64 api/ec2#1.123.0
Content-Length: 59
Amz-Sdk-Invocation-Id: [REDACTED]
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=AKIA[REDACTED]/20231003/us-east-1/ec2/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date, Signature=[REDACTED]
Content-Type: application/x-www-form-urlencoded
X-Amz-Date: 20231003T162837Z
Accept-Encoding: gzip
Action=CreateVpc&CidrBlock=11.0.0.0%2F16&Version=2016-11-15
SDK 2023/10/03 09:28:37 DEBUG Response
HTTP/1.1 200 OK
Content-Length: 914
Cache-Control: no-cache, no-store
Content-Type: text/xml;charset=UTF-8
Date: Tue, 03 Oct 2023 16:28:37 GMT
Server: AmazonEC2
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Amzn-Requestid: [REDACTED]
<?xml version="1.0" encoding="UTF-8"?>
<CreateVpcResponse xmlns="http://ec2.amazonaws.com/doc/2016-11-15/">
<requestId>[REDACTED]</requestId>
<vpc>
<vpcId>vpc-03d742b7c97aea389</vpcId>
<ownerId>[REDACTED]</ownerId>
<state>pending</state>
<cidrBlock>11.0.0.0/16</cidrBlock>
<cidrBlockAssociationSet>
<item>
<cidrBlock>11.0.0.0/16</cidrBlock>
<associationId>vpc-cidr-assoc-[REDACTED]</associationId>
<cidrBlockState>
<state>associated</state>
</cidrBlockState>
</item>
</cidrBlockAssociationSet>
<ipv6CidrBlockAssociationSet/>
<dhcpOptionsId>dopt-02e990d0c6df32328</dhcpOptionsId>
<instanceTenancy>default</instanceTenancy>
<isDefault>false</isDefault>
</vpc>
</CreateVpcResponse>
SDK 2023/10/03 09:28:37 DEBUG Request
POST / HTTP/1.1
Host: ec2.us-east-1.amazonaws.com
User-Agent: aws-sdk-go-v2/1.21.0 os/macos lang/go#1.19.1 md/GOOS#darwin md/GOARCH#arm64 api/ec2#1.123.0
Content-Length: 90
Amz-Sdk-Invocation-Id: [REDACTED]
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=AKIA[REDACTED]/20231003/us-east-1/ec2/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date, Signature=[REDACTED]
Content-Type: application/x-www-form-urlencoded
X-Amz-Date: 20231003T162837Z
Accept-Encoding: gzip
Action=CreateSubnet&CidrBlock=11.0.1.0%2F24&Version=2016-11-15&VpcId=vpc-03d742b7c97aea389
SDK 2023/10/03 09:28:38 DEBUG Response
HTTP/1.1 200 OK
Content-Length: 1302
Cache-Control: no-cache, no-store
Content-Type: text/xml;charset=UTF-8
Date: Tue, 03 Oct 2023 16:28:37 GMT
Server: AmazonEC2
Strict-Transportation-Security: max-age=31536000; includeSubDomains
X-Amzn-Requestid: [REDACTED]
<?xml version="1.0" encoding="UTF-8"?>
<CreateSubnetResponse xmlns="http://ec2.amazonaws.com/doc/2016-11-15/">
<requestId>[REDACTED]</requestId>
<subnet>
<subnetId>subnet-0ed487786a0539fea</subnetId>
<subnetArn>arn:aws:ec2:us-east-1:[REDACTED]:subnet/subnet-0ed487786a0539fea</subnetArn>
<state>available</state>
<ownerId>[REDACTED]</ownerId>
<vpcId>vpc-03d742b7c97aea389</vpcId>
<cidrBlock>11.0.1.0/24</cidrBlock>
<ipv6CidrBlockAssociationSet/>
<availableIpAddressCount>251</availableIpAddressCount>
<availabilityZone>us-east-1a</availabilityZone>
<availabilityZoneId>use1-az4</availabilityZoneId>
<defaultForAz>false</defaultForAz>
<mapPublicIpOnLaunch>false</mapPublicIpOnLaunch>
<assignIpv6AddressOnCreation>false</assignIpv6AddressOnCreation>
<privateDnsNameOptionsOnLaunch>
<hostnameType>ip-name</hostnameType>
<enableResourceNameDnsARecord>false</enableResourceNameDnsARecord>
<enableResourceNameDnsAAAARecord>false</enableResourceNameDnsAAAARecord>
</privateDnsNameOptionsOnLaunch>
<ipv6Native>false</ipv6Native>
<enableDns64>false</enableDns64>
</subnet>
</CreateSubnetResponse>
SDK 2023/10/03 09:28:38 DEBUG Request
POST / HTTP/1.1
Host: ec2.us-east-1.amazonaws.com
User-Agent: aws-sdk-go-v2/1.21.0 os/macos lang/go#1.19.1 md/GOOS#darwin md/GOARCH#arm64 api/ec2#1.123.0
Content-Length: 136
Amz-Sdk-Invocation-Id: [REDACTED]
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=AKIA[REDACTED]/20231003/us-east-1/ec2/aws4_request,
SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date, Signature=[REDACTED]
Content-Type: application/x-www-form-urlencoded
X-Amz-Date: 20231003T162838Z
Accept-Encoding: gzip
Action=CreateSecurityGroup&GroupDescription=My+security+group2&GroupName=MySecurityGroup2&Version=2016-11-15&VpcId=vpc-03d742b7c97aea389
SDK 2023/10/03 09:28:38 DEBUG Response
HTTP/1.1 200 OK
Content-Length: 283
Cache-Control: no-cache, no-store
Content-Type: text/xml;charset=UTF-8
Date: Tue, 03 Oct 2023 16:28:38 GMT
Server: AmazonEC2
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Amzn-Requestid: [REDACTED]
<?xml version="1.0" encoding="UTF-8"?>
<CreateSecurityGroupResponse xmlns="http://ec2.amazonaws.com/doc/2016-11-15/">
<requestId>[REDACTED]</requestId>
<return>true</return>
<groupId>sg-070e9304b4b5052a8</groupId>
</CreateSecurityGroupResponse>
SDK 2023/10/03 09:28:38 DEBUG Request
POST / HTTP/1.1
Host: ec2.us-east-1.amazonaws.com
User-Agent: aws-sdk-go-v2/1.21.0 os/macos lang/go#1.19.1 md/GOOS#darwin md/GOARCH#arm64 api/ec2#1.123.0
Content-Length: 175
Amz-Sdk-Invocation-Id: [REDACTED]
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=AKIA[REDACTED]/20231003/us-east-1/ec2/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date, Signature=[REDACTED]
Content-Type: application/x-www-form-urlencoded
X-Amz-Date: 20231003T162838Z
Accept-Encoding: gzip
Action=CreateVpcEndpoint&SecurityGroupId.1=sg-070e9304b4b5052a8&ServiceName=com.amazonaws.us-east-1.s3&Version=2016-11-15&VpcEndpointType=Interface&VpcId=vpc-03d742b7c97aea389
SDK 2023/10/03 09:28:38 DEBUG Response
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: no-cache, no-store
Content-Type: text/xml;charset=UTF-8
Date: Tue, 03 Oct 2023 16:28:38 GMT
Server: AmazonEC2
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: accept-encoding
X-Amzn-Requestid: [REDACTED]
<CreateVpcEndpointResponse xmlns="http://ec2.amazonaws.com/doc/2016-11-15/">
<requestId>[REDACTED]</requestId>
<vpcEndpoint>
<creationTimestamp>2023-10-03T16:28:38.795Z</creationTimestamp>
<dnsEntrySet/>
<dnsOptions>
<dnsRecordIpType>ipv4</dnsRecordIpType>
</dnsOptions>
<groupSet>
<item>
<groupId>sg-070e9304b4b5052a8</groupId>
<groupName>MySecurityGroup2</groupName>
</item>
</groupSet>
<ipAddressType>ipv4</ipAddressType>
<networkInterfaceIdSet/>
<ownerId>[REDACTED]</ownerId>
<privateDnsEnabled>false</privateDnsEnabled>
<requesterManaged>false</requesterManaged>
<routeTableIdSet/>
<serviceName>com.amazonaws.us-east-1.s3</serviceName>
<state>pending</state>
<subnetIdSet/>
<vpcEndpointId>vpce-02f257d4a7f27fe01</vpcEndpointId>
<vpcEndpointType>Interface</vpcEndpointType>
<vpcId>vpc-03d742b7c97aea389</vpcId>
</vpcEndpoint>
</CreateVpcEndpointResponse>I'm not using You can check out my code, it is all done in a synchronous fashion, so once subnet is created, the vpc creation code will get called. Let me know if this helps. |
RanVaknin
added
the
response-requested
|
This issue has not received a response in 1 week. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled. |
github-actions
bot
added
the
closing-soon
|
@RanVaknin where are you creating a vpcEndpointService? |
github-actions
bot
removed
closing-soon
RanVaknin
added
the
investigating
|
Hi there, Sorry for the long wait.
It has been a while since I last attempted to reproduce this, but I believe I did create this ahead of time. This is a service API issue, and not an issue with the SDK. The VPC service team did not define a waiter to await the status of the endpoint service, but even after writing my own waiter, its evident that the VPC server returns a false positive. I have created an internal ticket with the VPC service in order to fix this problem (ticket id Will let you know once we hear back. Thanks, |
RanVaknin
removed
the
investigating
|
Thanks @RanVaknin |
Describe the bug
We are using the aws go sdk v2 (ec2) package to create a VpcEndoint.
I we are working on a new VpcEndpointService,
We encounter an a 503 error stating that the VpcEndpointService isn't ready yet.
We tried checking the serviceState of the VpcEndpointService but we get 'ServiceStateAvailable' promptly after the creation.
With that said - we are still receiving the 503 errors when trying to create the VpcEndpoint.
Is there any way to check that the service is ready before creating the VpcEndpoint?
Expected Behavior
Once creating the VpcEndpointService, I don't expect to receive 503 service unavailable messages when trying to create a VpcEndpoint.
If the VpcEndpointService isn't ready - I would like to have some kind of (real) indication when it is indeed available.
As stated above, the serviceState isn't reliable.
Current Behavior
At the moment, we are receiving 503 errors (service unavailable) after trying to create a VpcEndpoint on a newly created VpcEndpointService.
Reproduction Steps
Create a new VpcEndpointService
Possible Solution
No response
Additional Information/Context
No response
AWS Go SDK V2 Module Versions Used
github.com/aws/aws-sdk-go-v2 v1.18.1
github.com/aws/aws-sdk-go-v2/service/ec2 v1.61.0
Compiler and Version used
go1.20.5
Operating System and version
darwin/amd64 - but runs in a kubernetes on docker image
The text was updated successfully, but these errors were encountered: