Unable to create the EC2 instance

[oskarszoon]

Hi there,

I've successfully used this setup in the past, worked perfectly.

But now I am unable to create the stack, it fails with a message about
AWS::AutoScaling::AutoScalingGroup ScanAutoScalingGroup Received FAILURE signal with UniqueId i-00aeb0f212a7dd7ef

I disabled the rollback and debugged the EC2 instance, in /var/log/cfn-init.log I found the following error message:

2017-01-13 10:51:12,253 [DEBUG] Using service modifier: /sbin/chkconfig
2017-01-13 10:51:12,253 [DEBUG] Setting service clamd.scan to enabled
2017-01-13 10:51:12,256 [INFO] enabled service clamd.scan
2017-01-13 10:51:12,256 [DEBUG] Restarting clamd.scan due to change detected in dependency
2017-01-13 10:51:12,256 [DEBUG] Using service runner: /sbin/service
2017-01-13 10:51:22,683 [ERROR] Could not restart service clamd.scan; return code was 1
2017-01-13 10:51:22,683 [DEBUG] Service output: Stopping clamd.scan: [FAILED]^M
Starting clamd.scan: [FAILED]^M

2017-01-13 10:51:22,684 [ERROR] Error encountered during build of config: Could not restart clamd.scan
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 517, in run_config
    CloudFormationCarpenter(config, self._auth_config).build(worklog)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 258, in build
    CloudFormationCarpenter._serviceTools[manager]().apply(services, changes)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/service_tools.py", line 161, in apply
    self._restart_service(service)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/service_tools.py", line 185, in _restart_service
    raise ToolError("Could not restart %s" % service)
ToolError: Could not restart clamd.scan
2017-01-13 10:51:22,686 [ERROR] -----------------------BUILD FAILED!------------------------
2017-01-13 10:51:22,687 [ERROR] Unhandled exception during build: Could not restart clamd.scan
Traceback (most recent call last):
  File "/opt/aws/bin/cfn-init", line 171, in <module>
    worklog.build(metadata, configSets)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 118, in build
    Contractor(metadata).build(configSets, self)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 505, in build
    self.run_config(config, worklog)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 517, in run_config
    CloudFormationCarpenter(config, self._auth_config).build(worklog)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 258, in build
    CloudFormationCarpenter._serviceTools[manager]().apply(services, changes)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/service_tools.py", line 161, in apply
    self._restart_service(service)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/service_tools.py", line 185, in _restart_service
    raise ToolError("Could not restart %s" % service)
ToolError: Could not restart clamd.scan
2017-01-13 10:51:22,979 [DEBUG] CloudFormation client initialized with endpoint https://cloudformation.eu-central-1.amazonaws.com
2017-01-13 10:51:22,979 [DEBUG] Signaling resource ScanAutoScalingGroup in stack s3-virusscan with unique ID i-00aeb0f212a7dd7ef and status FAILURE

The weird thing is that in the machine itself sudo service clamd.scan restart works fine and properly starts the clamd service (and shows it as stopped when first running). But the stack is of course still showing as CREATE_FAILED

Any help with further debugging, assistance or manually finishing the setup is greatly appreciated.

[michaelwittig]

hi @oskarszoon
any logs from clamd ?

[oskarszoon]

Hi @michaelwittig

I found the following in /var/log/messages

Jan 13 10:51:12 ip-10-1-11-171 clamd[2778]: Received 0 file descriptor(s) from systemd.
Jan 13 10:51:12 ip-10-1-11-171 clamd[2778]: clamd daemon 0.99.2 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
Jan 13 10:51:12 ip-10-1-11-171 clamd[2778]: Running as user clamscan (UID 497, GID 496)
Jan 13 10:51:12 ip-10-1-11-171 clamd[2778]: Log file size limited to 1048576 bytes.
Jan 13 10:51:12 ip-10-1-11-171 clamd[2778]: Reading databases from /var/lib/clamav
Jan 13 10:51:12 ip-10-1-11-171 clamd[2778]: Not loading PUA signatures.
Jan 13 10:51:12 ip-10-1-11-171 clamd[2778]: Bytecode: Security mode set to "TrustSigned".
Jan 13 10:51:20 ip-10-1-11-171 clamd[2778]: Loaded 5554528 signatures.
Jan 13 10:51:21 ip-10-1-11-171 clamd[2778]: LOCAL: Unix socket file /var/run/clamd.scan/clamd.sock
Jan 13 10:51:21 ip-10-1-11-171 clamd[2778]: LOCAL: Setting connection queue length to 200
Jan 13 10:51:21 ip-10-1-11-171 clamd[2778]: daemonize() failed: Cannot allocate memory
Jan 13 10:51:21 ip-10-1-11-171 clamd[2778]: Socket file removed.

So i'll try and redeploy using a different ec type, default from the stack config is t2.micro

[oskarszoon]

Switching to t2.small fixed the issue, so it was definitely a memory issue during the initialization.

@michaelwittig Thanks for pointing me in the right direction. Maybe you can add this new minimum requirement for the instances to your documentation or as a default value in the stack config?

[michaelwittig]

@oskarszoon thanks four your work here. Just to make sure that I get it right.
You used the t2.micro when having the issue
When upgrading to a t2.small the issue is gone?

[oskarszoon]

@michaelwittig Correct

[aquexata]

To use t2.micro (free tier) you can add this to the UserData as workaround:

#!/bin/bash -x
...
/usr/bin/fallocate -l 1G /swapfile && /bin/chmod 600 /swapfile && /sbin/mkswap /swapfile && /sbin/swapon /swapfile

[syed0000]

I am facing difficulty when I try to launch stack for s3 virus scan...its creating an instance but I am not able to see where is the instance its not displaying on ec2 dashboard.

If anybudy have experience to complete this implementation than pls explain