Update prefix-and-ip-target.md

[nicolajknudsen]

Corrected a few examples with WARM_PREFIX_TARGET=1 and added another example to the list. Small correction of the text clarifying that a t3.small instance has a maximum of 144 IP addresses in prefix mode, and not 176.

---

I'm only somewhat confident of these corrections, the motivation/reasoning that lead me to this change is as follows.

The metric awscni_ip_max shows 144 (9*16) IPs for a t3.small instance, not 176 (11*16) as would have been the case if 11 prefixes had been available. I've had a hard time finding the exact documentation for IP address assignment to ENIs. The first ENI attached to an instance is always assigned the primary IP of the instance, but additional ENIs apparently also have a preallocated IP, thus reducing the number of avalable prefixes to N-1, where N is the number of IPv4 adresses available in the ENI.

If my reasoning above is wrong, then the metric awscni_ip_max is reporting incorrectly low values.

The table with examples was inconsistent with adding zeroes in the 'Pod per Prefixes' column to show that there is always one entire empty prefix allocated. Additionally I've added another example (with 16 pods) to emphasize this behavior. Please do correct me if I'm wrong in this understanding of WARM_PREFIX_TARGET=1

What type of PR is this?
documentation

Which issue does this PR fix:
No issue created

What does this PR do / Why do we need it:
It fixes incorrect documentation

If an issue # is not available please add repro steps and logs from IPAMD/CNI showing the issue:

Testing done on this change:
None

Will this PR introduce any new dependencies?:
No

Will this break upgrades or downgrades? Has updating a running cluster been tested?:
No

Does this change require updates to the CNI daemonset config files to work?:
No

Does this PR introduce any user-facing change?:
No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[jdn5126]

@nicolajknudsen your corrections are accurate here, and thank you for doing this. This issue mentions the problem as well: #2313, but yeah on EC2, the primary IP for each ENI is not available to be assigned to pods.