Fix Blackhole implementation for e2e tests

Based on the ideas discussed in the issues [1] and PRs [2][3][6],
we switch from using a L4 reverse proxy to a L7 forward proxy
to properly block peer network traffic, without the need to use
external tools.

The design aims to implement only the minimal required features that
satisfies blocking incoming and outgoing peer traffic. Complicated
features such as packet reordering, packet delivery delay, etc. to
a future container-based solution.

[Background]

A peer will
(a) receive traffic from its peers
(b) initiate connections to its peers (via stream and pipeline).

Thus, the current mechanism of only blocking peer traffic via the peer's
existing reverse proxy is insufficient, since only scenario (a) is
handled, and network traffic in scenario (b) is not blocked at all.

[Proposed solution]

We introduce a L7 forward proxy for each peer, which will be proxying
all the connections initiated from a peer to its peers.

We will remove the current use of the L4 reverse proxy, as the L7
forward proxy holds the information of the destination, we can block
all incoming and outgoing traffic that is initiated from a peer to
others, without having to resort to external tools, such as iptables.

The modified architecture will look something like this:
```
A --- A's forward proxy --- B
   ^ newly introduced
```

[Implementation]

The main subtasks are
- redesigned as an L7 forward proxy
- introduce a new environment variable `E2E_TEST_FORWARD_PROXY_IP` to
bypass the limitation of with http.ProxyFromEnvironment
- implement a L7 forward proxy

Known limitations are
- Doesn't support unix socket, as L7 HTTP transport proxy only support
HTTP/HTTPS/and socks5 -> although e2e test supports unix sockets for
peer communication, but only few of the e2e test cases use unix sockets
as majority of e2e test cases use HTTP/HTTPS. It's been discussed and
decided that without the unix socket support is ok for now.
- it's L7 so we need to send a perfectly crafted HTTP request
- doesn’t support reordering, dropping, manipulating packets on-the-fly

[Testing]
- `make gofail-enable && make build && make gofail-disable && go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1`
- `make gofail-enable && make build && make gofail-disable && go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1`
- `go test -timeout 30s -run ^TestServer_ go.etcd.io/etcd/pkg/v3/proxy -v -failfast`

[References]
[1] issue etcd-io#17737
[2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole
[3] PR (V2) etcd-io#17891
[4] etcd-io#17938 (comment)
[5] etcd-io#17985 (comment)
[6] etcd-io#17938

Signed-off-by: Siyuan Zhang <[email protected]>
Signed-off-by: Iván Valdés Castillo <[email protected]>
Signed-off-by: Chun-Hung Tseng <[email protected]>

client/pkg/transport/transport.go

@@ -18,12 +18,29 @@ import (
 	"context"
 	"net"
 	"net/http"
+	"net/url"
+	"os"
 	"strings"
 	"time"
 )
 
 type unixTransport struct{ *http.Transport }
 
+var httpTransportProxyParsingFunc = determineHTTPTransportProxyParsingFunc
+
+func determineHTTPTransportProxyParsingFunc() func(req *http.Request) (*url.URL, error) {
+	// according to the comment of http.ProxyFromEnvironment: if the proxy URL is "localhost"
+	// (with or without a port number), then a nil URL and nil error will be returned.
+	// Thus, we workaround this limitation by manually setting an ENV named E2E_TEST_FORWARD_PROXY_IP
+	// and parse the URL (which is a localhost in our case)
+	if forwardProxy, exists := os.LookupEnv("E2E_TEST_FORWARD_PROXY_IP"); exists {
+		return func(req *http.Request) (*url.URL, error) {
+			return url.Parse(forwardProxy)
+		}
+	}
+	return http.ProxyFromEnvironment
+}
+
 func NewTransport(info TLSInfo, dialtimeoutd time.Duration) (*http.Transport, error) {
 	cfg, err := info.ClientConfig()
 	if err != nil {
@@ -39,7 +56,7 @@ func NewTransport(info TLSInfo, dialtimeoutd time.Duration) (*http.Transport, er
 	}
 
 	t := &http.Transport{
-		Proxy: http.ProxyFromEnvironment,
+		Proxy: httpTransportProxyParsingFunc(),
 		DialContext: (&net.Dialer{
 			Timeout:   dialtimeoutd,
 			LocalAddr: ipAddr,
@@ -60,7 +77,7 @@ func NewTransport(info TLSInfo, dialtimeoutd time.Duration) (*http.Transport, er
 		return dialer.DialContext(ctx, "unix", addr)
 	}
 	tu := &http.Transport{
-		Proxy:               http.ProxyFromEnvironment,
+		Proxy:               httpTransportProxyParsingFunc(),
 		DialContext:         dialContext,
 		TLSHandshakeTimeout: 10 * time.Second,
 		TLSClientConfig:     cfg,