6.3.4　使用fortio触发熔断

接下来创建一个客户端，用来向后端服务发送请求，观察是否会触发熔断策略。这里要使用一个简单的负载测试客户端，名字叫fortio。这个客户端可以控制连接数量、并发数以及发送HTTP请求的延迟。使用这一客户端，能够有效地触发前面在目标规则中设置的熔断策略。执行如下命名，或者切换到代码目录执行kubectl apply-n cb-f fortio-deploy.yaml：

kubectl apply -n cb -f - <<EOF
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: fortio-deploy
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: fortio
    spec:
      containers:
      - name: fortio
        image: istio/fortio:latest_release
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
          name: http-fortio
        - containerPort: 8079
          name: grpc-ping

EOF

进入客户端pod并使用fortio工具来调用service1服务。在上面的熔断设置中指定了maxConnections：1以及http1MaxPendingRequests：1。这意味着如果超过了一个连接同时发起请求，Istio就会熔断，阻止后续的请求或连接。

首先，确认Fortio_POD运行正常：

FORTIO_POD=$(kubectl get pod -n cb | grep fortio | awk '{ print $1 }')

然后，尝试一下3个并发连接（-c 3），发送30请求（-n 30）：

kubectl exec -n cb -it $FORTIO_POD  -c fortio /usr/local/bin/fortio -- load -c 3 -qps 0 -n 30 -loglevel Warning  http://service1:5678/hello
12:05:40 I logger.go:97> Log level is now 3 Warning (was 2 Info)
Fortio 1.0.1 running at 0 queries per second, 4->4 procs, for 30 calls: http://service1:
5678/hello
Starting at max qps with 3 thread(s) [gomax 4] for exactly 30 calls (10 per thread + 0)
12:05:40 W http_client.go:604> Parsed non ok code 503 (HTTP/1.1 503)
12:05:40 W http_client.go:604> Parsed non ok code 503 (HTTP/1.1 503)
12:05:40 W http_client.go:604> Parsed non ok code 503 (HTTP/1.1 503)
12:05:40 W http_client.go:604> Parsed non ok code 503 (HTTP/1.1 503)
12:05:40 W http_client.go:604> Parsed non ok code 503 (HTTP/1.1 503)
12:05:40 W http_client.go:604> Parsed non ok code 503 (HTTP/1.1 503)
12:05:40 W http_client.go:604> Parsed non ok code 503 (HTTP/1.1 503)
12:05:40 W http_client.go:604> Parsed non ok code 503 (HTTP/1.1 503)
12:05:40 W http_client.go:604> Parsed non ok code 503 (HTTP/1.1 503)
12:05:40 W http_client.go:604> Parsed non ok code 503 (HTTP/1.1 503)
12:05:40 W http_client.go:604> Parsed non ok code 503 (HTTP/1.1 503)
12:05:40 W http_client.go:604> Parsed non ok code 503 (HTTP/1.1 503)
12:05:40 W http_client.go:604> Parsed non ok code 503 (HTTP/1.1 503)
12:05:40 W http_client.go:604> Parsed non ok code 503 (HTTP/1.1 503)
Ended after 57.002494ms : 30 calls. qps=526.29
Aggregated Function Time : count 30 avg 0.0039758417 +/- 0.003147 min 0.000236787 max 
0.010303748 sum 0.119275252
# range, mid point, percentile, count
>= 0.000236787 <= 0.001 , 0.000618394 , 36.67, 11
> 0.001 <= 0.002 , 0.0015 , 40.00, 1
> 0.003 <= 0.004 , 0.0035 , 50.00, 3
> 0.004 <= 0.005 , 0.0045 , 53.33, 1
> 0.005 <= 0.006 , 0.0055 , 70.00, 5
> 0.006 <= 0.007 , 0.0065 , 80.00, 3
> 0.007 <= 0.008 , 0.0075 , 86.67, 2
> 0.008 <= 0.009 , 0.0085 , 96.67, 3
> 0.01 <= 0.0103037 , 0.0101519 , 100.00, 1
# target 50% 0.004
# target 75% 0.0065
# target 90% 0.00833333
# target 99% 0.0102126
# target 99.9% 0.0102946
Sockets used: 16 (for perfect keepalive, would be 3)
Code 200 : 16 (53.3 %)
Code 503 : 14 (46.7 %)
Response Header Sizes : count 30 avg 112 +/- 104.8 min 0 max 210 sum 3360
Response Body/Total Sizes : count 30 avg 276.33333 +/- 51.6 min 217 max 324 sum 8290
All done 30 calls (plus 0 warmup) 3.976 ms avg, 526.3 qps

可以看到熔断行为生效，只有53.3%的请求获得通过，剩余请求被熔断器拦截了。运行多次可以看到超过了一个连接同时发起请求，Istio就会熔断，阻止后续的请求或连接。查询istio-proxy的状态，获取更多相关信息：

kubectl exec -it -n cb $FORTIO_POD  -c istio-proxy  -- sh -c 'curl localhost:
15000/stats' |grep service1 | grep pending
cluster.outbound|5678||service1.cb.svc.cluster.local.upstream_rq_pending_active: 0
cluster.outbound|5678||service1.cb.svc.cluster.local.upstream_rq_pending_failure_eject: 0
cluster.outbound|5678||service1.cb.svc.cluster.local.upstream_rq_pending_overflow: 27
cluster.outbound|5678||service1.cb.svc.cluster.local.upstream_rq_pending_total: 85

其中，upstream_rq_pending_overf low的值是27，说明有27次调用被标志为熔断。

调整熔断触发规则，调整最大连接数，具体如下：

kubectl apply -n cb -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: service1
spec:
  host: service1
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 5
      http:
        http1MaxPendingRequests: 5
        maxRequestsPerConnection: 5
    outlierDetection:
      consecutiveErrors: 1
      interval: 1s
      baseEjectionTime: 3m
      maxEjectionPercent: 100
EOF

再次尝试一下3个并发连接（-c 3），发送30请求（-n 30）：

kubectl exec -n cb -it $FORTIO_POD  -c fortio /usr/local/bin/fortio -- load -c 3 -qps 0 -n 30 -loglevel Warning  http://service1:5678/hello
12:11:43 I logger.go:97> Log level is now 3 Warning (was 2 Info)
Fortio 1.0.1 running at 0 queries per second, 4->4 procs, for 30 calls: http://service1:5678/hello
Starting at max qps with 3 thread(s) [gomax 4] for exactly 30 calls (10 per thread + 0)
Ended after 86.830201ms : 30 calls. qps=345.5
Aggregated Function Time : count 30 avg 0.0082158036 +/- 0.003694 min 0.002951869 
max 0.015740718 sum 0.246474108
# range, mid point, percentile, count
>= 0.00295187 <= 0.003 , 0.00297593 , 3.33, 1
> 0.003 <= 0.004 , 0.0035 , 16.67, 4
> 0.004 <= 0.005 , 0.0045 , 23.33, 2
> 0.005 <= 0.006 , 0.0055 , 36.67, 4
> 0.006 <= 0.007 , 0.0065 , 40.00, 1
> 0.007 <= 0.008 , 0.0075 , 53.33, 4
> 0.009 <= 0.01 , 0.0095 , 76.67, 7
> 0.01 <= 0.011 , 0.0105 , 80.00, 1
> 0.011 <= 0.012 , 0.0115 , 83.33, 1
> 0.012 <= 0.014 , 0.013 , 90.00, 2
> 0.014 <= 0.0157407 , 0.0148704 , 100.00, 3
# target 50% 0.00775
# target 75% 0.00992857
# target 90% 0.014
# target 99% 0.0155666
# target 99.9% 0.0157233
Sockets used: 3 (for perfect keepalive, would be 3)
Code 200 : 30 (100.0 %)
Response Header Sizes : count 30 avg 210.2 +/- 0.4 min 210 max 211 sum 6306
Response Body/Total Sizes : count 30 avg 324.2 +/- 0.4 min 324 max 325 sum 9726
All done 30 calls (plus 0 warmup) 8.216 ms avg, 345.5 qps

可以看到所有请求都通过了，当前的并发请求不会引起熔断触发。