golang grpc 为何会造成内存泄露?

V2EX = way to explore

V2EX 是一个关于分享和探索的地方

现在注册

已注册用户请登录

这是一个创建于 1099 天前的主题，其中的信息可能已经有所发展或是发生改变。

一个很简单的 grpc 程序, 使用 go-zero 生成的代码, 主要用来代替客户端发送 http 请求, 并且返回 response body/response headers, 仅此而已, 压力测试时发现内存不断的上升!没有停止或者下降, 直到到操作系统 kill, 以下是主要代码:


type Request struct {
}

var httpClientPool = sync.Pool{New: func() interface{}{
	return &http.Client{
		Transport: &http.Transport{
			TLSClientConfig: &tls.Config{
				InsecureSkipVerify: true,
			},
		},
	}
}}


var bufPool = sync.Pool{
	New: func() interface{} {
		return bytes.NewBuffer([]byte{})
	},
}


func (l *RequestLogic) Request(in *cache.RequestMsg) (*cache.ResponseMsg, error) {
	var result = new(cache.ResponseMsg)

	req, err := http.NewRequest(in.Method, string(in.Url), bytes.NewReader(in.Body))
	if err != nil {
		result.ErrorMsg = err.Error()
		return result, errors.Wrap(err, "build request error")
	}
	for k, v := range in.Headers {
		req.Header.Add(k, v)
	}


	client := httpClientPool.Get().(*http.Client)
	defer httpClientPool.Put(client)
	client.Timeout = request_timeout
	if in.ProxyUrl != "" {
		u, _ := url.Parse(in.ProxyUrl)
		if u != nil {
			client.Transport.(*http.Transport).Proxy = http.ProxyURL(u)
		}
	}

	resp, err := client.Do(req)
	if err != nil {
		return result, errors.Wrap(err, "request error")
	}
	defer  resp.Body.Close()

	result.Url = []byte(resp.Request.URL.String())
	result.StatusCode = int32(resp.StatusCode)

	var w = bufPool.Get().(*bytes.Buffer)
	defer bufPool.Put(w)
	if err := gz.Compress(resp.Body, w); err != nil {
		return result, errors.Wrap(err, "gz compress error")
	}

	result.IsCompressor = true
	result.Body = w.Bytes()

	result.Headers = make(map[string]string)
	for k, values := range resp.Header {
		for _, v := range values {
			result.Headers[k] = v
		}
	}

	return result, nil
}

也通过 pprof 查看了, 都是 bytes 分配导致, 但是貌似不是我写的程序问题?

Showing nodes accounting for 930.40MB, 99.07% of 939.12MB total
Dropped 42 nodes (cum <= 4.70MB)
Showing top 10 nodes out of 38
      flat  flat%   sum%        cum   cum%
  760.34MB 80.96% 80.96%   760.34MB 80.96%  google.golang.org/protobuf/proto.MarshalOptions.marshal
  151.13MB 16.09% 97.06%   151.13MB 16.09%  bytes.makeSlice
   12.23MB  1.30% 98.36%    12.23MB  1.30%  google.golang.org/grpc/internal/transport.newBufWriter (inline)
    6.19MB  0.66% 99.02%     6.19MB  0.66%  bufio.NewReaderSize (inline)
    0.50MB 0.053% 99.07%    18.92MB  2.01%  google.golang.org/grpc/internal/transport.NewServerTransport
         0     0% 99.07%   150.62MB 16.04%  bytes.(*Buffer).Write
         0     0% 99.07%   151.13MB 16.09%  bytes.(*Buffer).grow
         0     0% 99.07%   150.62MB 16.04%  compress/flate.(*Writer).Write
         0     0% 99.07%   150.62MB 16.04%  compress/flate.(*compressor).deflate
         0     0% 99.07%   150.62MB 16.04%  compress/flate.(*compressor).write

这咋优化呀?还是直接放弃掉 rpc, 构建一个 http 代理服务器?

3 条回复 • 2021-11-20 19:47:13 +08:00