Kafka 内存陡然降低问题

2019-01-09 10:13:54 +08:00
 qwefdrt
最近一直出现 Kafka 集群中所有节点内存瞬间降低的问题。
然后当时的日志里显示：
[2019-01-09 01:08:05,297] WARN Send worker leaving thread (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2019-01-09 01:08:05,296] INFO Received connection request /(org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2019-01-09 01:08:05,447] ERROR Unreasonable buffer length: 100000 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2019-01-09 01:08:05,547] INFO Received connection request /(org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2019-01-09 01:08:05,564] WARN Connection broken for id 8444275087900673, my id = 1, error = (org.apache.zookeeper.server.quorum.QuorumCnxManager)
java.io.IOException: Received packet with invalid packet: 0
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1012)
[2019-01-09 01:08:05,565] WARN Interrupting SendWorker (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2019-01-09 01:08:05,595] WARN Interrupted while waiting for message on queue (org.apache.zookeeper.server.quorum.QuorumCnxManager)
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1094)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:74)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:929)
[2019-01-09 01:08:05,597] WARN Send worker leaving thread (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2019-01-09 01:08:05,597] INFO Received connection request (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2019-01-09 01:08:05,646] WARN Connection broken for id 3377699988963328, my id = 1, error = (org.apache.zookeeper.server.quorum.QuorumCnxManager)
java.io.IOException: Received packet with invalid packet: 0
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1012)
[2019-01-09 01:08:05,670] WARN Interrupting SendWorker (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2019-01-09 01:08:05,670] WARN Interrupted while waiting for message on queue (org.apache.zookeeper.server.quorum.QuorumCnxManager)

请问 ERROR Unreasonable buffer length: 100000 这个是什么原因造成的，应该如何解决？
5756 次点击
所在节点    Kafka
11 条回复
leonme
2019-01-09 10:17:24 +08:00
问大家一个问题：
leonme
2019-01-09 10:17:50 +08:00
@leonme 忽略
petelin
2019-01-09 11:27:44 +08:00
我搜了一下网上的资料, 有一个贴了相关的源码, 看起来是要发送端发送的数据太大了(你这个例子是 10 万字节), 超过了缓冲区, 然后就 close socket 了. (一条消息不会这么大吧, 感觉也有可能是协议解析有问题, 读到是 10 万字节,其实没这么多)
试试发送小一点的数据?

期待下文.
qwefdrt
2019-01-09 11:41:10 +08:00
@petelin 我也是这样认为的，不知道为什么会有这么大的数据。 请问有什么方法限制吗？
petelin
2019-01-09 11:45:50 +08:00
@qwefdrt 客户端业务逻辑限制一下？另外内存陡然下降是因为释放了 socket 吗？报错之后还能不能发消息接收消息
qwefdrt
2019-01-09 12:25:48 +08:00
@petelin 目前还能收发消息，我一遇到这个问题就通过重启来解决哈哈
hackerwin7
2019-01-09 13:17:20 +08:00
kafka 版本？
qwefdrt
2019-01-09 13:44:43 +08:00
@hackerwin7 kafka_2.11-2.0.0
hackerwin7
2019-01-09 14:18:12 +08:00
Kafka 日志？ 写 zookeeper 写太大了
qwefdrt
2019-01-09 18:04:14 +08:00
@hackerwin7 我贴的是 Kafka 日志。Zookeeper 写太大了具体指的是？
hackerwin7
2019-01-09 21:38:09 +08:00
@qwefdrt 这是 zk 的日志，你是不是起的内嵌的 zk ？ 然后我没有看到 broker 相关的日志所以才叫你贴 Kafka 的日志，然后这个 error unreasonable buffer length 其实是 zk 抛出来的，zk 在 parse 的时候一旦 datainpustream 的 length 小于 0 或者超过 Max 就会报这个

这是一个专为移动设备优化的页面（即为了让你能够在 Google 搜索结果里秒开这个页面），如果你希望参与 V2EX 社区的讨论，你可以继续到 V2EX 上打开本讨论主题的完整版本。

https://www.v2ex.com/t/525259

V2EX 是创意工作者们的社区，是一个分享自己正在做的有趣事物、交流想法，可以遇见新朋友甚至新机会的地方。

V2EX is a community of developers, designers and creative people.

© 2021 V2EX