昨天遇到很奇怪的一个问题, kafka-streams OOM.

2019-09-24 09:33:47 +08:00
 scalaer

报错堆栈:

log4j:WARN No appenders could be found for logger (org.apache.kafka.streams.kstream.internals.InternalStreamsBuilder).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "bugu_rc_streaming-fabc5587-8f91-493c-b0de-cde8b809e442-StreamThread-1" java.lang.OutOfMemoryError: Java heap space
  at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
  at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
  at org.apache.kafka.common.memory.MemoryPool$1.tryAllocate(MemoryPool.java:30)
  at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:112)
  at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:424)
  at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:385)
  at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:651)
  at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:572)
  at org.apache.kafka.common.network.Selector.poll(Selector.java:483)
  at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:539)
  at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:262)
  at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233)
  at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1281)
  at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1225)
  at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1201)
  at org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:88)
  at org.apache.kafka.streams.processor.internals.TaskManager.updateNewAndRestoringTasks(TaskManager.java:328)
  at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:867)
  at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:805)
  at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:774)

不管调多大内存, 一直 oom, 换了一个消费组就正常了, 这是啥情况?

1448 次点击
所在节点    问与答
1 条回复
xaoduer
2019-09-24 16:49:52 +08:00
一脸懵逼

这是一个专为移动设备优化的页面(即为了让你能够在 Google 搜索结果里秒开这个页面),如果你希望参与 V2EX 社区的讨论,你可以继续到 V2EX 上打开本讨论主题的完整版本。

https://www.v2ex.com/t/603579

V2EX 是创意工作者们的社区,是一个分享自己正在做的有趣事物、交流想法,可以遇见新朋友甚至新机会的地方。

V2EX is a community of developers, designers and creative people.

© 2021 V2EX