kafka 消息超出限制解决

最近Kafka在测试中遇到了一个问题
which is the value of the max.request.size configuration.
参考博主 https://huaweicloud.csdn.net/637eea6cdf016f70ae4c983e.html

Kafka配置修改

server.properties中添加

message.max.bytes=5242880
#每个分区试图获取的消息字节数。要大于等于message.max.bytes
replica.fetch.max.bytes=6291456
#原有的socket.send.buffer.bytes,socket.receive.buffer.bytes,socket.request.max.bytes也要改成和message.max.bytes一样的大小

# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=5242880
# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=5242880

# The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600

producer.properties中添加

#请求的最大大小为字节。要小于 message.max.bytes
max.request.size = 5242880

consumer.properties中添加

#每个提取请求中为每个主题分区提取的消息字节数。要大于等于message.max.bytes
fetch.message.max.bytes=6291456

重启kafka

java代码配置

配置以下配置设置最大上传大小解决:
  kafka:
    bootstrap-servers: xxx
    #生产者
    producer:
      # key/value的序列化
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.apache.kafka.common.serialization.StringSerializer
      # 消息投递失败，重试次数
      retries: 3
      #最大发送数量
      batch-size: 20000
      #32MB的批处理缓冲区
      buffer-memory: 33554432
      properties:
        max:
          request:
            size: 20971520

本作品采用知识共享署名-非商业性使用 4.0 国际许可协议进行许可。
本文链接：https://www.blog.ycisch.com/archives/750.html