5. Kafka - Kafka-Configs

Kafka의 환경설정은 다양하게 할 수 있다. Kafka Producer의 설정은 Java Application 형태로 개발하였으면 

Property의 항목에 입력하는 K:V 형태로 입력할 수 있고, 

 

Broker나 Topic에 관련한 환경설정은 docker run 시에 입력한다던지,

docker-compose.yml 파일 안에서 입력한다던지 등 다양하게 입력할 수 있다.

 

물론 bitnami-kafka, confluent-kafka, apache-kafka 등 패키징 된 kafka를 제공하는 곳들이라면

환경설정을 매핑해주는 자체적인 환경설정 Key들이 있을 수 있다.

 

여기서는 confluent-kafka의 kafka-configs를 사용한다.

 

configs 설정 확인

docker exec -it kafka kafka-configs --bootstrap-server localhost:9092 --entity-type (brokers/topics) --entity-name (broker id/topic name) --all -describe

 

configs값 설정

docker exec -it kafka kafka-configs --bootstrap-server localhost:9092 --entity-type (brokers/topics) --entity-name (broker id/topic name) --alter --add-config property키=value값

 

configs값 지우기(초기화)

docker exec -it kafka kafka-configs --bootstrap-server localhost:9092 --entity-type (brokders/topic) --entity-name (broker id/topic name) --alter --delete-config property키

 

요렇게 쳐보면

docker exec -it kafka kafka-configs --bootstrap-server localhost:9092 --entity-type brokers --entity-name 1 --all --describe

 

All configs for broker 1 are:

log.cleaner.min.compaction.lag.ms=0 sensitive=false synonyms={DEFAULT_CONFIG:log.cleaner.min.compaction.lag.ms=0}
  offsets.topic.num.partitions=50 sensitive=false synonyms={DEFAULT_CONFIG:offsets.topic.num.partitions=50}
  sasl.oauthbearer.jwks.endpoint.refresh.ms=3600000 sensitive=false synonyms={DEFAULT_CONFIG:sasl.oauthbearer.jwks.endpoint.refresh.ms=3600000}
  log.flush.interval.messages=9223372036854775807 sensitive=false synonyms={DEFAULT_CONFIG:log.flush.interval.messages=9223372036854775807}
  controller.socket.timeout.ms=30000 sensitive=false synonyms={DEFAULT_CONFIG:controller.socket.timeout.ms=30000}
  principal.builder.class=org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder sensitive=false synonyms={DEFAULT_CONFIG:principal.builder.class=org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder}
  log.flush.interval.ms=null sensitive=false synonyms={}
  controller.quoruhttp://m.request.timeout.ms=2000 sensitive=false synonyms={DEFAULT_CONFIG:controller.quoruhttp://m.request.timeout.ms=2000}
  sasl.oauthbearer.expected.audience=null sensitive=false synonyms={}
  min.insync.replicas=1 sensitive=false synonyms={DEFAULT_CONFIG:min.insync.replicas=1}
  nuhttp://m.recovery.threads.per.data.dir=1 sensitive=false synonyms={DEFAULT_CONFIG:nuhttp://m.recovery.threads.per.data.dir=1}
  ssl.keystore.type=JKS sensitive=false synonyms={DEFAULT_CONFIG:ssl.keystore.type=JKS}
  zookeeper.ssl.protocol=TLSv1.2 sensitive=false synonyms={DEFAULT_CONFIG:zookeeper.ssl.protocol=TLSv1.2}
  sasl.mechanishttp://m.inter.broker.protocol=GSSAPI sensitive=false synonyms={DEFAULT_CONFIG:sasl.mechanishttp://m.inter.broker.protocol=GSSAPI}
  metadata.log.segment.bytes=1073741824 sensitive=false synonyms={DEFAULT_CONFIG:metadata.log.segment.bytes=1073741824}
  fetch.purgatory.purge.interval.requests=1000 sensitive=false synonyms={DEFAULT_CONFIG:fetch.purgatory.purge.interval.requests=1000}
  ssl.endpoint.identification.algorithm=https sensitive=false synonyms={DEFAULT_CONFIG:ssl.endpoint.identification.algorithm=https}
  zookeeper.ssl.keystore.location=null sensitive=false synonyms={}
  replica.socket.timeout.ms=30000 sensitive=false synonyms={DEFAULT_CONFIG:replica.socket.timeout.ms=30000}
  message.max.bytes=1048588 sensitive=false synonyms={DEFAULT_CONFIG:message.max.bytes=1048588}
  max.connection.creation.rate=2147483647 sensitive=false synonyms={DEFAULT_CONFIG:max.connection.creation.rate=2147483647}
  connections.max.reauth.ms=0 sensitive=false synonyms={DEFAULT_CONFIG:connections.max.reauth.ms=0}
  log.flush.offset.checkpoint.interval.ms=60000 sensitive=false synonyms={DEFAULT_CONFIG:log.flush.offset.checkpoint.interval.ms=60000}
  zookeeper.clientCnxnSocket=null sensitive=false synonyms={}
  zookeeper.ssl.client.enable=false sensitive=false synonyms={DEFAULT_CONFIG:zookeeper.ssl.client.enable=false}
  quota.window.num=11 sensitive=false synonyms={DEFAULT_CONFIG:quota.window.num=11}
  sasl.oauthbearer.clock.skew.seconds=30 sensitive=false synonyms={DEFAULT_CONFIG:sasl.oauthbearer.clock.skew.seconds=30}
  zookeeper.connect=zookeeper:2181 sensitive=false synonyms={STATIC_BROKER_CONFIG:zookeeper.connect=zookeeper:2181}
  authorizer.class.name= sensitive=false synonyms={DEFAULT_CONFIG:authorizer.class.name=}
  password.encoder.secret=null sensitive=true synonyms={}
  num.replica.fetchers=1 sensitive=false synonyms={DEFAULT_CONFIG:num.replica.fetchers=1}
  alter.log.dirs.replication.quota.window.size.seconds=1 sensitive=false synonyms={DEFAULT_CONFIG:alter.log.dirs.replication.quota.window.size.seconds=1}
  sasl.oauthbearer.jwks.endpoint.url=null sensitive=false synonyms={}
  log.roll.jitter.hours=0 sensitive=false synonyms={DEFAULT_CONFIG:log.roll.jitter.hours=0}
  password.encoder.old.secret=null sensitive=true synonyms={}
  log.cleaner.delete.retention.ms=86400000 sensitive=false synonyms={DEFAULT_CONFIG:log.cleaner.delete.retention.ms=86400000}
  sasl.login.retry.backoff.ms=100 sensitive=false synonyms={DEFAULT_CONFIG:sasl.login.retry.backoff.ms=100}
  queued.max.requests=500 sensitive=false synonyms={DEFAULT_CONFIG:queued.max.requests=500}
  log.cleaner.threads=1 sensitive=false synonyms={DEFAULT_CONFIG:log.cleaner.threads=1}
  sasl.kerberos.service.name=null sensitive=false synonyms={}
  socket.request.max.bytes=104857600 sensitive=false synonyms={DEFAULT_CONFIG:socket.request.max.bytes=104857600}
  log.message.timestamp.type=CreateTime sensitive=false synonyms={DEFAULT_CONFIG:log.message.timestamp.type=CreateTime}
  zookeeper.ssl.keystore.type=null sensitive=false synonyms={}
  connections.max.idle.ms=600000 sensitive=false synonyms={DEFAULT_CONFIG:connections.max.idle.ms=600000}
  zookeeper.set.acl=false sensitive=false synonyms={DEFAULT_CONFIG:zookeeper.set.acl=false}
  delegation.token.expiry.time.ms=86400000 sensitive=false synonyms={DEFAULT_CONFIG:delegation.token.expiry.time.ms=86400000}
  max.connections=2147483647 sensitive=false synonyms={DEFAULT_CONFIG:max.connections=2147483647}
  transaction.state.log.num.partitions=50 sensitive=false synonyms={DEFAULT_CONFIG:transaction.state.log.num.partitions=50}
  controller.quoruhttp://m.election.timeout.ms=1000 sensitive=false synonyms={DEFAULT_CONFIG:controller.quoruhttp://m.election.timeout.ms=1000}
  listener.security.protocol.map=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT sensitive=false synonyms={STATIC_BROKER_CONFIG:listener.security.protocol.map=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT, DEFAULT_CONFIG:listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL}
  log.retention.hours=168 sensitive=false synonyms={DEFAULT_CONFIG:log.retention.hours=168}
  client.quota.callback.class=null sensitive=false synonyms={}
  ssl.provider=null sensitive=false synonyms={}
  delete.records.purgatory.purge.interval.requests=1 sensitive=false synonyms={DEFAULT_CONFIG:delete.records.purgatory.purge.interval.requests=1}
  log.roll.ms=null sensitive=false synonyms={}
  ssl.cipher.suites= sensitive=false synonyms={DEFAULT_CONFIG:ssl.cipher.suites=}
  controller.quoruhttp://m.retry.backoff.ms=20 sensitive=false synonyms={DEFAULT_CONFIG:controller.quoruhttp://m.retry.backoff.ms=20}
  zookeeper.ssl.keystore.password=null sensitive=true synonyms={}
  broker.session.timeout.ms=9000 sensitive=false synonyms={DEFAULT_CONFIG:broker.session.timeout.ms=9000}
  security.inter.broker.protocol=PLAINTEXT sensitive=false synonyms={DEFAULT_CONFIG:security.inter.broker.protocol=PLAINTEXT}
  delegation.token.secret.key=null sensitive=true synonyms={}
  node.id=1 sensitive=false synonyms={STATIC_BROKER_CONFIG:node.id=1, DEFAULT_CONFIG:node.id=-1}
  replica.high.watermark.checkpoint.interval.ms=5000 sensitive=false synonyms={DEFAULT_CONFIG:replica.high.watermark.checkpoint.interval.ms=5000}
  replication.quota.window.size.seconds=1 sensitive=false synonyms={DEFAULT_CONFIG:replication.quota.window.size.seconds=1}
  sasl.kerberos.ticket.renew.window.factor=0.8 sensitive=false synonyms={DEFAULT_CONFIG:sasl.kerberos.ticket.renew.window.factor=0.8}
  zookeeper.connection.timeout.ms=null sensitive=false synonyms={}
  metrics.recording.level=INFO sensitive=false synonyms={DEFAULT_CONFIG:metrics.recording.level=INFO}
  password.encoder.cipher.algorithm=AES/CBC/PKCS5Padding sensitive=false synonyms={DEFAULT_CONFIG:password.encoder.cipher.algorithm=AES/CBC/PKCS5Padding}
  ssl.principal.mapping.rules=DEFAULT sensitive=false synonyms={DEFAULT_CONFIG:ssl.principal.mapping.rules=DEFAULT}
  replica.selector.class=null sensitive=false synonyms={}
  max.connections.per.ip=2147483647 sensitive=false synonyms={DEFAULT_CONFIG:max.connections.per.ip=2147483647}
  background.threads=10 sensitive=false synonyms={DEFAULT_CONFIG:background.threads=10}
  request.timeout.ms=30000 sensitive=false synonyms={DEFAULT_CONFIG:request.timeout.ms=30000}
  log.message.format.version=3.0-IV1 sensitive=false synonyms={DEFAULT_CONFIG:log.message.format.version=3.0-IV1}
  sasl.login.class=null sensitive=false synonyms={}
  log.dir=/tmp/kafka-logs sensitive=false synonyms={DEFAULT_CONFIG:log.dir=/tmp/kafka-logs}
  log.segment.bytes=1073741824 sensitive=false synonyms={DEFAULT_CONFIG:log.segment.bytes=1073741824}
  replica.fetch.response.max.bytes=10485760 sensitive=false synonyms={DEFAULT_CONFIG:replica.fetch.response.max.bytes=10485760}
  group.max.session.timeout.ms=1800000 sensitive=false synonyms={DEFAULT_CONFIG:group.max.session.timeout.ms=1800000}
  controller.listener.names=null sensitive=false synonyms={}
  controller.quoruhttp://m.append.linger.ms=25 sensitive=false synonyms={DEFAULT_CONFIG:controller.quoruhttp://m.append.linger.ms=25}
  log.segment.delete.delay.ms=60000 sensitive=false synonyms={DEFAULT_CONFIG:log.segment.delete.delay.ms=60000}
  log.retention.minutes=null sensitive=false synonyms={}
  log.dirs=/var/lib/kafka/data sensitive=false synonyms={STATIC_BROKER_CONFIG:log.dirs=/var/lib/kafka/data}
  controlled.shutdown.enable=true sensitive=false synonyms={DEFAULT_CONFIG:controlled.shutdown.enable=true}
  socket.connection.setup.timeout.max.ms=30000 sensitive=false synonyms={DEFAULT_CONFIG:socket.connection.setup.timeout.max.ms=30000}
  log.message.timestamp.difference.max.ms=9223372036854775807 sensitive=false synonyms={DEFAULT_CONFIG:log.message.timestamp.difference.max.ms=9223372036854775807}
  sasl.oauthbearer.scope.claim.name=scope sensitive=false synonyms={DEFAULT_CONFIG:sasl.oauthbearer.scope.claim.name=scope}
  password.encoder.key.length=128 sensitive=false synonyms={DEFAULT_CONFIG:password.encoder.key.length=128}
  sasl.login.refresh.min.period.seconds=60 sensitive=false synonyms={DEFAULT_CONFIG:sasl.login.refresh.min.period.seconds=60}
  sasl.oauthbearer.expected.issuer=null sensitive=false synonyms={}
  sasl.login.read.timeout.ms=null sensitive=false synonyms={}
  transaction.abort.timed.out.transaction.cleanup.interval.ms=10000 sensitive=false synonyms={DEFAULT_CONFIG:transaction.abort.timed.out.transaction.cleanup.interval.ms=10000}
  sasl.kerberos.kinit.cmd=/usr/bin/kinit sensitive=false synonyms={DEFAULT_CONFIG:sasl.kerberos.kinit.cmd=/usr/bin/kinit}
  log.cleaner.io.max.bytes.per.second=1.7976931348623157E308 sensitive=false synonyms={DEFAULT_CONFIG:log.cleaner.io.max.bytes.per.second=1.7976931348623157E308}
  auto.leader.rebalance.enable=true sensitive=false synonyms={DEFAULT_CONFIG:auto.leader.rebalance.enable=true}
  leader.imbalance.check.interval.seconds=300 sensitive=false synonyms={DEFAULT_CONFIG:leader.imbalance.check.interval.seconds=300}
  log.cleaner.min.cleanable.ratio=0.5 sensitive=false synonyms={DEFAULT_CONFIG:log.cleaner.min.cleanable.ratio=0.5}
  replica.lag.time.max.ms=30000 sensitive=false synonyms={DEFAULT_CONFIG:replica.lag.time.max.ms=30000}
  num.network.threads=3 sensitive=false synonyms={DEFAULT_CONFIG:num.network.threads=3}
  ssl.keystore.key=null sensitive=true synonyms={}
  sasl.client.callback.handler.class=null sensitive=false synonyms={}
  metrics.num.samples=2 sensitive=false synonyms={DEFAULT_CONFIG:metrics.num.samples=2}
  socket.send.buffer.bytes=102400 sensitive=false synonyms={DEFAULT_CONFIG:socket.send.buffer.bytes=102400}
  password.encoder.keyfactory.algorithm=null sensitive=false synonyms={}
  socket.receive.buffer.bytes=102400 sensitive=false synonyms={DEFAULT_CONFIG:socket.receive.buffer.bytes=102400}
  sasl.oauthbearer.sub.claim.name=sub sensitive=false synonyms={DEFAULT_CONFIG:sasl.oauthbearer.sub.claim.name=sub}
  replica.fetch.min.bytes=1 sensitive=false synonyms={DEFAULT_CONFIG:replica.fetch.min.bytes=1}
  broker.rack=null sensitive=false synonyms={}
  zookeeper.ssl.truststore.password=null sensitive=true synonyms={}
  unclean.leader.election.enable=false sensitive=false synonyms={DEFAULT_CONFIG:unclean.leader.election.enable=false}
  sasl.oauthbearer.jwks.endpoint.retry.backoff.ms=100 sensitive=false synonyms={DEFAULT_CONFIG:sasl.oauthbearer.jwks.endpoint.retry.backoff.ms=100}
  offsets.retention.check.interval.ms=600000 sensitive=false synonyms={DEFAULT_CONFIG:offsets.retention.check.interval.ms=600000}
  producer.purgatory.purge.interval.requests=1000 sensitive=false synonyms={DEFAULT_CONFIG:producer.purgatory.purge.interval.requests=1000}
  controller.quorum.voters= sensitive=false synonyms={DEFAULT_CONFIG:controller.quorum.voters=}
  metrics.sample.window.ms=30000 sensitive=false synonyms={DEFAULT_CONFIG:metrics.sample.window.ms=30000}
  log.retention.check.interval.ms=300000 sensitive=false synonyms={DEFAULT_CONFIG:log.retention.check.interval.ms=300000}
  sasl.login.refresh.window.jitter=0.05 sensitive=false synonyms={DEFAULT_CONFIG:sasl.login.refresh.window.jitter=0.05}
  leader.imbalance.per.broker.percentage=10 sensitive=false synonyms={DEFAULT_CONFIG:leader.imbalance.per.broker.percentage=10}
  controller.quota.window.num=11 sensitive=false synonyms={DEFAULT_CONFIG:controller.quota.window.num=11}
  metric.reporters= sensitive=false synonyms={DEFAULT_CONFIG:metric.reporters=}
  sasl.oauthbearer.token.endpoint.url=null sensitive=false synonyms={}
  auto.create.topics.enable=true sensitive=false synonyms={DEFAULT_CONFIG:auto.create.topics.enable=true}
  ssl.engine.factory.class=null sensitive=false synonyms={}
  replica.socket.receive.buffer.bytes=65536 sensitive=false synonyms={DEFAULT_CONFIG:replica.socket.receive.buffer.bytes=65536}
  zookeeper.ssl.truststore.location=null sensitive=false synonyms={}
  replica.fetch.wait.max.ms=500 sensitive=false synonyms={DEFAULT_CONFIG:replica.fetch.wait.max.ms=500}
  password.encoder.iterations=4096 sensitive=false synonyms={DEFAULT_CONFIG:password.encoder.iterations=4096}
  default.replication.factor=1 sensitive=false synonyms={DEFAULT_CONFIG:default.replication.factor=1}
  ssl.truststore.password=null sensitive=true synonyms={}
  sasl.kerberos.principal.to.local.rules=DEFAULT sensitive=false synonyms={DEFAULT_CONFIG:sasl.kerberos.principal.to.local.rules=DEFAULT}
  log.preallocate=false sensitive=false synonyms={DEFAULT_CONFIG:log.preallocate=false}
  transactional.id.expiration.ms=604800000 sensitive=false synonyms={DEFAULT_CONFIG:transactional.id.expiration.ms=604800000}
  control.plane.listener.name=null sensitive=false synonyms={}
  transaction.state.log.replication.factor=1 sensitive=false synonyms={STATIC_BROKER_CONFIG:transaction.state.log.replication.factor=1, DEFAULT_CONFIG:transaction.state.log.replication.factor=3}
  num.io.threads=8 sensitive=false synonyms={DEFAULT_CONFIG:num.io.threads=8}
  sasl.login.refresh.buffer.seconds=300 sensitive=false synonyms={DEFAULT_CONFIG:sasl.login.refresh.buffer.seconds=300}
  offsets.commit.required.acks=-1 sensitive=false synonyms={DEFAULT_CONFIG:offsets.commit.required.acks=-1}
  connection.failed.authentication.delay.ms=100 sensitive=false synonyms={DEFAULT_CONFIG:connection.failed.authentication.delay.ms=100}
  delete.topic.enable=true sensitive=false synonyms={DEFAULT_CONFIG:delete.topic.enable=true}
  quota.window.size.seconds=1 sensitive=false synonyms={DEFAULT_CONFIG:quota.window.size.seconds=1}
  ssl.truststore.type=JKS sensitive=false synonyms={DEFAULT_CONFIG:ssl.truststore.type=JKS}
  offsets.commit.timeout.ms=5000 sensitive=false synonyms={DEFAULT_CONFIG:offsets.commit.timeout.ms=5000}
  zookeeper.ssl.ocsp.enable=false sensitive=false synonyms={DEFAULT_CONFIG:zookeeper.ssl.ocsp.enable=false}
  broker.heartbeat.interval.ms=2000 sensitive=false synonyms={DEFAULT_CONFIG:broker.heartbeat.interval.ms=2000}
  sasl.mechanism.controller.protocol=GSSAPI sensitive=false synonyms={DEFAULT_CONFIG:sasl.mechanism.controller.protocol=GSSAPI}
  log.cleaner.max.compaction.lag.ms=9223372036854775807 sensitive=false synonyms={DEFAULT_CONFIG:log.cleaner.max.compaction.lag.ms=9223372036854775807}
  zookeeper.ssl.enabled.protocols=null sensitive=false synonyms={}
  log.retention.ms=null sensitive=false synonyms={}
  alter.log.dirs.replication.quota.window.num=11 sensitive=false synonyms={DEFAULT_CONFIG:alter.log.dirs.replication.quota.window.num=11}
  log.cleaner.enable=true sensitive=false synonyms={DEFAULT_CONFIG:log.cleaner.enable=true}
  controller.quoruhttp://m.fetch.timeout.ms=2000 sensitive=false synonyms={DEFAULT_CONFIG:controller.quoruhttp://m.fetch.timeout.ms=2000}
  offsets.load.buffer.size=5242880 sensitive=false synonyms={DEFAULT_CONFIG:offsets.load.buffer.size=5242880}
  ssl.client.auth=none sensitive=false synonyms={DEFAULT_CONFIG:ssl.client.auth=none}
  controlled.shutdown.max.retries=3 sensitive=false synonyms={DEFAULT_CONFIG:controlled.shutdown.max.retries=3}
  offsets.topic.replication.factor=1 sensitive=false synonyms={STATIC_BROKER_CONFIG:offsets.topic.replication.factor=1, DEFAULT_CONFIG:offsets.topic.replication.factor=3}
  zookeeper.ssl.truststore.type=null sensitive=false synonyms={}
  transaction.state.log.min.isr=1 sensitive=false synonyms={STATIC_BROKER_CONFIG:transaction.state.log.min.isr=1, DEFAULT_CONFIG:transaction.state.log.min.isr=2}
  ssl.secure.random.implementation=null sensitive=false synonyms={}
  sasl.kerberos.ticket.renew.jitter=0.05 sensitive=false synonyms={DEFAULT_CONFIG:sasl.kerberos.ticket.renew.jitter=0.05}
  ssl.trustmanager.algorithm=PKIX sensitive=false synonyms={DEFAULT_CONFIG:ssl.trustmanager.algorithm=PKIX}
  zookeeper.session.timeout.ms=18000 sensitive=false synonyms={DEFAULT_CONFIG:zookeeper.session.timeout.ms=18000}
  log.retention.bytes=-1 sensitive=false synonyms={DEFAULT_CONFIG:log.retention.bytes=-1}
  controller.quota.window.size.seconds=1 sensitive=false synonyms={DEFAULT_CONFIG:controller.quota.window.size.seconds=1}
  sasl.jaas.config=null sensitive=true synonyms={}
  sasl.kerberos.min.time.before.relogin=60000 sensitive=false synonyms={DEFAULT_CONFIG:sasl.kerberos.min.time.before.relogin=60000}
  offsets.retention.minutes=10080 sensitive=false synonyms={DEFAULT_CONFIG:offsets.retention.minutes=10080}
  replica.fetch.backoff.ms=1000 sensitive=false synonyms={DEFAULT_CONFIG:replica.fetch.backoff.ms=1000}
  inter.broker.protocol.version=3.1-IV0 sensitive=false synonyms={DEFAULT_CONFIG:inter.broker.protocol.version=3.1-IV0}
  kafka.metrics.reporters= sensitive=false synonyms={DEFAULT_CONFIG:kafka.metrics.reporters=}
  num.partitions=1 sensitive=false synonyms={DEFAULT_CONFIG:num.partitions=1}
  ssl.keystore.certificate.chain=null sensitive=true synonyms={}
  socket.connection.setup.timeout.ms=10000 sensitive=false synonyms={DEFAULT_CONFIG:socket.connection.setup.timeout.ms=10000}
  broker.id.generation.enable=true sensitive=false synonyms={DEFAULT_CONFIG:broker.id.generation.enable=true}
  listeners=PLAINTEXT://0.0.0.0:29092,PLAINTEXT_HOST://0.0.0.0:9092 sensitive=false synonyms={STATIC_BROKER_CONFIG:listeners=PLAINTEXT://0.0.0.0:29092,PLAINTEXT_HOST://0.0.0.0:9092, DEFAULT_CONFIG:listeners=PLAINTEXT://:9092}
  ssl.enabled.protocols=TLSv1.2,TLSv1.3 sensitive=false synonyms={DEFAULT_CONFIG:ssl.enabled.protocols=TLSv1.2,TLSv1.3}
  inter.broker.listener.name=null sensitive=false synonyms={}
  alter.config.policy.class.name=null sensitive=false synonyms={}
  delegation.token.expiry.check.interval.ms=3600000 sensitive=false synonyms={DEFAULT_CONFIG:delegation.token.expiry.check.interval.ms=3600000}
  log.flush.scheduler.interval.ms=9223372036854775807 sensitive=false synonyms={DEFAULT_CONFIG:log.flush.scheduler.interval.ms=9223372036854775807}
  zookeeper.max.in.flight.requests=10 sensitive=false synonyms={DEFAULT_CONFIG:zookeeper.max.in.flight.requests=10}
  log.index.size.max.bytes=10485760 sensitive=false synonyms={DEFAULT_CONFIG:log.index.size.max.bytes=10485760}
  ssl.keymanager.algorithm=SunX509 sensitive=false synonyms={DEFAULT_CONFIG:ssl.keymanager.algorithm=SunX509}
  sasl.login.callback.handler.class=null sensitive=false synonyms={}
  replica.fetch.max.bytes=1048576 sensitive=false synonyms={DEFAULT_CONFIG:replica.fetch.max.bytes=1048576}
  zookeeper.ssl.crl.enable=false sensitive=false synonyms={DEFAULT_CONFIG:zookeeper.ssl.crl.enable=false}
  sasl.server.callback.handler.class=null sensitive=false synonyms={}
  log.cleaner.dedupe.buffer.size=134217728 sensitive=false synonyms={DEFAULT_CONFIG:log.cleaner.dedupe.buffer.size=134217728}
  log.cleaner.io.buffer.size=524288 sensitive=false synonyms={DEFAULT_CONFIG:log.cleaner.io.buffer.size=524288}
  create.topic.policy.class.name=null sensitive=false synonyms={}
  ssl.truststore.certificates=null sensitive=true synonyms={}
  controlled.shutdown.retry.backoff.ms=5000 sensitive=false synonyms={DEFAULT_CONFIG:controlled.shutdown.retry.backoff.ms=5000}
  security.providers=null sensitive=false synonyms={}
  log.roll.hours=168 sensitive=false synonyms={DEFAULT_CONFIG:log.roll.hours=168}
  log.cleanup.policy=delete sensitive=false synonyms={DEFAULT_CONFIG:log.cleanup.policy=delete}
  initial.broker.registration.timeout.ms=60000 sensitive=false synonyms={DEFAULT_CONFIG:initial.broker.registration.timeout.ms=60000}
  log.flush.start.offset.checkpoint.interval.ms=60000 sensitive=false synonyms={DEFAULT_CONFIG:log.flush.start.offset.checkpoint.interval.ms=60000}
  log.roll.jitter.ms=null sensitive=false synonyms={}
  transaction.state.log.segment.bytes=104857600 sensitive=false synonyms={DEFAULT_CONFIG:transaction.state.log.segment.bytes=104857600}
  offsets.topic.segment.bytes=104857600 sensitive=false synonyms={DEFAULT_CONFIG:offsets.topic.segment.bytes=104857600}
  jmx.hostname=null sensitive=true synonyms={STATIC_BROKER_CONFIG:jmx.hostname=null}
  sasl.login.retry.backoff.max.ms=10000 sensitive=false synonyms={DEFAULT_CONFIG:sasl.login.retry.backoff.max.ms=10000}
  group.initial.rebalance.delay.ms=1 sensitive=false synonyms={STATIC_BROKER_CONFIG:group.initial.rebalance.delay.ms=1, DEFAULT_CONFIG:group.initial.rebalance.delay.ms=3000}
  log.index.interval.bytes=4096 sensitive=false synonyms={DEFAULT_CONFIG:log.index.interval.bytes=4096}
  log.cleaner.backoff.ms=15000 sensitive=false synonyms={DEFAULT_CONFIG:log.cleaner.backoff.ms=15000}
  ssl.truststore.location=null sensitive=false synonyms={}
  offset.metadata.max.bytes=4096 sensitive=false synonyms={DEFAULT_CONFIG:offset.metadata.max.bytes=4096}
  ssl.keystore.password=null sensitive=true synonyms={}
  zookeeper.sync.time.ms=2000 sensitive=false synonyms={DEFAULT_CONFIG:zookeeper.sync.time.ms=2000}
  fetch.max.bytes=57671680 sensitive=false synonyms={DEFAULT_CONFIG:fetch.max.bytes=57671680}
  metadata.max.retention.bytes=-1 sensitive=false synonyms={DEFAULT_CONFIG:metadata.max.retention.bytes=-1}
  compression.type=producer sensitive=false synonyms={DEFAULT_CONFIG:compression.type=producer}
  sasl.login.connect.timeout.ms=null sensitive=false synonyms={}
  max.connections.per.ip.overrides= sensitive=false synonyms={DEFAULT_CONFIG:max.connections.per.ip.overrides=}
  sasl.login.refresh.window.factor=0.8 sensitive=false synonyms={DEFAULT_CONFIG:sasl.login.refresh.window.factor=0.8}
  kafka.metrics.polling.interval.secs=10 sensitive=false synonyms={DEFAULT_CONFIG:kafka.metrics.polling.interval.secs=10}
  metadata.log.max.record.bytes.between.snapshots=20971520 sensitive=false synonyms={DEFAULT_CONFIG:metadata.log.max.record.bytes.between.snapshots=20971520}
  metadata.max.retention.ms=604800000 sensitive=false synonyms={DEFAULT_CONFIG:metadata.max.retention.ms=604800000}
  controller.quoruhttp://m.election.backoff.max.ms=1000 sensitive=false synonyms={DEFAULT_CONFIG:controller.quoruhttp://m.election.backoff.max.ms=1000}
  max.incremental.fetch.session.cache.slots=1000 sensitive=false synonyms={DEFAULT_CONFIG:max.incremental.fetch.session.cache.slots=1000}
  delegation.token.master.key=null sensitive=true synonyms={}
  ssl.key.password=null sensitive=true synonyms={}
  reserved.broker.max.id=1000 sensitive=false synonyms={DEFAULT_CONFIG:reserved.broker.max.id=1000}
  transaction.remove.expired.transaction.cleanup.interval.ms=3600000 sensitive=false synonyms={DEFAULT_CONFIG:transaction.remove.expired.transaction.cleanup.interval.ms=3600000}
  log.message.downconversion.enable=true sensitive=false synonyms={DEFAULT_CONFIG:log.message.downconversion.enable=true}
  ssl.protocol=TLSv1.3 sensitive=false synonyms={DEFAULT_CONFIG:ssl.protocol=TLSv1.3}
  metadata.log.segment.ms=604800000 sensitive=false synonyms={DEFAULT_CONFIG:metadata.log.segment.ms=604800000}
  transaction.state.log.load.buffer.size=5242880 sensitive=false synonyms={DEFAULT_CONFIG:transaction.state.log.load.buffer.size=5242880}
  ssl.keystore.location=null sensitive=false synonyms={}
  sasl.enabled.mechanisms=GSSAPI sensitive=false synonyms={DEFAULT_CONFIG:sasl.enabled.mechanisms=GSSAPI}
  nuhttp://m.replica.alter.log.dirs.threads=null sensitive=false synonyms={}
  zookeeper.ssl.cipher.suites=null sensitive=false synonyms={}
  group.min.session.timeout.ms=6000 sensitive=false synonyms={DEFAULT_CONFIG:group.min.session.timeout.ms=6000}
  log.cleaner.io.buffer.load.factor=0.9 sensitive=false synonyms={DEFAULT_CONFIG:log.cleaner.io.buffer.load.factor=0.9}
  transaction.max.timeout.ms=900000 sensitive=false synonyms={DEFAULT_CONFIG:transaction.max.timeout.ms=900000}
  metadata.log.dir=null sensitive=false synonyms={}
  process.roles= sensitive=false synonyms={DEFAULT_CONFIG:process.roles=}
  group.max.size=2147483647 sensitive=false synonyms={DEFAULT_CONFIG:group.max.size=2147483647}
  sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms=10000 sensitive=false synonyms={DEFAULT_CONFIG:sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms=10000}
  delegation.token.max.lifetime.ms=604800000 sensitive=false synonyms={DEFAULT_CONFIG:delegation.token.max.lifetime.ms=604800000}
  broker.id=1 sensitive=false synonyms={STATIC_BROKER_CONFIG:broker.id=1, DEFAULT_CONFIG:broker.id=-1}
  offsets.topic.compression.codec=0 sensitive=false synonyms={DEFAULT_CONFIG:offsets.topic.compression.codec=0}
  zookeeper.ssl.endpoint.identification.algorithm=HTTPS sensitive=false synonyms={DEFAULT_CONFIG:zookeeper.ssl.endpoint.identification.algorithm=HTTPS}
  replication.quota.window.num=11 sensitive=false synonyms={DEFAULT_CONFIG:replication.quota.window.num=11}
  advertised.listeners=PLAINTEXT://kafka:29092,PLAINTEXT_HOST://kafka:9092 sensitive=false synonyms={STATIC_BROKER_CONFIG:advertised.listeners=PLAINTEXT://kafka:29092,PLAINTEXT_HOST://kafka:9092}
  jmx.port=null sensitive=true synonyms={STATIC_BROKER_CONFIG:jmx.port=null}
  queued.max.request.bytes=-1 sensitive=false synonyms={DEFAULT_CONFIG:queued.max.request.bytes=-1}

 

이렇게 정말 많은 동적/정적 환경설정 파라미터가 나온다.

 

이번엔 토픽 레벨의 환경설정 파일을 봐본다.

기존에 만들었던 토픽을 읽어보자

docker exec -it kafka kafka-configs --bootstrap-server localhost:9092 --entity-type topics --entity-name kafka-to-logstash --all --describe

 

All configs for topic kafka-to-logstash are:
  compression.type=producer sensitive=false synonyms={DEFAULT_CONFIG:compression.type=producer}
  leader.replication.throttled.replicas= sensitive=false synonyms={}
  message.downconversion.enable=true sensitive=false synonyms={DEFAULT_CONFIG:log.message.downconversion.enable=true}
  min.insync.replicas=1 sensitive=false synonyms={DEFAULT_CONFIG:min.insync.replicas=1}
  segment.jitter.ms=0 sensitive=false synonyms={}
  cleanup.policy=delete sensitive=false synonyms={DEFAULT_CONFIG:log.cleanup.policy=delete}
  flush.ms=9223372036854775807 sensitive=false synonyms={}
  follower.replication.throttled.replicas= sensitive=false synonyms={}
  segment.bytes=1073741824 sensitive=false synonyms={DEFAULT_CONFIG:log.segment.bytes=1073741824}
  retention.ms=604800000 sensitive=false synonyms={}
  flush.messages=9223372036854775807 sensitive=false synonyms={DEFAULT_CONFIG:log.flush.interval.messages=9223372036854775807}
  message.format.version=3.0-IV1 sensitive=false synonyms={DEFAULT_CONFIG:log.message.format.version=3.0-IV1}
  max.compaction.lag.ms=9223372036854775807 sensitive=false synonyms={DEFAULT_CONFIG:log.cleaner.max.compaction.lag.ms=9223372036854775807}
  file.delete.delay.ms=60000 sensitive=false synonyms={DEFAULT_CONFIG:log.segment.delete.delay.ms=60000}
  max.message.bytes=1048588 sensitive=false synonyms={DEFAULT_CONFIG:message.max.bytes=1048588}
  min.compaction.lag.ms=0 sensitive=false synonyms={DEFAULT_CONFIG:log.cleaner.min.compaction.lag.ms=0}
  message.timestamp.type=CreateTime sensitive=false synonyms={DEFAULT_CONFIG:log.message.timestamp.type=CreateTime}
  preallocate=false sensitive=false synonyms={DEFAULT_CONFIG:log.preallocate=false}
  min.cleanable.dirty.ratio=0.5 sensitive=false synonyms={DEFAULT_CONFIG:log.cleaner.min.cleanable.ratio=0.5}
  index.interval.bytes=4096 sensitive=false synonyms={DEFAULT_CONFIG:log.index.interval.bytes=4096}
  unclean.leader.election.enable=false sensitive=false synonyms={DEFAULT_CONFIG:unclean.leader.election.enable=false}
  retention.bytes=-1 sensitive=false synonyms={DEFAULT_CONFIG:log.retention.bytes=-1}
  delete.retention.ms=86400000 sensitive=false synonyms={DEFAULT_CONFIG:log.cleaner.delete.retention.ms=86400000}
  segment.ms=604800000 sensitive=false synonyms={}
  message.timestamp.difference.max.ms=9223372036854775807 sensitive=false synonyms={DEFAULT_CONFIG:log.message.timestamp.difference.max.ms=9223372036854775807}
  segment.index.bytes=10485760 sensitive=false synonyms={DEFAULT_CONFIG:log.index.size.max.bytes=10485760}

 

이번에는 환경설정을 바꿔보자.

 

docker exec -it kafka kafka-configs --bootstrap-server localhost:9092 --entity-type topics --entity-name kafka-to-logstash --alter --add-config max.message.bytes=2500000

 

이후 다시 grep하여 찾아본다.

 

docker exec -it kafka kafka-configs --bootstrap-server localhost:9092 --entity-type topics --entity-name kafka-to-logstash --all --describe | grep max.message.bytes

  max.message.bytes=2500000 sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:max.message.bytes=2500000, DEFAULT_CONFIG:message.max.bytes=1048588}

 

잘 적용되었다.

이제 다시 초기화 해보자.

 

docker exec -it kafka kafka-configs --bootstrap-server localhost:9092 --entity-type topics --entity-name kafka-to-logstash --alter --delete-config max.message.bytes

 

잘 적용되었는지 확인해보면 적용이 잘 되어있음을 볼 수 있다.

 

docker exec -it kafka kafka-configs --bootstrap-server localhost:9092 --entity-type topics --entity-name kafka-to-logstash --all --describe | grep max.message.bytes

 

  max.message.bytes=1048588 sensitive=false synonyms={DEFAULT_CONFIG:message.max.bytes=1048588}