Ubuntu에 Docker, Kafka, Kafka-UI, Redis, Zookeeper, Opensearch, Logstash 설치하기

Ubuntu나 CentOS 같은 리눅스를 설치하기 위해서는 가상머신이 필요하다.

윈도우 사용자로서 VMware Workstation이나 Virtual Box등이 선택지가 될 수 있는데

VirtualBox에서 Ubuntu를 설치하는 방법은 아래 링크에 잘 나와있다.

 

https://ubuntu.com/tutorials/how-to-run-ubuntu-desktop-on-a-virtual-machine-using-virtualbox#1-overview

 

우분투가 설치된 이후에는 

 

기본적인 것들과 docker, docker-compose를 설치하고 

confluent-kafka, confluent-zookeeper, opensearch, logstash를 설치해본다.

sudo apt-get install apt-transport-https ca-certificates curl gnupg-agent software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt-get update && sudo apt-get upgrade
sudo apt-get install docker-ce docker-ce-cli containerd.io
sudo apt-get install docker-compose
docker-compose -v

 

위의 명령어를 순차적으로 한줄씩 실행하여본다.

 

그리고, 아래의 명령어로

logstash.yml파일로 기본적인 opensearch output host를 잡아준 후

docker-compose로 한번에 confluent-kafka, zookeeper, opensearch, logstash-oss-with-output-plugin 이미지를

도커 허브로부터 당겨와서 한번에 기본 설정으로 올려보도록 하자.

 

vi /data/logstash.yml

input {
  stdin {}


output {
  elasticsearch {
    hosts => ["opensearch:9200"]
    index => "logs-%{+YYYY.MM.dd}"
  }
}

 

vi /data/docker-compose.yml

version: '3'
services:
  opensearch:
    image: opensearchproject/opensearch:2.9.0
    container_name: opensearch
    restart: always
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch
      - discovery.seed_hosts=opensearch
      - cluster.initial_cluster_manager_nodes=opensearch
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
      - discovery.type=single-node
    ulimits:
      memlock:
        soft: 0
        hard: 0
      nofile:
        soft: 65535
        hard: 65535
    volumes:
      - /data/opensearch:/usr/share/opensearch/data
    ports:
      - 9200:9200
      - 9600:9600
    networks:
      - opensearch-net

  logstash-oss:
     image: opensearchproject/logstash-oss-with-opensearch-output-plugin:7.13.2
     container_name: logstash-oss
     volumes:
       - /data/logstash/config/logstash.yml:/config/logstash.yml
         #- /data/logstash/pipeline:/usr/share/logstash/pipeline
     ports:
       - 5044:5044
     restart: always
     environment:
       - ES_HOST=opensearch
     networks:
       - opensearch-net
     depends_on:
       - opensearch
     links:
       - opensearch

  zookeeper:
     image: confluentinc/cp-zookeeper:7.1.2
     hostname: zookeeper
     container_name: zookeeper
     restart: always
     ports:
       - "2181:2181"
     environment:
       ZOOKEEPER_CLIENT_PORT: 2181
       ZOOKEEPER_TICK_TIME: 2000
  kafka:
     image: confluentinc/cp-kafka:7.1.2
     hostname: kafka
     container_name: kafka
     depends_on:
       - zookeeper
     ports:
       - "29092:29092"
       - "9092:9092"
       - "9101:9101"
     restart: always
     environment:
       KAFKA_BROKER_ID: 1
       KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
       KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
       KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://kafka:9092
       KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
       KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
       KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
       KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 1
       KAFKA_JMX_PORT: 9101
       KAFKA_JMX_HOSTNAME: kafka

  kafka-ui:
    image: provectuslabs/kafka-ui
    container_name: kafka-ui
    ports:
      - "8080:8080"
    restart: always
    environment:
      - KAFKA_CLUSTERS_0_NAME=kafka
      - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka:9092
      - KAFKA_CLUSTERS_0_ZOOKEEPER=zookeeper:2181

  redis-container:
    image: redis:7.0
    container_name: redis
    ports: "6379:6379"
    volumes:
      - ./redis/data:/data
      - ./redis/conf/redis.conf:/usr/local/conf/redis.conf
    restart: always
    command: redis-server /usr/local/conf/redis.conf


networks:
  opensearch-net:
    driver: bridge

 

이렇게 한 후, 아래 명령어로 실행하면 구동이 잘 된다.

cd /data
docker-compose up -d