minio集群部署

由于一些原因，没有在k8s中部署，而是采用了二进制部署集群。步骤如下：

配置数据目录

这里也没有多余的盘，所以直接创建目录。

for i in {1..2};do mkdir -p /minio/data$i;done

部署minio

下载minio

mkdir -p /minio/{bin,scripts,etc}
wget https://dl.min.io/server/minio/release/linux-amd64/minio -O /minio/bin/minio
chmod +x /minio/bin/minio
ln -s /minio/bin/minio /usr/local/bin/minio

启动minio

cat > /minio/scripts/run.sh << EOF
#!/bin/env bash
export MINIO_ROOT_USER=minio
export MINIO_ROOT_PASSWORD=minio123456
minio server --config-dir /minio/etc \
  --console-address ":9001" \
  http://192.168.200.7{3...5}/minio/data{1...2}
EOF
chmod +x /minio/scripts/run.sh

cat > /usr/lib/systemd/system/minio.service << EOF
[Unit]
Description=Minio service
Documentation=https://docs.minio.io/

[Service]
WorkingDirectory=/minio/
ExecStart=/minio/scripts/run.sh

Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl start minio
systemctl enable minio
systemctl status minio.service

访问console

console http://ip:9001
但是无法访问，查看minio报错信息如下:

原因是新版本的minio强制使用非root目录使用的磁盘。

这里没有多余的磁盘，所以采用docker部署。

docker部署

使用nginx来负载均衡，这里只用了一台nginx，也可以使用多台再加上keepalive。

minio1的docker-compose.yaml如下：

version: '3.7'
services:
  minio1:
    image: minio/minio:RELEASE.2022-11-26T22-43-32Z
    volumes:
      - ./minio/data1:/data1
      - ./minio/data2:/data2
    environment:
      MINIO_ROOT_USER: minio
      MINIO_ROOT_PASSWORD: 123456
    network_mode: host
    command: server http://192.168.200.7{3...5}/data{1...2} --address :9000 --console-address :9001
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3
  nginx:
    container_name: nginx
    image: nginx:1.19.2-alpine
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    ports:
      - "80:9000"
    extra_hosts:
      - "minio1:192.168.200.73"
      - "minio2:192.168.200.74"
      - "minio3:192.168.200.75"

nginx.conf

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    # include /etc/nginx/conf.d/*.conf;

    upstream minio {
        server minio1:9001;
        server minio2:9001;
        server minio3:9001;
    }

    server {
        listen       9000;
        listen  [::]:9000;
        server_name  localhost;

         # To allow special characters in headers
         ignore_invalid_headers off;
         # Allow any size file to be uploaded.
         # Set to a value such as 1000m; to restrict file size to a specific value
         client_max_body_size 0;
         # To disable buffering
         proxy_buffering off;

        location / {
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;

            proxy_connect_timeout 300;
            # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            chunked_transfer_encoding off;

            proxy_pass http://minio;
        }
    }
}

minio2,3的docker-compose.yaml如下：

version: '3.7'
services:
  minio1:
    image: minio/minio:RELEASE.2022-11-26T22-43-32Z
    volumes:
      - ./minio/data1:/data1
      - ./minio/data2:/data2
    environment:
      MINIO_ROOT_USER: minio
      MINIO_ROOT_PASSWORD: 123456
    network_mode: host
    command: server http://192.168.200.7{3...5}/data{1...2} --address :9000 --console-address :9001
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

启动

docker-compose up -d

登录

访问：http://192.168.200.73

查看节点