Initial MinIO setup with CI/CD
All checks were successful
Deploy MinIO Production / deploy (push) Successful in 32s

This commit is contained in:
Frank John Begornia
2026-01-06 09:47:50 +08:00
parent c3c3e38f28
commit c9d417bf14
4 changed files with 803 additions and 0 deletions

349
.gitea/workflows/README.md Normal file
View File

@@ -0,0 +1,349 @@
# MinIO Storage - Gitea Actions
This directory contains automated CI/CD workflows for MinIO deployment and maintenance.
## Workflows
### 1. `deploy.yml` - Production Deployment
**Trigger:** Push to `main` or `master` branch
**What it does:**
- Uploads docker-compose.prod.yml to server
- Creates/updates MinIO production container
- Sets up buckets automatically
- Performs health checks
**Deployment location:** `/var/www/apps/minio-storage`
**Access:**
- S3 API: https://minio.crewsportswear.com
- Console: https://console.crewsportswear.com
---
### 2. `deploy-dev.yml` - Development Deployment
**Trigger:** Push to `dev` branch
**What it does:**
- Deploys MinIO to development environment
- Uses default credentials (minioadmin/minioadmin123)
- Exposed on ports 9000/9001
- Sets up development buckets
**Deployment location:** `/var/www/apps/minio-storage-dev`
**Access:**
- S3 API: http://dev.crewsportswear.com:9000
- Console: http://dev.crewsportswear.com:9001
---
### 3. `backup.yml` - Automated Backups
**Trigger:**
- Daily at 2 AM (cron: `0 2 * * *`)
- Manual via workflow_dispatch
**What it does:**
- Backs up all MinIO buckets
- Compresses backups (.tar.gz)
- Stores in `/var/backups/minio/`
- Keeps last 7 days of backups
- Verifies backup completion
**Manual trigger:**
```bash
# Via Gitea UI: Actions → Backup MinIO Buckets → Run Workflow
```
---
## Required Secrets
Configure these in your Gitea repository settings:
**Settings → Secrets → Actions**
| Secret | Description | Example |
|--------|-------------|---------|
| `DEPLOY_SSH_KEY` | Private SSH key for deployment | `-----BEGIN OPENSSH PRIVATE KEY-----...` |
| `DEPLOY_HOST` | Production server hostname/IP | `123.45.67.89` or `server.example.com` |
| `DEPLOY_USER` | SSH username on server | `deploy` or `ubuntu` |
### Generate SSH Key
```bash
# On your local machine
ssh-keygen -t ed25519 -C "gitea-minio-deploy" -f ~/.ssh/gitea_minio_deploy
# Copy public key to server
ssh-copy-id -i ~/.ssh/gitea_minio_deploy.pub user@server
# Copy private key content to Gitea secret
cat ~/.ssh/gitea_minio_deploy
```
---
## Deployment Process
### Production Deployment
1. **Make changes** to docker-compose.prod.yml or setup scripts
2. **Commit and push** to `main` branch:
```bash
git add .
git commit -m "Update MinIO configuration"
git push origin main
```
3. **Watch workflow** in Gitea: Repository → Actions
4. **Verify deployment:**
```bash
curl https://minio.crewsportswear.com/minio/health/live
# Should return: OK
```
### Development Deployment
1. **Create/switch to dev branch:**
```bash
git checkout -b dev
```
2. **Push changes:**
```bash
git push origin dev
```
3. **Access dev environment:**
- Console: http://dev.crewsportswear.com:9001
### Manual Workflow Trigger
You can manually trigger any workflow:
1. Go to: **Repository → Actions**
2. Select workflow (e.g., "Deploy MinIO Production")
3. Click **"Run Workflow"**
4. Select branch
5. Click **"Run Workflow"** button
---
## First-Time Setup
### 1. Configure Server
Ensure your server has:
- Docker and Docker Compose installed
- Networks created: `traefik-public`, `crew-app-net`
- Traefik running (for production HTTPS)
```bash
# On server
docker network create traefik-public
docker network create crew-app-net
```
### 2. Configure Gitea Secrets
Add the three required secrets (see table above).
### 3. Initial Deployment
```bash
# Clone repo
git clone your-gitea-url/minio-storage.git
cd minio-storage
# Push to trigger deployment
git push origin main
```
### 4. Post-Deployment Setup
After first deployment, SSH into server and configure:
```bash
ssh user@server
# Set production credentials
cd /var/www/apps/minio-storage
nano .env
# Update these:
MINIO_ROOT_USER=your_secure_username
MINIO_ROOT_PASSWORD=your_secure_password_32chars
# For secured console, add BasicAuth
htpasswd -nb admin YourPassword
# Copy output to .env:
TRAEFIK_CONSOLE_AUTH='admin:$$apr1$$...'
# Restart
docker compose down && docker compose up -d
```
---
## Monitoring
### Check Deployment Status
```bash
# On server
cd /var/www/apps/minio-storage
# Check container status
docker ps | grep minio
# Check logs
docker logs crew-minio-prod
# View recent deployments
ls -lt /var/www/apps/minio-storage
```
### View Backups
```bash
# On server
ls -lh /var/backups/minio/
# Extract a backup
cd /tmp
tar -xzf /var/backups/minio/minio_backup_YYYYMMDD_HHMMSS.tar.gz
# Restore if needed (see README.md for restore procedure)
```
---
## Troubleshooting
### Deployment Failed
```bash
# Check Gitea Actions logs
# Repository → Actions → Failed workflow → View logs
# Common issues:
# 1. SSH key not added to server authorized_keys
# 2. Server disk space full
# 3. Docker not running on server
```
### Container Not Starting
```bash
ssh user@server
cd /var/www/apps/minio-storage
# Check .env file exists
cat .env
# Check Docker logs
docker logs crew-minio-prod
# Check networks
docker network ls | grep crew-app-net
# Recreate manually
docker compose down
docker compose up -d
```
### Backup Failed
```bash
# Check backup directory permissions
ls -ld /var/backups/minio/
# Check disk space
df -h
# Manual backup
cd /var/www/apps/minio-storage
docker exec crew-minio-prod mc mirror backup/crewsportswear /tmp/manual_backup
```
---
## Workflow Customization
### Change Backup Schedule
Edit `.gitea/workflows/backup.yml`:
```yaml
on:
schedule:
- cron: '0 2 * * *' # Daily 2 AM
# Change to:
- cron: '0 */6 * * *' # Every 6 hours
- cron: '0 0 * * 0' # Weekly on Sunday
```
### Add Post-Deployment Hooks
Edit `.gitea/workflows/deploy.yml`, add step:
```yaml
- name: Notify team
shell: sh
run: |
curl -X POST https://your-webhook-url \
-d "MinIO deployed successfully"
```
### Custom Health Checks
Add to deploy.yml:
```yaml
- name: Test bucket access
shell: sh
run: |
# Test upload
echo "test" > test.txt
curl -X PUT https://minio.crewsportswear.com/crewsportswear/test.txt \
--upload-file test.txt
```
---
## Security Best Practices
1. **Rotate SSH keys** regularly
2. **Use strong MinIO credentials** (32+ characters)
3. **Enable BasicAuth** for console (production)
4. **Monitor backup logs** for failures
5. **Test restore procedure** quarterly
6. **Limit Gitea Actions secrets** access to admins only
---
## Related Documentation
- [Main README](../README.md) - MinIO setup and usage
- [Security Guide](../SECURITY.md) - Security best practices
- [Gitea Actions Docs](https://docs.gitea.com/usage/actions/overview)
---
## Quick Reference
```bash
# View workflows
ls .gitea/workflows/
# Trigger production deploy
git push origin main
# Trigger dev deploy
git push origin dev
# Manual backup (via Gitea UI)
Repository → Actions → Backup MinIO Buckets → Run Workflow
# Check deployment
ssh user@server
docker ps | grep minio
docker logs crew-minio-prod
```

129
.gitea/workflows/backup.yml Normal file
View File

@@ -0,0 +1,129 @@
name: Backup MinIO Buckets
on:
schedule:
# Run daily at 2 AM
- cron: '0 2 * * *'
workflow_dispatch:
jobs:
backup:
runs-on: ubuntu-latest
container:
image: catthehacker/ubuntu:act-latest
steps:
# 1⃣ Setup SSH
- name: Setup SSH
shell: sh
env:
DEPLOY_SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
DEPLOY_HOST: ${{ secrets.DEPLOY_HOST }}
run: |
mkdir -p ~/.ssh
chmod 700 ~/.ssh
echo "$DEPLOY_SSH_KEY" > ~/.ssh/id_ed25519
chmod 600 ~/.ssh/id_ed25519
ssh-keyscan -H "$DEPLOY_HOST" >> ~/.ssh/known_hosts
# 2⃣ Run backup on server
- name: Backup MinIO buckets
shell: sh
env:
DEPLOY_USER: ${{ secrets.DEPLOY_USER }}
DEPLOY_HOST: ${{ secrets.DEPLOY_HOST }}
run: |
ssh -i ~/.ssh/id_ed25519 $DEPLOY_USER@$DEPLOY_HOST << 'EOF'
set -e
BACKUP_BASE="/var/backups/minio"
BACKUP_DIR="$BACKUP_BASE/$(date +%Y%m%d_%H%M%S)"
CONTAINER_NAME="crew-minio-prod"
echo "📦 Starting MinIO backup to $BACKUP_DIR"
sudo mkdir -p "$BACKUP_DIR"
sudo chown $USER:$USER "$BACKUP_DIR"
# Check if container is running
if ! docker ps --format '{{.Names}}' | grep -q "$CONTAINER_NAME"; then
echo "❌ MinIO container is not running"
exit 1
fi
echo "🔧 Installing MinIO client if needed"
docker exec $CONTAINER_NAME sh -c "
if ! command -v mc &> /dev/null; then
curl -sSL https://dl.min.io/client/mc/release/linux-amd64/mc -o /usr/local/bin/mc
chmod +x /usr/local/bin/mc
fi
" || echo "mc already installed"
echo "⚙️ Configuring MinIO client"
docker exec $CONTAINER_NAME mc alias set backup http://localhost:9000 \
${MINIO_ROOT_USER:-minioadmin} ${MINIO_ROOT_PASSWORD:-minioadmin123}
# Backup each bucket
BUCKETS="crewsportswear merchbay merchbay-admin crew-admin email-reports"
for BUCKET in $BUCKETS; do
echo "💾 Backing up bucket: $BUCKET"
# Create bucket directory
sudo mkdir -p "$BACKUP_DIR/$BUCKET"
sudo chown $USER:$USER "$BACKUP_DIR/$BUCKET"
# Export bucket to container temp
docker exec $CONTAINER_NAME mc mirror --overwrite backup/$BUCKET /tmp/backup_$BUCKET/
# Copy from container to host
docker cp $CONTAINER_NAME:/tmp/backup_$BUCKET/. "$BACKUP_DIR/$BUCKET/"
# Cleanup container temp
docker exec $CONTAINER_NAME rm -rf /tmp/backup_$BUCKET
# Count files
FILE_COUNT=$(find "$BACKUP_DIR/$BUCKET" -type f | wc -l)
echo " ✓ Backed up $FILE_COUNT files from $BUCKET"
done
echo "📊 Backup statistics"
du -sh "$BACKUP_DIR"
du -sh "$BACKUP_DIR"/*
echo "🗜️ Compressing backup"
cd "$BACKUP_BASE"
tar -czf "minio_backup_$(date +%Y%m%d_%H%M%S).tar.gz" "$(basename $BACKUP_DIR)"
echo "🧹 Removing uncompressed backup"
rm -rf "$BACKUP_DIR"
echo "🗑️ Cleaning old backups (keeping last 7 days)"
find "$BACKUP_BASE" -name "minio_backup_*.tar.gz" -type f -mtime +7 -delete
echo "📋 Current backups:"
ls -lh "$BACKUP_BASE"/minio_backup_*.tar.gz 2>/dev/null || echo "No compressed backups found"
echo "✅ Backup completed successfully!"
EOF
# 3⃣ Verify backup
- name: Verify backup
shell: sh
env:
DEPLOY_USER: ${{ secrets.DEPLOY_USER }}
DEPLOY_HOST: ${{ secrets.DEPLOY_HOST }}
run: |
ssh -i ~/.ssh/id_ed25519 $DEPLOY_USER@$DEPLOY_HOST << 'EOF'
LATEST_BACKUP=$(ls -t /var/backups/minio/minio_backup_*.tar.gz 2>/dev/null | head -1)
if [ -f "$LATEST_BACKUP" ]; then
echo "✅ Latest backup verified:"
ls -lh "$LATEST_BACKUP"
BACKUP_SIZE=$(du -sh "$LATEST_BACKUP" | awk '{print $1}')
echo "📦 Backup size: $BACKUP_SIZE"
else
echo "❌ No backup file found!"
exit 1
fi
EOF

View File

@@ -0,0 +1,155 @@
name: Deploy MinIO Development
on:
push:
branches:
- dev
workflow_dispatch:
jobs:
deploy:
runs-on: ubuntu-latest
container:
image: catthehacker/ubuntu:act-latest
steps:
# 1⃣ Checkout code
- name: Checkout code
shell: sh
run: |
git clone $GITHUB_SERVER_URL/$GITHUB_REPOSITORY.git /workspace/repo
cd /workspace/repo
git checkout $GITHUB_REF_NAME
# 2⃣ Setup SSH
- name: Setup SSH
shell: sh
env:
DEPLOY_SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
DEPLOY_HOST: ${{ secrets.DEPLOY_HOST }}
run: |
mkdir -p ~/.ssh
chmod 700 ~/.ssh
echo "$DEPLOY_SSH_KEY" > ~/.ssh/id_ed25519
chmod 600 ~/.ssh/id_ed25519
ssh-keyscan -H "$DEPLOY_HOST" >> ~/.ssh/known_hosts
# 3⃣ Upload compose files
- name: Upload compose and scripts
shell: sh
env:
DEPLOY_USER: ${{ secrets.DEPLOY_USER }}
DEPLOY_HOST: ${{ secrets.DEPLOY_HOST }}
run: |
scp -i ~/.ssh/id_ed25519 \
/workspace/repo/docker-compose.yml \
/workspace/repo/setup-buckets.sh \
${DEPLOY_USER}@${DEPLOY_HOST}:/tmp/
# 4⃣ Deploy on server
- name: Deploy MinIO Dev on server
shell: sh
env:
DEPLOY_USER: ${{ secrets.DEPLOY_USER }}
DEPLOY_HOST: ${{ secrets.DEPLOY_HOST }}
run: |
ssh -i ~/.ssh/id_ed25519 $DEPLOY_USER@$DEPLOY_HOST << 'EOF'
set -e
DEPLOY_DIR="/var/www/apps/minio-storage-dev"
sudo mkdir -p "$DEPLOY_DIR"
sudo chown $USER:$USER "$DEPLOY_DIR"
echo "📄 Updating compose file and scripts"
cp /tmp/docker-compose.yml "$DEPLOY_DIR/"
cp /tmp/setup-buckets.sh "$DEPLOY_DIR/"
chmod +x "$DEPLOY_DIR/setup-buckets.sh"
cd "$DEPLOY_DIR"
echo "🔍 Checking .env file"
if [ ! -f .env ]; then
echo "⚠️ .env file not found, creating dev config"
echo "# MinIO Development Configuration" > .env
echo "MINIO_ROOT_USER=minioadmin" >> .env
echo "MINIO_ROOT_PASSWORD=minioadmin123" >> .env
echo "MINIO_SERVER_URL=http://dev.crewsportswear.com:9000" >> .env
echo "MINIO_BROWSER_REDIRECT_URL=http://dev.crewsportswear.com:9001" >> .env
echo "MINIO_PORT=9000" >> .env
echo "MINIO_CONSOLE_PORT=9001" >> .env
echo ""
echo "✅ Created dev .env file"
fi
echo "🔧 Fixing .env permissions"
sudo chown $USER:$USER .env
sudo chmod 600 .env
echo "🌐 Ensure crew-app-net network"
docker network inspect crew-app-net >/dev/null 2>&1 || \
docker network create crew-app-net
echo "📊 Current MinIO Dev status"
if docker ps --format '{{.Names}}' | grep -q crew-minio; then
echo " MinIO Dev is currently running"
docker ps | grep crew-minio
else
echo " MinIO Dev is not running (first deployment)"
fi
echo "🚀 Starting/updating MinIO Dev container"
docker compose pull
docker compose up -d
echo "⏳ Waiting for MinIO to be ready"
sleep 10
if docker ps --format '{{.Names}}' | grep -q crew-minio; then
echo "✅ MinIO Dev container is running"
docker ps | grep crew-minio
echo "🪣 Setting up development buckets"
# Update script to use dev container name
sed -i 's/crew-minio-prod/crew-minio/g' "$DEPLOY_DIR/setup-buckets.sh"
bash "$DEPLOY_DIR/setup-buckets.sh" || echo "⚠️ Bucket setup had some warnings (buckets may already exist)"
else
echo "❌ MinIO Dev container failed to start"
docker compose logs
exit 1
fi
echo "🧹 Cleanup"
rm -f /tmp/docker-compose.yml /tmp/setup-buckets.sh
echo "✅ MinIO Dev deployment completed!"
echo "🌐 MinIO S3 API: http://dev.crewsportswear.com:9000"
echo "🌐 MinIO Console: http://dev.crewsportswear.com:9001"
echo " Username: minioadmin"
echo " Password: minioadmin123"
EOF
# 5⃣ Health check
- name: Health check
shell: sh
env:
DEPLOY_HOST: ${{ secrets.DEPLOY_HOST }}
run: |
echo "⏳ Waiting for MinIO Dev to be ready..."
sleep 10
echo "🔍 Testing MinIO health endpoint..."
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" --max-time 30 http://${DEPLOY_HOST}:9000/minio/health/live || echo "000")
if [ "$HTTP_CODE" = "200" ]; then
echo "✅ MinIO Dev health check passed! (HTTP $HTTP_CODE)"
echo "🎉 Development deployment successful!"
echo ""
echo "📋 Access points:"
echo " - API: http://${DEPLOY_HOST}:9000"
echo " - Console: http://${DEPLOY_HOST}:9001"
else
echo "⚠️ MinIO health check status: HTTP $HTTP_CODE"
echo " MinIO might need more time to initialize."
echo " Check manually: curl http://${DEPLOY_HOST}:9000/minio/health/live"
fi

170
.gitea/workflows/deploy.yml Normal file
View File

@@ -0,0 +1,170 @@
name: Deploy MinIO Production
on:
push:
branches:
- main
- master
workflow_dispatch:
jobs:
deploy:
runs-on: ubuntu-latest
container:
image: catthehacker/ubuntu:act-latest
steps:
# 1⃣ Checkout code
- name: Checkout code
shell: sh
run: |
git clone $GITHUB_SERVER_URL/$GITHUB_REPOSITORY.git /workspace/repo
cd /workspace/repo
git checkout $GITHUB_REF_NAME
# 2⃣ Setup SSH
- name: Setup SSH
shell: sh
env:
DEPLOY_SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
DEPLOY_HOST: ${{ secrets.DEPLOY_HOST }}
run: |
mkdir -p ~/.ssh
chmod 700 ~/.ssh
echo "$DEPLOY_SSH_KEY" > ~/.ssh/id_ed25519
chmod 600 ~/.ssh/id_ed25519
ssh-keyscan -H "$DEPLOY_HOST" >> ~/.ssh/known_hosts
# 3⃣ Upload compose files
- name: Upload compose and scripts
shell: sh
env:
DEPLOY_USER: ${{ secrets.DEPLOY_USER }}
DEPLOY_HOST: ${{ secrets.DEPLOY_HOST }}
run: |
scp -i ~/.ssh/id_ed25519 \
/workspace/repo/docker-compose.prod.yml \
/workspace/repo/setup-buckets.sh \
${DEPLOY_USER}@${DEPLOY_HOST}:/tmp/
# 4⃣ Deploy on server
- name: Deploy MinIO on server
shell: sh
env:
DEPLOY_USER: ${{ secrets.DEPLOY_USER }}
DEPLOY_HOST: ${{ secrets.DEPLOY_HOST }}
run: |
ssh -i ~/.ssh/id_ed25519 $DEPLOY_USER@$DEPLOY_HOST << 'EOF'
set -e
DEPLOY_DIR="/var/www/apps/minio-storage"
sudo mkdir -p "$DEPLOY_DIR"
sudo chown $USER:$USER "$DEPLOY_DIR"
echo "📄 Updating compose file and scripts"
cp /tmp/docker-compose.prod.yml "$DEPLOY_DIR/docker-compose.yml"
cp /tmp/setup-buckets.sh "$DEPLOY_DIR/"
chmod +x "$DEPLOY_DIR/setup-buckets.sh"
cd "$DEPLOY_DIR"
echo "🔍 Checking .env file"
if [ ! -f .env ]; then
echo "⚠️ .env file not found, creating from example"
echo "# MinIO Production Configuration" > .env
echo "MINIO_ROOT_USER=admin_$(date +%s)" >> .env
echo "MINIO_ROOT_PASSWORD=$(openssl rand -base64 32 | tr -d '/+=' | cut -c1-32)" >> .env
echo "MINIO_SERVER_URL=https://minio.crewsportswear.com" >> .env
echo "MINIO_BROWSER_REDIRECT_URL=https://console.crewsportswear.com" >> .env
echo "" >> .env
echo "# BasicAuth for console (generate with: htpasswd -nb admin password)" >> .env
echo "# TRAEFIK_CONSOLE_AUTH='admin:\$\$apr1\$\$...'" >> .env
echo ""
echo "⚠️ IMPORTANT: Update .env with proper credentials!"
echo " Generated random password in $DEPLOY_DIR/.env"
echo " Save these credentials securely!"
fi
echo "🔧 Fixing .env permissions"
sudo chown $USER:$USER .env
sudo chmod 600 .env
echo "🌐 Ensure networks"
docker network inspect traefik-public >/dev/null 2>&1 || \
docker network create traefik-public
docker network inspect crew-app-net >/dev/null 2>&1 || \
docker network create crew-app-net
echo "📊 Current MinIO status"
if docker ps --format '{{.Names}}' | grep -q crew-minio; then
echo " MinIO is currently running"
docker ps | grep crew-minio
else
echo " MinIO is not running (first deployment)"
fi
echo "🚀 Starting/updating MinIO container"
docker compose pull
docker compose up -d
echo "⏳ Waiting for MinIO to be ready"
sleep 10
if docker ps --format '{{.Names}}' | grep -q crew-minio; then
echo "✅ MinIO container is running"
docker ps | grep crew-minio
echo "🪣 Setting up buckets"
# Run bucket setup script
bash "$DEPLOY_DIR/setup-buckets.sh" || echo "⚠️ Bucket setup had some warnings (buckets may already exist)"
else
echo "❌ MinIO container failed to start"
docker compose logs
exit 1
fi
echo "🧹 Cleanup"
rm -f /tmp/docker-compose.prod.yml /tmp/setup-buckets.sh
echo "✅ MinIO production deployment completed!"
echo "🌐 MinIO S3 API: https://minio.crewsportswear.com"
echo "🌐 MinIO Console: https://console.crewsportswear.com"
echo ""
echo "📝 Credentials stored in: $DEPLOY_DIR/.env"
EOF
# 5⃣ Health check
- name: Health check
shell: sh
run: |
echo "⏳ Waiting for MinIO to be ready..."
sleep 15
echo "🔍 Testing MinIO health endpoint..."
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" --max-time 30 https://minio.crewsportswear.com/minio/health/live || echo "000")
if [ "$HTTP_CODE" = "200" ]; then
echo "✅ MinIO health check passed! (HTTP $HTTP_CODE)"
echo "🎉 Production deployment successful!"
echo ""
echo "📋 Next steps:"
echo " 1. Access console: https://console.crewsportswear.com"
echo " 2. Configure app .env files with MinIO credentials"
echo " 3. Migrate images from old server"
else
echo "⚠️ MinIO health check status: HTTP $HTTP_CODE"
echo ""
echo "💡 If this is first deployment, MinIO might need more time to initialize."
echo " Check manually: curl https://minio.crewsportswear.com/minio/health/live"
echo ""
echo "🔍 Troubleshooting:"
echo " 1. Check if container is running:"
echo " docker ps | grep crew-minio"
echo ""
echo " 2. Check MinIO logs:"
echo " docker logs crew-minio-prod"
echo ""
echo " 3. Check Traefik routing:"
echo " docker logs traefik"
fi