Admin

INFRA Cluster Management SOP - Create, Destroy, Scale-out, Scale-in, and Monitor

Here are some admin tasks related to the INFRA module (because who doesn’t love infrastructure management?):


Install INFRA

Use the infra.yml playbook to install the INFRA module on Infra nodes:

./infra.yml     # Install INFRA module on infra group

Remove INFRA

Use the infra-rm.yml playbook to uninstall the INFRA module from Infra nodes:

./infra-rm.yml  # Uninstall INFRA module from infra group

Expand INFRA

To scale out an existing INFRA deployment, first modify the infra group by adding new node IPs and assigning unique infra_seq numbers:

all:
  children:
    infra:
      hosts:
        10.10.10.10: { infra_seq: 1 } # Existing node #1
        10.10.10.11: { infra_seq: 2 } # New node #2 (fresh meat!)

Then use the infra.yml playbook to install INFRA on the new node:

./infra.yml -l 10.10.10.11    # Install INFRA on the new node

Local Repo

Use these playbook tasks to manage local package repositories (YUM/APT) on Infra nodes (because who doesn’t love a good package party?):

./infra.yml -t repo              # Create local repo from internet or offline packages

./infra.yml -t repo_dir          # Create local repo directory
./infra.yml -t repo_check        # Check if local repo exists
./infra.yml -t repo_prepare      # Use existing local repo if available
./infra.yml -t repo_build        # Build local repo from upstream if not exists
./infra.yml     -t repo_upstream     # Add upstream repo/list files
./infra.yml     -t repo_remove       # Remove existing repo files if repo_remove=true
./infra.yml     -t repo_add          # Add upstream repo files to /etc/yum.repos.d (or apt)
./infra.yml     -t repo_url_pkg      # Download packages defined in repo_url_packages
./infra.yml     -t repo_cache        # Create metadata cache with yum makecache / apt update
./infra.yml     -t repo_boot_pkg     # Install bootstrap packages (createrepo_c, yum-utils, etc)
./infra.yml     -t repo_pkg          # Download packages & deps from upstream
./infra.yml     -t repo_create       # Create local repo with createrepo_c / dpkg-dev
./infra.yml     -t repo_use          # Add new repo to /etc/yum.repos.d | apt sources
./infra.yml -t repo_nginx        # Start nginx as file server if not running

Commonly used commands (the greatest hits):

./infra.yml     -t repo_upstream     # Add upstream repos defined in repo_upstream
./infra.yml     -t repo_pkg          # Download packages and their dependencies
./infra.yml     -t repo_create       # Create/update local yum/apt repo

Managing Nginx

./infra.yml -t nginx                       # Reset Nginx component
./infra.yml -t nginx_index                 # Re-render Nginx homepage
./infra.yml -t nginx_config,nginx_reload   # Re-render config & expose new upstream services

If users specify a cert name in the certbot field of infra_portal, you can get free HTTPS certs with certbot:

# Get free HTTPS certs for real domains with certbot (SSL all the things!)
./infra.yml -t nginx_certbot,nginx_reload -e certbot_sign=true

Manage Infra Components

Use these playbook tasks to manage infrastructure components on Infra nodes (it’s like conducting an orchestra, but with servers):

./infra.yml -t infra           # Configure infrastructure
./infra.yml -t infra_env       # Configure env vars: env_dir, env_pg, env_pgadmin, env_var
./infra.yml -t infra_pkg       # Install required packages: infra_pkg_yum, infra_pkg_pip
./infra.yml -t infra_user      # Setup infra OS user group
./infra.yml -t infra_cert      # Issue certs for infra components
./infra.yml -t dns             # Configure DNSMasq: dns_config, dns_record, dns_launch
./infra.yml -t nginx           # Configure Nginx: nginx_config, nginx_cert, nginx_static, nginx_launch, nginx_exporter
./infra.yml -t prometheus      # Configure Prometheus: prometheus_clean, prometheus_dir, prometheus_config, prometheus_launch, prometheus_reload
./infra.yml -t alertmanager    # Configure AlertManager: alertmanager_config, alertmanager_launch
./infra.yml -t pushgateway     # Configure PushGateway: pushgateway_config, pushgateway_launch
./infra.yml -t blackbox        # Configure Blackbox Exporter: blackbox_launch
./infra.yml -t grafana         # Configure Grafana: grafana_clean, grafana_config, grafana_plugin, grafana_launch, grafana_provision
./infra.yml -t loki            # Configure Loki: loki_clean, loki_dir, loki_config, loki_launch
./infra.yml -t infra_register  # Register infra components to prometheus

Other useful tasks (for when you need that extra infrastructure TLC):

./infra.yml -t nginx_index                        # Re-render Nginx homepage
./infra.yml -t nginx_config,nginx_reload          # Re-render config & expose new upstream services
./infra.yml -t prometheus_conf,prometheus_reload   # Re-generate Prometheus config & reload
./infra.yml -t prometheus_rule,prometheus_reload   # Re-copy rules & alerts, then reload
./infra.yml -t grafana_plugin                     # Download Grafana plugins (VPN might be needed)




Last modified 2025-04-09: add vps (58e0f01)