Using Ansible to build a high availablity Sabnzbd usenet downloader

I’m limited to about 40MB/s on downloads on my VPC at Digital Ocean, but I run Sabnzbd for downloading large files from usenet. It doesn’t take long to download at all, but out of curiosity I wanted to see if I could parallelize this and download multiple files at the same. I use Sonarr for searching usenet for freely distributable training videos which then sends them to SABnzbd for downloading. Since Sonarr can send multiple files to sabnzbd which get queued up, I figured I can reduce the queue by downloading them at the same time.

Using Ansible and Terraform (devops automation tools), I can spin up VPC on demand, provision them, configure them as sabnzbd download nodes and then destroy the instances when complete.

The instances all run the same sabnzbd config and the instances use haproxy for round-robin distribution. I will probably change this to Consul, but I just wanted something quick so I used a basic haproxy config.

Terraform builds 4 sabnzbd, 1 haproxy, and 1 ELK instance. It configures a VIP which I point Sonarr to. Here’s the terraform config that builds a Sabnzbd server:

resource "digitalocean_droplet" "sab1" {
    image = "centos-7-x64"
    name = "sab1"
    tags = ["sab"]
    region = "nyc1"
    size = "2gb"
    private_networking = true
    ssh_keys = [
      "${var.ssh_fingerprint}"
    ]
connection {
      user = "root"
      type = "ssh"
      private_key = "${file(var.pvt_key)}"
      timeout = "2m"
}
provisioner "remote-exec" {
    inline = [
      "export PATH=$PATH:/usr/bin",
      "sudo yum -y install epel-release"
    ]
  }
}

Here is a terraform play to provision 6 new hosts (1 Elasticsearch, 1 HAproxy and 4 Sabnzbd):

Here is the ansible playbook that configuress the sabnzbd instances:

#!/usr/bin/env ansible-playbook
---
- hosts: all
  gather_facts: yes
  become: yes

  tasks:
    - name: TASK | Create usenet group
      group:
        name: usenet
        state: present

    - name: TASK | Create usenet user
      user:
        name: usenet
        shell: /bin/bash
        groups: usenet
        append: yes
        password: [ removed ]
        update_password: on_create

    - name: TASK | Create /apps directory
      file:
        path: /apps
        state: directory
        owner: usenet
        group: usenet
        mode: 0755

    - name: TASK | Create /apps/data/.sabnzbd directory
      file:
        path: /apps/data/.sabnzbd
        state: directory
        owner: usenet
        group: usenet
        mode: 0755

    - name: TASK | Install EPEL Repo
      action: >
        yum name=epel-release state=present update_cache=yes

    - name: TASK | Install packages
      action: >
        yum name={{ item }} state=present update_cache=yes
      with_items:
        - vim
        - git
        - mlocate
        - par2cmdline
        - p7zip
        - unzip
        - tar
        - gcc
        - python-feedparser
        - python-configobj
        - python-cheetah
        - python-dbus
        - python-devel
        - screen
        - vim
        - htop
        - iftop
        - bind-utils
        - tree
        - jq
        - telnet
        - lsof
        - tcpdump
        - nload

    - name: TASK | Install unrar
      yum:
        name: ftp://rpmfind.net/linux/dag/redhat/el7/en/x86_64/dag/RPMS/unrar-5.0.3-1.el7.rf.x86_64.rpm
        state: present

    - name: TASK | Install yenc
      yum:
        name: https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/22/Everything/x86_64/os/Packages/p/python-yenc-0.4.0-4.fc22.x86_64.rpm
        state: present

    - name: TASK | Clone SAB repo
      git:
        repo: 'https://github.com/sabnzbd/sabnzbd.git'
        dest: /apps/sabnzbd
        version: master
        clone: yes
        update: yes

    - name: TASK | Change owner of SAB to usenet
      file:
        path: /apps
        state: directory
        owner: usenet
        group: usenet
        mode: 0755

    - name: TASK | Install systemd unit file for SAB
      template: src="templates/sabnzbd.service.j2" dest="/etc/systemd/system/sabnzbd.service" mode=0644

    - name: TASK | Copy SAB config
      template: src="templates/sabnzbd_config.ini.j2" dest="/apps/data/.sabnzbd/sabnzbd_config.ini" mode=0644

    - name: TASK | Enable and start SAB service
      systemd:
        name: sabnzbd
        enabled: yes
        state: started

    - name: TASK | Install filebeat
      yum:
        name: https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.6.2-x86_64.rpm
        state: present
        update_cache: yes

    - name: TASK | Copy filebeat config file
      template: src="templates/filebeat.yml.j2" dest="/etc/filebeat/filebeat.yml" mode=0644

    - name: TASK | Enable and start filebeat
      systemd:
        name: filebeat
        enabled: yes
        state: restarted

    - name: TASK | Start updatedb for locate
      shell: "updatedb >/dev/null 2>&1 &"

Technology used: