High Availability Proxmox and OPNsense (Partial Redudancy) – Network & Server Simulations Part1

High Availability Proxmox and OPNsense (Partial Redudancy) – Network & Server Simulations Part1

Penjelasan Topology

  • Terdapat 2 Router Firewall OPNsense dan 2 WAN untuk terkoneksi ke internet. OPNsense-1 (Master), OPNsense-2 (Backup) dan WAN1 (Master), WAN2 (Backup).
  • Akses internet (NAT) dari VM & Container yang ada pada cluster Proxmox secara default akan diarahkan ke gateway OPNsense-1. Jika OPNsense-1 bermasalah (down), maka gateway akan berpindah ke OPNsense-2.
  • Akses internet (NAT) secara default akan diarahkan ke WAN1. Jika WAN1 bermasalah (down), maka akan berpindah ke WAN2.
  • Jika secara bersamaan router firewall OPNsense-1 dan WAN1 bermasalah (down), maka gateway akan berpindah ke OPNsense-2 dan akses internet akan menggunakan WAN2.
  • Pada OPNsense-1 dan OPNsense-2 akan diinstall HAproxy untuk load balancing beberapa VM & Container yang ada pada cluster proxmox.
  • Akses Port Forwarding (DNAT) untuk VM & Container akan diarahkan ke Virtual IP HAProxy (WAN1 & WAN2).
  • Pada DNS server akan dibuatkan domain yang pointing ke Virtual IP yang ada di WAN1 (HAProxy) dan WAN2 (HAProxy). Jika IP address WAN1 bermasalah (down), maka akses ke domain akan diarahkan ke Virtual IP WAN2 (HAProxy).
  • Jika salah satu cable network yang terhubung ke L3-SW1 & L3-SW2 bermaslah, maka koneksi network di cluster Proxmox masih bisa terhubung.

Pre-Requirement

  • Laptop/Komputer: CPU 4Core/8Thread, 32GB RAM, 250GB SSD
  • Enable Intel-VT/AMD-V/SVM Bios setting
  • VMware Workstation
  • GNS3
  • Enable Hardware Acceleration KVM/HAXM GNS3 QEMU setting
  • GNS3-VM: 8 CPU dan 20 GB RAM

Spesifikasi VM

VMOSCPURAMDISKNIC
OPNsense-1OPNsense 24.12 Core1.5 GB3 GB6
OPNsense-2OPNsense 24.12 Core1.5 GB3 GB6
PVE1Proxmox VE 8.14 Core4 GB30 GB + 50 GB6
PVE2Proxmox VE 8.14 Core4 GB30 GB + 50 GB6
PBSProxmox Backup 3.14 Core4 GB30 GB + 50 GB3
NFSTrueNAS Core 134 Core4 GB30 GB + 50 GB3
PC-RemoteWindows 72 Core1 GB30 GB1
PC-ClientWindows 72 Core1 GB30 GB1
L3-SW1MikroTik CHR 7.11 P11 Core256 MB100 MB10
L3-SW2MikroTik CHR 7.11 P11 Core256 MB100 MB10
WAN1MikroTik CHR 7.11 P11 Core256 MB100 MB2
WAN2MikroTik CHR 7.11 P11 Core256 MB100 MB2
INTERNETMikroTik CHR 7.11 P11 Core256 MB100 MB4
DNSMikroTik CHR 7.11 P11 Core256 MB100 MB1
Router-ClientMikroTik CHR 7.11 P11 Core256 MB100 MB2

Interface & IP Address

  • OPNsense-1
InterfaceIdentifierPortVLAN IDIP AddressGatewayProtocol
LANlanPort110.10.10.11
WANwanPort2210.1.1.11210.1.1.1
lagg0opt5Port3+Port4roundrobin
VLAN10opt1lagg010192.168.10.11
VLAN20opt2lagg020192.168.20.11
pfSyncopt3Port4172.1.1.11
WAN2opt4Port5220.1.1.11220.1.1.1
WANVirtual IP210.1.1.13
WAN2Virtual IP220.1.1.13
LANVirtual IP10.10.10.1
  • OPNsense-2
InterfaceIdentifierPortVLAN IDIP AddressGatewayProtocol
LANlanPort110.10.10.12
WANwanPort2210.1.1.12210.1.1.1
lagg0opt5ort3+Port4roundrobin
VLAN10opt1lagg010192.168.10.12
VLAN20opt2lagg020192.168.20.12
pfSyncopt3Port4172.1.1.12
WAN2opt4Port5220.1.1.12220.1.1.1
WANVirtual IP210.1.1.13
WAN2Virtual IP220.1.1.13
LANVirtual IP10.10.10.1
  • PVE1
InterfaceTypePortVLAN IDIP AddressGatewayBond ModeVLAN aware
vmbr0Linux BridgePort110.10.10.10010.10.10.1
bond0Linux BondPort2+Port3balance-rr
bond1Linux BondPort4+Port5192.168.100.100balance-rr
ensXNetwork DevicePort6172.1.1.100
vmbr1Linux Bridgebond0check
vmbr1.10Linux VLANvmbr110192.168.10.100
vmbr1.20Linux VLANvmbr120192.168.20.100
  • PVE2
InterfaceTypePortVLAN IDIP AddressGatewayBond ModeVLAN aware
vmbr0Linux BridgePort110.10.10.20010.10.10.1
bond0Linux BondPort2+Port3balance-rr
bond1Linux BondPort4+Port5192.168.100.200balance-rr
ensXNetwork DevicePort6172.1.1.200
vmbr1Linux Bridgebond0check
vmbr1.10Linux VLANvmbr110192.168.10.200
vmbr1.20Linux VLANvmbr120192.168.20.200
  • PBS
InterfaceTypePortIP AddressGatewayBond Mode
ensXNetwork DevicePort110.10.10.15010.10.10.1
bond0Linux BondPort2+Port3192.168.100.150balance-rr
  • NFS
InterfaceTypePortIP AddressGatewayProtocol
em0PhysicalPort110.10.10.25010.10.10.1
lagg0Link AggregationPort2+Port3192.168.100.250roundrobin
  • L3-SW1
InterfaceTypeMTUModePortIP Address
bonding1Bonding3000Balance rrether1+ether2
bonding2Bonding3000Balance rrether3+ether4
bonding3Bonding3000Balance rrether5+ether6
bonding4Bonding3000Balance rrether7+ether8
bridge1Bridge3000bonding1+bonding2+bonding3+bonding4
ether10Ethernet1500ether1010.10.10.101
  • L3-SW2
InterfaceTypeMTUModePortIP Address
bonding1Bonding3000Balance rrether1+ether2
bonding2Bonding3000Balance rrether3+ether4
bonding3Bonding3000Balance rrether5+ether6
bonding4Bonding3000Balance rrether7+ether8
bridge1Bridge3000bonding1+bonding2+bonding3+bonding4
ether10Ethernet1500ether1010.10.10.102
  • WAN1
InterfacePortIP AddressGatewayDst. Address
ether1ether1110.1.1.2110.1.1.10.0.0.0/0
ether2ether2210.1.1.1
  • WAN2
InterfacePortIP AddressGatewayDst. Address
ether1ether1120.1.1.2120.1.1.10.0.0.0/0
ether2ether2220.1.1.1
  • INTERNET
InterfacePortIP AddressGatewayDst. Address
ether1ether1110.1.1.1110.1.1.2210.1.1.0/24
ether2ether2120.1.1.1120.1.1.2220.1.1.0/24
ether3ether3130.1.1.1
ether4ether48.8.8.1
  • DNS
InterfacePortIP AddressGatewayDst. Address
ether1ether18.8.8.88.8.8.10.0.0.0/0
  • Router-Client
InterfacePortIP AddressGatewayDst. Address
ether1ether1130.1.1.2130.1.1.10.0.0.0/0
ether2ether2192.168.1.1
  • PC-Remote
InterfaceIP AddressGateway
Network10.10.10.5010.10.10.1
  • PC-Client
InterfaceIP AddressGateway
Network192.168.1.100192.168.1.1

OPNsense1


  • Setting IP Address

Tambahkan Interface Link Aggregation: Interface > Other Type > LAGG
Tambahkan Interface VLAN: Interface > Other Type > VLAN
Aktifkan Interface: Interface > Assignments

InterfaceIdentifierPortVLAN IDIP AddressGatewayProtocol
LANlanPort110.10.10.11
WANwanPort2210.1.1.11210.1.1.1
lagg0opt5Port3+Port4roundrobin
VLAN10opt1lagg010192.168.10.11
VLAN20opt2lagg020192.168.20.11
pfSyncopt3Port4172.1.1.11
WAN2opt4Port5220.1.1.11220.1.1.1
  • Setting Virtual IP

Interfaces > Virtual IPs > Settings > Add (+) > advanced mode

OptionsValueValueValueValueValue
ModeCARPCARPCARPCARPCARP
InterfaceWANLANVALN10VLAN20WAN2
Address210.1.1.13/2410.10.10.1/24192.168.10.1/24192.168.20.1/24220.1.1.13/24
Passwordopnsenseopnsenseopnsenseopnsenseopnsense
VHID13662
Advbase11111
Advskew00000
DescriptionWAN VIPLAN VIPVLAN10 VIPVLAN20 VIPWAN2 VIP
  • Setting Gateway

System > Gateway > Configuration

OptionsPrimary GatewayBackup Gateway
NameWAN1WAN2
InterfaceWANWAN2
Address FamilyIPv4IPv4
IP Address210.1.1.1220.1.1.1
Upstream GatewayCheckCheck
Monitor IP110.1.1.1210.1.1.1
Priority1100

System > Gateway > Group

OptionsValue
Group NameGateway
Gateway WAN1Tier1
Gateway WAN2Tier2
Trigger LevelPacket Loss
Pool OptionsRound Robin
  • Setting Firewall Rules LAN

Firewall > Rules > LAN

OptionsRule1Rule2Rule3
ActionPassPassPass
InterfaceLANLANLAN
Directionininin
TCP/IP VersionIPv4IPv4IPv4
ProtocolanyCARPany
SourceLAN netanyLAN net
Destinationanyanyany
GatewaydefaultdefaultGateway
  • Setting Firewall Rules pfSync

Firewall > Rules > pfSync

OptionsRule1Rule2
ActionPassPass
InterfacepfSyncpfSync
Directioninin
TCP/IP VersionIPv4IPv4
ProtocolanyCARP
SourcepfSync netany
Destinationanyany
Gatewaydefaultdefault
  • Setting Firewall Rules VLAN10

Firewall > Rules > VLAN10

OptionsRule1Rule2Rule3
ActionPassPassPass
InterfaceVLAN10VLAN10VLAN10
Directionininin
TCP/IP VersionIPv4IPv4IPv4
ProtocolanyCARPany
SourceVLAN10 netanyVLAN10
Destinationanyanyany
GatewaydefaultdefaultGateway
  • Setting Firewall Rules VLAN20

Firewall > Rules > VLAN20

OptionsRule1Rule2Rule3
ActionPassPassPass
InterfaceVLAN20VLAN20VLAN20
Directionininin
TCP/IP VersionIPv4IPv4IPv4
ProtocolanyCARPany
SourceVLAN20 netanyVLAN20
Destinationanyanyany
GatewaydefaultdefaultGateway
  • Setting Firewall Rules WAN

Firewall > Rules > WAN

OptionsRule1Rule2Rule3
ActionPassPassPass
InterfaceWANWANWAN
Directionininin
TCP/IP VersionIPv4IPv4IPv4
ProtocolanyCARPICMP
SourceWAN netanyany
Destinationanyanyany
Gatewaydefaultdefaultdefault
  • Setting Firewall Rules WAN2

Firewall > Rules > WAN2

OptionsRule1Rule2Rule3
ActionPassPassPass
InterfaceWAN2WAN2WAN2
Directionininin
TCP/IP VersionIPv4IPv4IPv4
ProtocolanyCARPICMP
SourceWAN2 netanyany
Destinationanyanyany
Gatewaydefaultdefaultdefault
  • Setting Firewall Advanced

Firewall > Settings > Advanced

OptionsValue
Replection for port forwardsCheck
Replection for 1:1Check
Automatic outbound NAT for ReplectionCheck
  • Setting Firewall Outbound

Firewall > NAT > Outbound > Manual outbound NAT rule generation

OptionsRule1Rule2Rule3Rule4Rule5Rule6
InterfaceWANWAN2WANWAN2WANWAN2
TCP/IPIPv4IPv4IPv4IPv4IPv4IPv4
Protocolanyanyanyanyanyany
Source AddrLAN netLAN netVLAN10 netVLAN10 netVLAN20 netVLAN20 net
Source Portanyanyanyanyanyany
Destination Addranyanyanyanyanyany
Destination Portanyanyanyanyanyany
Translation/TargetWAN AddrWAN2 AddrWAN AddrWAN2 AddrWAN AddrWAN2 Addr
  • Setting High Availability

System > High Availability > Settings

OptionsValue
Synchronize Statescheck
Synchronize InterfacepfSync
Synchronize Peer IP172.1.1.12
Synchronize Config to IP172.1.1.12
Remote System Usernameroot
Remote System Password(masukan password root opnsense2)
Services to synchronizeDashboard, DHCPD, Virtual IPs, Cron, Firewall Rules, Aliases, NAT, Alliases
  • Setting Cron Job

System > Settings > Cron

OptionsValue
Minutes*
Hours*
Day of the Month*
Month*
Day of the Week*
CommandHA update and reconfigure backup
DescriptionHA

OPNsense2


  • Setting IP Address

Tambahkan Interface Link Aggregation: Interface > Other Type > LAGG
Tambahkan Interface VLAN: Interface > Other Type > VLAN
Aktifkan Interface: Interface > Assignments

InterfaceIdentifierPortVLAN IDIP AddressGatewayProtocol
LANlanPort110.10.10.12
WANwanPort2210.1.1.12210.1.1.1
lagg0opt5Port3+Port4roundrobin
VLAN10opt1lagg010192.168.10.12
VLAN20opt2lagg020192.168.20.12
pfSyncopt3Port4172.1.1.12
WAN2opt4Port5220.1.1.12220.1.1.1
  • Setting Gateway

System > Gateway > Configuration

OptionsPrimary GatewayBackup Gateway
NameWAN1WAN2
InterfaceWANWAN2
Address FamilyIPv4IPv4
IP Address210.1.1.1220.1.1.1
Upstream GatewayCheckCheck
Monitor IP110.1.1.1210.1.1.1
Priority1100

System > Gateway > Group

OptionsValue
Group NameGateway
Gateway WAN1Tier1
Gateway WAN2Tier2
Trigger LevelPacket Loss
Pool OptionsRound Robin
  • Setting Firewall Rules pfSync

Firewall > Rules > pfSync

OptionsRule1Rule2
ActionPassPass
InterfacepfSyncpfSync
Directioninin
TCP/IP VersionIPv4IPv4
ProtocolanyCARP
SourcepfSync netany
Destinationanyany
Gatewaydefaultdefault
  • Setting Firewall Advanced

Firewall > Settings > Advanced

OptionsValue
Replection for port forwardsCheck
Replection for 1:1Check
Automatic outbound NAT for ReplectionCheck
  • Setting High Availability

System > High Availability > Settings

OptionsValue
Synchronize Statescheck
Synchronize InterfacepfSync
Synchronize Peer IP172.1.1.11
Synchronize Config to IP172.1.1.11
Remote System Usernameroot
Remote System Password(masukan password root opnsense1)
Services to synchronizeDashboard, DHCPD, Virtual IPs, Cron, Firewall Rules, Aliases, NAT, Alliases

Verifikasi HA

OPNsense1: Lobby > Dashboard > Add Widget > CARP > Save Settings
Reboot OPNsense1 dan OPNsense2
Verifikasi status High Availability: System > High Availability > Status

Verifikasi status CARP: Interfaces > Virtual IPs > Status > Enter Persistent CARP Maintenance Mode

Before

After

Non-aktifkan kembali Persistent maintenance mode pada OPNsense1
Verifikasi status CARP pada OPNsense1 pastikan sudah kembali menjadi Master dan pada OPNsense2 sudah kembali menjadi Backup


PVE1 (Proxmox VE)


  • Setting IP Address

pve1 > System > Network
vmbr0: digunakan untuk akses host proxmox
vmbr1: digunakan untuk akses virtual machine dan container proxmox dengan mode vlan

InterfaceTypePortVLAN IDIP AddressGatewayBond ModeVLAN aware
vmbr0Linux BridgePort110.10.10.10010.10.10.1
bond0Linux BondPort2+Port3 balance-rr
bond1Linux BondPort4+Port5192.168.100.100balance-rr
ensXNetwork DevicePort6172.1.1.100
vmbr1Linux Bridgebond0check
vmbr1.10Linux VLANvmbr110192.168.10.100
vmbr1.20Linux VLANvmbr120192.168.20.100

File konfigurasi /etc/network/interfaces

auto lo
iface lo inet loopback

auto ens5
iface ens5 inet manual

auto ens6
iface ens6 inet manual
        mtu 3000

auto ens7
iface ens7 inet manual
        mtu 3000

auto ens8
iface ens8 inet manual
        mtu 3000

auto ens9
iface ens9 inet manual
        mtu 3000

auto ens10
iface ens10 inet static
        address 172.1.1.100/24

auto bond0
iface bond0 inet manual
        bond-slaves ens6 ens7
        bond-miimon 100
        bond-mode balance-rr
        mtu 3000

auto bond1
iface bond1 inet static
        address 192.168.100.100/24
        bond-slaves ens8 ens9
        bond-miimon 100
        bond-mode balance-rr
        mtu 3000

auto vmbr0
iface vmbr0 inet static
        address 10.10.10.100/24
        gateway 10.10.10.101
        bridge-ports ens5
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 3000

auto vmbr1.10
iface vmbr1.10 inet static
        address 192.168.10.100/24
        mtu 2000

auto vmbr1.20
iface vmbr1.20 inet static
        address 192.168.20.100/24
        mtu 2000

source /etc/network/interfaces.d/*
  • Setting File Hosts

pve1 > System > Hosts

10.10.10.100 pve1.local pve1
192.168.100.100 pve1.local pve1
172.1.1.100 pve1.local pve1
10.10.10.200 pve2.local pve2
192.168.100.200 pve2.local pve2
172.1.1.200 pve2.local pve2
  • Setting Cluster

Datacenter > pve1 > Cluster > Create Cluster
Cluster Name: sysopsid > Cluster Network: Link 0 (172.1.1.100), Link 1 (192.168.100.100), Link 2 (10.10.10.100)

  • Setting Ceph Storage

pve1 > Ceph > Install Ceph
Public network: 172.1.1.100/24 , Cluster network: 192.168.100.100/24


PVE2 (Proxmox VE)


  • Setting IP Address

pve2 > System > Network
vmbr0: digunakan untuk akses host proxmox
vmbr1: digunakan untuk akses virtual machine dan container proxmox dengan mode vlan

InterfaceTypePortVLAN IDIP AddressGatewayBond ModeVLAN aware
vmbr0Linux BridgePort110.10.10.20010.10.10.1
bond0Linux BondPort2+Port3 balance-rr
bond1Linux BondPort4+Port5192.168.100.200balance-rr
ensXNetwork DevicePort6172.1.1.200
vmbr1Linux Bridgebond0check
vmbr1.10Linux VLANvmbr110192.168.10.200
vmbr1.20Linux VLANvmbr120192.168.20.200

File konfigurasi /etc/network/interfaces

auto lo
iface lo inet loopback

auto ens5
iface ens5 inet manual

auto ens6
iface ens6 inet manual
        mtu 3000

auto ens7
iface ens7 inet manual
        mtu 3000

auto ens8
iface ens8 inet manual
        mtu 3000

auto ens9
iface ens9 inet manual
        mtu 3000

auto ens10
iface ens10 inet static
        address 172.1.1.200/24

auto bond0
iface bond0 inet manual
        bond-slaves ens6 ens7
        bond-miimon 100
        bond-mode balance-rr
        mtu 3000

auto bond1
iface bond1 inet static
        address 192.168.100.200/24
        bond-slaves ens8 ens9
        bond-miimon 100
        bond-mode balance-rr
        mtu 3000

auto vmbr0
iface vmbr0 inet static
        address 10.10.10.200/24
        gateway 10.10.10.101
        bridge-ports ens5
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 3000

auto vmbr1.10
iface vmbr1.10 inet static
        address 192.168.10.200/24
        mtu 2000

auto vmbr1.20
iface vmbr1.20 inet static
        address 192.168.20.200/24
        mtu 2000

source /etc/network/interfaces.d/*
  • Setting File Hosts

pve2 > System > Hosts

10.10.10.100 pve1.local pve1
192.168.100.100 pve1.local pve1
172.1.1.100 pve1.local pve1
10.10.10.200 pve2.local pve2
192.168.100.200 pve2.local pve2
172.1.1.200 pve2.local pve2
  • Setting Cluster

Datacenter > pve2 > Cluster > Join Cluster
Copy Join Information dari pve1 > masukan root password pve1 > Cluster Network: Link 0 (172.1.1.200), Link 1 (192.168.100.200), Link 2 (10.10.10.200)

  • Setting Ceph Storage

pve2 > Ceph > Install Ceph
Public network: 172.1.1.100/24 , Cluster network: 192.168.100.100/24


PBS (Proxmox Backup Server)


  • Setting IP Address

Configuration > Network Interfaces

InterfaceTypePortIP AddressGatewayBond Mode
ensXNetwork DevicePort110.10.10.15010.10.10.1
bond0Linux BondPort2+Port3 balance-rr

File konfigurasi /etc/network/interfaces

auto lo
iface lo inet loopback

auto ens5
iface ens5 inet static
        address 10.10.10.150/24
        gateway 10.10.10.1

auto ens6
iface ens6 inet manual
        mtu 3000

auto ens7
iface ens7 inet manual
        mtu 3000

auto bond0
iface bond0 inet static
        address 192.168.100.150/24
        bond-mode balance-rr
        bond-slaves ens6 ens7
        mtu 3000

source /etc/network/interfaces.d/*
  • Setting Datastore ZFS

Administration > Storage / Disks > ZFS > Create: ZFS > Name: backup


NFS (TrueNAS)


  • Setting IP Address

Network > Interfaces

InterfaceTypePortIP AddressGatewayProtocol
em0PhysicalPort110.10.10.25010.10.10.1
lagg0Link AggregationPort2+Port3192.168.100.250roundrobin
  • Setting NFS Storage

Storage > Pool
Sharing > Unix Shares (NFS)


L3-SW1 (MikroTik CHR)


  • Setting Interface Bonding dan Bridge
InterfaceTypeMTUModePortIP Address
bonding1Bonding3000Balance rrether1+ether2
bonding2Bonding3000Balance rrether3+ether4
bonding3Bonding3000Balance rrether5+ether6
bonding4Bonding3000Balance rrether7+ether8
bridge1Bridge3000bonding1+bonding2+bonding3+bonding4
ether10Ethernet1500ether1010.10.10.101

L3-SW2 (MikroTik CHR)


  • Setting Interface Bonding dan Bridge
InterfaceTypeMTUModePortIP Address
bonding1Bonding3000Balance rrether1+ether2
bonding2Bonding3000Balance rrether3+ether4
bonding3Bonding3000Balance rrether5+ether6
bonding4Bonding3000Balance rrether7+ether8
bridge1Bridge3000bonding1+bonding2+bonding3+bonding4
ether10Ethernet1500ether1010.10.10.102

WAN1 (MikroTik CHR)


  • Setting IP Address dan Gateway
InterfacePortIP AddressGatewayDst. Address
ether1ether1110.1.1.2110.1.1.10.0.0.0/0
ether2ether2210.1.1.1

WAN2 (MikroTik CHR)


  • Setting IP Address dan Gateway
InterfacePortIP AddressGatewayDst. Address
ether1ether1120.1.1.2120.1.1.10.0.0.0/0
ether2ether2220.1.1.1

INTERNET (MikroTik CHR)


  • Setting IP Address dan Gateway (Static Routing)
InterfacePortIP AddressGatewayDst. Address
ether1ether1110.1.1.1110.1.1.2210.1.1.0/24
ether2ether2120.1.1.1120.1.1.2220.1.1.0/24
ether3ether3130.1.1.1
ether4ether48.8.8.1

DNS (MikroTik CHR)


  • Setting IP Address, Gateway dan DNS Static
InterfacePortIP AddressGatewayDst. Address
ether1ether18.8.8.88.8.8.10.0.0.0/0
DomainTypeAddress (Value)
web01.sys-ops.idA210.1.1.101
web01.sys-ops.idA220.1.1.101
web02.sys-ops.idA210.1.1.102
web02.sys-ops.idA220.1.1.102
web.sys-ops.idA210.1.1.103
web.sys-ops.idA220.1.1.103

Router-Client (MikroTik CHR)


  • Setting IP Address dan Gateway
InterfacePortIP AddressGatewayDst. Address
ether1ether1130.1.1.2130.1.1.10.0.0.0/0
ether2ether2192.168.1.1

PC-Remote (Windows 7)


  • Setting IP Address dan Gateway
InterfaceIP AddressGateway
Network10.10.10.5010.10.10.1

PC-Client (Windows 7)


  • Setting IP Address dan Gateway
InterfaceIP AddressGateway
Network192.168.1.100192.168.1.1

Konfigurasi VM & CT


  • Setting IP Address untuk VM dan CT, gunakan bridge vmbr1 dan vlan tag yang sebelumnya sudah di setting (vlan id 10 dan 20)
  • Testing PING dari VM & CT ke gateway dan dns

Konfigurasi Port Forwarding


  • Setting Virtual IP (OPNsense-1)

Interfaces > Virtual IPs > Settings > Add (+) > advanced mode
Pastikan System High Availability sudah aktif dan berjalan dengan lancar

OptionsValueValueValueValueValueValue
ModeCARPCARPCARPCARPCARPCARP
InterfaceWANWANWANWAN2WAN2WAN2
Address210.1.1.101/32210.1.1.102/32210.1.1.103/32220.1.1.101/32220.1.1.102/32220.1.1.103/32
Passwordopnsenseopnsenseopnsenseopnsenseopnsenseopnsense
VHID111213212223
Advbase111111
Advskew000000
DescriptionVIPVIPVIPVIPVIPVIP
  • Setting NAT One-to-One

Firewall > NAT > One-to-One

OptionsRule1Rule2Rule3Rule4
InterfaceWANWANWAN2WAN2
TypeBINATBINATBINATBINAT
External Network210.1.1.101210.1.1.102220.1.1.101220.1.1.102
Source192.168.10.101192.168.10.102192.168.10.101192.168.10.102
Destinationanyanyanyany
NAT reflectionUse system defaultUse system defaultUse system defaultUse system default
Descriptionweb01 via wan1web02 via wan1web01 via wan2web02 via wan2
  • Setting NAT Port Forward

Firewall > NAT > Port Forward

OptionsRule1Rule2Rule3Rule4
InterfaceWANWANWAN2WAN2
TCP/IPIPv4IPv4IPv4IPv4
ProtocolTCPTCPTCPTCP
Destination210.1.1.101 (VIP)210.1.1.102 (VIP)220.1.1.101 (VIP)220.1.1.102 (VIP)
Destination Port Rangefrom: HTTP to: HTTPfrom: HTTP to: HTTPfrom: HTTP to: HTTPfrom: HTTP to: HTTP
Redirect Target PortHTTPHTTPHTTPHTTP
Pool OptionsDefaultDefaultDefaultDefault
NAT ReflectionUse system defaultUse system defaultUse system defaultUse system default
Descriptionweb01 via wan1web02 via wan1web01 via wan2web02 via wan2
  • Testing akses web01 dan web02 menggunakan Virtual IP dari WAN1 dan WAN2 lewat browser di PC-Client

Konfigurasi HAProxy


  • Install HAproxy plugin pada OPNsense-1 dan OPNsense-2

System > Firmware > Plugins > os-haproxy

  • Aktifkan HAProxy Load Balancer pada High Availability di OPNsense-1

System > High Availabiliy > Settings

  • Setting HAProxy pada OPNsense-1

Services > HAProxy > Settings > Real Servers

OptionsValueValue
EnableCheckCheck
Name or Prefixweb01web02
Typestaticstatic
FQDN or IP192.168.10.101192.168.10.102
Port8080
Modeactiveactive
  • Aktifkan Health Monitor

Services > HAProxy > Settings > Rules & Checks > Health Monitors

OptionsValue
Namehealth-check
Check TypeTCP
SSL PreferencesNone
Check Intervals3s
Port to Check80
  • Aktifkan Backend Pool

Services > HAProxy > Settings > Virtual Services > Backend Pools

OptionsValue
EnableCheck
Nameweb-check
ModeHTTP (Layer 7)
Balancing AlgorithmLeast Connection
Serversweb01 dan web02
Enable Health CheckingCheck
Health Monitorhealth-check
Log Status ChangesCheck
Enable HTTP/2Check
Advertise Protocol (APLN)HTTP/2 dan HTTP/1.1
Persistance typeNone
Table typeNone
  • Aktifkan Public Service

Services > HAProxy > Settings > Virtual Services > Public Services

OptionsValue
EnableCheck
Namefront-web
Listen Addresses210.1.1.103:80 dan 220.1.1.103:80
TypeHTTP/HTTPS (SSL offloading)
Default Backend Poolweb-back
Enable HTTP/2Check
Advertise Protocols (APLN)HTTP/2 dan HTTP/1.1
Table typeNone
  • Aktifkan Firewall Rules untuk akses port 80 HAProxy

Firewall > Rules > WAN & WAN2

OptionsRule1Rule2
ActionPassPass
InterfaceWANWAN2
Directioninin
TCP/IP VersionIPv4IPv4
ProtocolTCPTCP
Sourceanyany
Destination210.1.1.103220.1.1.103
Destination port rangefrom: HTTP to: HTTPfrom: HTTP to: HTTP
Gatewaydefaultdefault
DescriptionAccess HAProxy wan1Access HAProxy wan2
  • Testing akses web01 dan web02 menggunakan domain web.sys-ops.id (yang sebelumnya sudah di setting pada DNS server) di PC-Client

Pengujian High Availability (Failover)


  • Kondisi normal sebelum dilakukan pengujian

OPNsense-1 status CARP menjadi Master dan OPNsense-2 status CARP menjadi Backup.
Koneksi Internet dari VM & CT melalui gateway OPNsense-1 dan WAN1


  • Pengujian 1 : Powered Off OPNsense-1

OPNsense-2 status CARP menjadi Master
Akses web01 dan web02 menggunakan domain web.sys-ops.id (sudah di pointing ke Virtual IP HAProxy) dari PC-Client
Koneksi Internet dari VM & CT berganti melalui gateway OPNsense-2 dan WAN1
Test PING dan Traceroute ke domain web.sys-ops.id dari PC-Client


  • Pengujian 2 : Powered Off OPNsense-1 dan WAN1

Akses web01 dan web02 menggunakan domain web.sys-ops.id (sudah di pointing ke Virtual IP HAProxy) dari PC-Client.
Koneksi Internet dari VM & CT berganti melalui gateway OPNsense-2 dan WAN2
Test PING dan Traceroute ke domain web.sys-ops.id dari PC-Client


  • Pengujian 3 : Hapus line cable yang terhubung ke L3-SW1 sehingga hanya menggunakan 1 cable network yang terhubung ke OPNsense-1, OPNsense-2, PVE1 dan PVE2

Koneksi dari VM & CT masih berjalan dengan normal begitu pula sebaliknya akses dari PC-Client ke domain web.sys-ops.id masih normal


Pada artikel ini belum bisa full redudancy, dikarenakan switch untuk cluster proxmox (L3-SW1 dan L3-SW2) tidak mempunyai backup switch (masih single switch). Karena pada saat testing failover jika menggunakan 2 siwtch active-backup masih terjadi / network tidak stabil. Untuk kedepannya akan di update untuk versi full reducancy part 2.

Referensi:

herdiana3389

A system administrator with skills in system administration, virtualization, linux, windows, networking, cloud computing, container, etc.