Difference between revisions of "Proxmox All-in-One"

From Da Nerd Mage Wiki
Jump to navigation Jump to search
Tag: Reverted
 
(23 intermediate revisions by the same user not shown)
Line 1: Line 1:
= Installing PVE =
Start with the port you will want as your WAN port (primary) from pfSense connected to your network
Install PVE
Fix repositories
* Add "pve-no-subscription"
* Disable "pve-enterprise"
* Refresh updates
* Upgrade
== A Note about Clustering ==
<span style="color: rgb(132, 63, 161);">(I have not yet fully verified whether this works or not... Watch for either this line or this entire sub-section to disappear.)</span>
If you plan to add this server to a cluster, '''do it now'''. Once you install the first VM or LXC, Proxmox will not let you join a cluster.
<span style="color: rgb(132, 63, 161);">Bummer: So far, pfSense VM doesn't start because PVE is waiting for quorum in the cluster...&nbsp; :{{!}}</span>
Apparently, PVECM will not allow any VMs to start until it can contact the rest of the cluster and establish Quorate
It is quite possible to get it going from the console by logging in and running:
* <code>pvecm expected 1</code>
Sadly, still need to figure out the whole addressing issue.
= Configure your second Network Port =
Add a second Linux Bridge (This will be named '''vmbr1''')
* (Datacenter / Server ... System / Network)
* No addresses or gateways
* Assign your, as yet unused, second physical Ethernet port to this bridge
** For good measure, put something along the lines of "Local Network" in the comment for this one
** (You could edit vmbr0 & put "The Interwebs" in it's comment too...)
* <span style="color: rgb(132, 63, 161); font-size: 18pt;" >Do NOT Forget to '''Apply Configuration'''...</span>
= pfSense =
= pfSense =
* Install [[pfSense{{!}}pfSense]] ([https://getlabsdone.com/how-to-install-pfsense-on-proxmox-step-by-step/ a link])
Installing [[pfSense{{!}}pfSense]] ([https://getlabsdone.com/how-to-install-pfsense-on-proxmox-step-by-step/ a link])
** Set up a second network bridge internally (& give it a secondary physical NIC if you have one)
 
** Build the pfSense VM
([[PfSense{{!}}More information about setting up pfSense]])
*** 8GB drive, 4 cores, 4096MB RAM
* Download the pfSense DVD ISO to your desktop
*** Add a second network device (on the second bridge... duh...)
* Un-gzip it
* Upload it to the '''ISO Images''' folder on your '''Local''' datastore
* Build the pfSense VM
** ISO image: pfSense-CE-2.7.0-RELEASE-amd64.iso
** Guest OS Type: Other
** 8GB drive, 4 cores, 4096MB RAM
*** (I'd suggest setting Processor Type to '''host''')
** Use the original bridge (vmbr0) as the first network port & the new bridge (vmbr1) as the second port.
*** You'll need to go into Hardware for the VM to add in the second Network Device AFTER creating the VM.
*** For some reason, when you choose "other" as your OS type, PVE defaults the network device model to "Intel E1000".<br>This seems unreliable for pfSense. Select "VirtIO (paravirtualized)" instead.
** Configure the pfSense VM to start at boot.
*** go into Options for the VM...
*** Strongly reccomend setting it to boot FIRST & give a startup delay of at least a couple of minutes.
** <span style="color: rgb(186, 55, 42);">Do a BACKUP</span>
** <span style="color: rgb(186, 55, 42);">Do a BACKUP</span>
** Open the VM console & pretend you're building a normal pfSense router
** Open the VM console & pretend you're building a normal pfSense router<br>
** Once the VM is booted into pfSense...
** '''<span style="color: rgb(186, 55, 42);">Do a BACKUP</span>'''<br>
** '''<span style="color: rgb(186, 55, 42);">Do a BACKUP</span>'''
** Then move on to:


= Management VM =
= Management VM =
* Pick your favourite OS & build a VM <span style="color: rgb(132, 63, 161);">(Or... Ya know... Since ya gave that second network bridge a physical NIC (Ya did, right?)... You could just plug a computer in there.)</span>
* Pick your favourite OS & build a VM <br><span style="color: rgb(132, 63, 161);">(Or... Ya know... Since ya gave that second network bridge a physical NIC (Ya did, right?)... You could just plug a computer in there.)</span>
** Point its network device at the second network bridge
** Point its network device at the second network bridge (vmbr1).
** Sign into '''<nowiki>https://192.168.1.1</nowiki>''' (from the Management VM)
* '''<span style="color: rgb(186, 55, 42);">Do a BACKUP</span>'''
** Go into Services / DHCP Server / LAN
 
*** Under Servers, add in your DNS server(s) address(es)
== Further VMs ==
** Restart networking on the Management VM
'''Note:''' Any further VMs created on this server need to have their network interfaces on '''vmbr1''' or else they'll appear on the WAN port.
** Feed access to this VM through the pfSense firewall
* & continue messing with pfSense
** '''<span style="color: rgb(186, 55, 42);">Do a BACKUP</span>'''


= pfSense Configuration =
= pfSense Configuration =
(assumption: pfSense LAN network is the default of 192.168.1.0/24)
*Sign into '''<nowiki>https://192.168.1.1</nowiki>''' (from the Management VM) (or... [[PfSense#pfSense with_WAN_inside_a_LAN{{!}}If working inside your LAN]])
**In '''Services / DNS Resolver / General Settings''', under '''Host Overrides'''
***set up a DNS entry for PVE. (I like 192.168.1.2)
** In '''Firewall / NAT / Port Forward'''
*** set up port forwarding for the pfSense UI (port 443)
*** set up port forwarding for the PVE UI (port 8006)
*** set up port forwarding for SSH (port 22) to the Management VM (if used...)
** QEMU Guest Agent would be handy too...
*** [https://forum.netgate.com/topic/162083/pfsense-vm-on-proxmox-qemu-agent-installation PfSense VM on ProxMox : Qemu-agent installation]
*'''<span style="color: #ba372a;">Do a BACKUP</span>'''
'''Do note:''' These port forwards may be a security risk when you take the system live. They are here for ease of access while configuring the system.
= Taking it LIVE =
= Taking it LIVE =
Here's where things get a bit fugly...
Up to this point, your server works fine on an internal network. Unfortunately, as far as the world outside the box is concerned, there are 2 machines there. The '''Proxmox''' install AND a '''pfSense''' install. They both show up on the network.
Up to this point, your server works fine on an internal network. Unfortunately, as far as the world outside the box is concerned, there are 2 machines there. The '''Proxmox''' install AND a '''pfSense''' install. They both show up on the network.


Line 32: Line 84:
Let's fix that.
Let's fix that.


# Swap all of the network connections on all existing VMs
* Sign into the physical machine (PVE)
#* Go into the '''Hardware''' tab for each VM...
** Make backups of the 2 files we're about to modify
#* edit any '''Network Device'''(s) they have configured...
*** <code>cp /etc/network/interfaces /etc/network/interfaces.BAK</code>
#* basically, swap them to the opposite '''Bridge''' (vmbr) from what they currently use.
*** <code>cp /etc/hosts /etc/hosts.BAK</code>
# SSH into Proxmox & change vmbr0 back to static
** edit /etc/network/interfaces
#* Give it the IP address your pfSense VM should be assigning it & the IP of the pfSense VM as its gateway.
*** Move the address & gateway from vmbr0 to vmbr1 and change them to those assigned for PVE on the pfSense VM.
#* Example here:
** edit /etc/hosts
auto lo
*** Change the address to that assigned for PVE on the pfSense VM<br>
iface lo inet loopback
Reboot the machine
 
iface eno2 inet manual
Wait at least a couple minutes for pfSense to fully boot.
 
iface eno1 inet manual
<span style="color: rgb(132, 63, 161);">'''At this point, the machine shows up on your network as a single device (The pfSense VM!)'''</span>
 
auto vmbr0
You can now browse to https://'''MachineAddress'''/ to access pfSense or https://'''MachineAddress''':8006/ to access the PVE UI to do further setup of the system.
iface vmbr0 inet static
address 192.168.1.2/24
gateway 192.168.1.1
bridge-ports eno1
bridge-stp off
bridge-fd 0
#Internal Network (+ Eth0)
auto vmbr1
iface vmbr1 inet manual
bridge-ports eno2
bridge-stp off
bridge-fd 0
#InterWebs (Eth1)


#* It'd be a good idea to edit <code>/etc/hosts</code> to match the new address.
(Where '''MachineAddress''' is the address or name assigned to it by your local network...)
#* Example Here:
127.0.0.1 localhost.localdomain localhost
192.168.1.2 pve.tinkernet.ca pve
# The following lines are desirable for IPv6 capable hosts
::1    ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
root@pve:~#


# Switch the physical network cables.
Congratulations!&nbsp; You have built a router out of a server.&nbsp; If you plug the first Ethernet port into the Internet and connect a switch to the second Ethernet port, this machine can replace the router provided by your ISP...
#* If NIC #1 is currently plugged into your network, swap it for NIC #2 (or whichever NIC you set your second bridge up to use...)
# Reboot the machine.

Latest revision as of 15:26, 24 November 2023

Installing PVE

Start with the port you will want as your WAN port (primary) from pfSense connected to your network

Install PVE

Fix repositories

  • Add "pve-no-subscription"
  • Disable "pve-enterprise"
  • Refresh updates
  • Upgrade

A Note about Clustering

(I have not yet fully verified whether this works or not... Watch for either this line or this entire sub-section to disappear.)

If you plan to add this server to a cluster, do it now. Once you install the first VM or LXC, Proxmox will not let you join a cluster.

Bummer: So far, pfSense VM doesn't start because PVE is waiting for quorum in the cluster...  :|

Apparently, PVECM will not allow any VMs to start until it can contact the rest of the cluster and establish Quorate

It is quite possible to get it going from the console by logging in and running:

  • pvecm expected 1

Sadly, still need to figure out the whole addressing issue.

Configure your second Network Port

Add a second Linux Bridge (This will be named vmbr1)

  • (Datacenter / Server ... System / Network)
  • No addresses or gateways
  • Assign your, as yet unused, second physical Ethernet port to this bridge
    • For good measure, put something along the lines of "Local Network" in the comment for this one
    • (You could edit vmbr0 & put "The Interwebs" in it's comment too...)
  • Do NOT Forget to Apply Configuration...

pfSense

Installing pfSense (a link)

(More information about setting up pfSense)

  • Download the pfSense DVD ISO to your desktop
  • Un-gzip it
  • Upload it to the ISO Images folder on your Local datastore
  • Build the pfSense VM
    • ISO image: pfSense-CE-2.7.0-RELEASE-amd64.iso
    • Guest OS Type: Other
    • 8GB drive, 4 cores, 4096MB RAM
      • (I'd suggest setting Processor Type to host)
    • Use the original bridge (vmbr0) as the first network port & the new bridge (vmbr1) as the second port.
      • You'll need to go into Hardware for the VM to add in the second Network Device AFTER creating the VM.
      • For some reason, when you choose "other" as your OS type, PVE defaults the network device model to "Intel E1000".
        This seems unreliable for pfSense. Select "VirtIO (paravirtualized)" instead.
    • Configure the pfSense VM to start at boot.
      • go into Options for the VM...
      • Strongly reccomend setting it to boot FIRST & give a startup delay of at least a couple of minutes.
    • Do a BACKUP
    • Open the VM console & pretend you're building a normal pfSense router
    • Do a BACKUP

Management VM

  • Pick your favourite OS & build a VM
    (Or... Ya know... Since ya gave that second network bridge a physical NIC (Ya did, right?)... You could just plug a computer in there.)
    • Point its network device at the second network bridge (vmbr1).
  • Do a BACKUP

Further VMs

Note: Any further VMs created on this server need to have their network interfaces on vmbr1 or else they'll appear on the WAN port.

pfSense Configuration

(assumption: pfSense LAN network is the default of 192.168.1.0/24)

  • Sign into https://192.168.1.1 (from the Management VM) (or... If working inside your LAN)
    • In Services / DNS Resolver / General Settings, under Host Overrides
      • set up a DNS entry for PVE. (I like 192.168.1.2)
    • In Firewall / NAT / Port Forward
      • set up port forwarding for the pfSense UI (port 443)
      • set up port forwarding for the PVE UI (port 8006)
      • set up port forwarding for SSH (port 22) to the Management VM (if used...)
    • QEMU Guest Agent would be handy too...
  • Do a BACKUP

Do note: These port forwards may be a security risk when you take the system live. They are here for ease of access while configuring the system.

Taking it LIVE

Up to this point, your server works fine on an internal network. Unfortunately, as far as the world outside the box is concerned, there are 2 machines there. The Proxmox install AND a pfSense install. They both show up on the network.

So...

Let's fix that.

  • Sign into the physical machine (PVE)
    • Make backups of the 2 files we're about to modify
      • cp /etc/network/interfaces /etc/network/interfaces.BAK
      • cp /etc/hosts /etc/hosts.BAK
    • edit /etc/network/interfaces
      • Move the address & gateway from vmbr0 to vmbr1 and change them to those assigned for PVE on the pfSense VM.
    • edit /etc/hosts
      • Change the address to that assigned for PVE on the pfSense VM

Reboot the machine

Wait at least a couple minutes for pfSense to fully boot.

At this point, the machine shows up on your network as a single device (The pfSense VM!)

You can now browse to https://MachineAddress/ to access pfSense or https://MachineAddress:8006/ to access the PVE UI to do further setup of the system.

(Where MachineAddress is the address or name assigned to it by your local network...)

Congratulations!  You have built a router out of a server.  If you plug the first Ethernet port into the Internet and connect a switch to the second Ethernet port, this machine can replace the router provided by your ISP...

Retrieved from ‘https://wiki.nerdmage.ca/index.php?title=Proxmox_All-in-One&oldid=1909’