PVE Guest crash detection - Revision history

Tinker: /* Next Steps */

2024-06-22T18:16:46Z

Next Steps

@@ Line 98: / Line 98: @@
 </syntaxhighlight>
 This is a couple minutes of output.  Notice that everything show up twice...
 === Next Steps ===

Tinker: /* The Problem */

2024-06-13T15:02:45Z

The Problem

← Older revision		Revision as of 11:02, 13 June 2024
Line 2:		Line 2:
	Sometimes VMs/LXCs crash & there's no way to have Proxmox detect & reset them...		Sometimes VMs/LXCs crash & there's no way to have Proxmox detect & reset them...

	(Current theory is that the problem that started this project is the journalling service occasionally runs into a disk-write problem when the nitely backups are happening.)		(Current theory is that the problem that started this project is the journalling service occasionally runs into a disk-write problem when the nitely backups are happening. This seems to make systemd very unhappy & it locks the VM up tight...)

	= Possible Solution(s) =		= Possible Solution(s) =

Tinker: /* The Problem */

2024-06-13T15:01:39Z

The Problem

← Older revision		Revision as of 11:01, 13 June 2024
Line 1:		Line 1:
	= The Problem =		= The Problem =
	Sometimes VMs/LXCs crash & there's no way to have Proxmox detect & reset them...		Sometimes VMs/LXCs crash & there's no way to have Proxmox detect & reset them...

			(Current theory is that the problem that started this project is the journalling service occasionally runs into a disk-write problem when the nitely backups are happening.)

	= Possible Solution(s) =		= Possible Solution(s) =
	Have the VM/LXC itself run a watchdog & send a message regularly to Proxmox.		Have the VM/LXC itself run a watchdog & send a message regularly to Proxmox.

Tinker: /* An example of the messages */

2024-05-20T02:54:44Z

An example of the messages

← Older revision		Revision as of 22:54, 19 May 2024
Line 94:		Line 94:
	^C		^C
	</syntaxhighlight>		</syntaxhighlight>
			This is a couple minutes of output. Notice that everything show up twice...

	=== Next Steps ===		=== Next Steps ===

Tinker: /* An example of the messages */

2024-05-20T02:41:40Z

An example of the messages

← Older revision		Revision as of 22:41, 19 May 2024
Line 70:		Line 70:
	VM 1001 1716172680 pfSense310		VM 1001 1716172680 pfSense310
	CT 1050 1716172681 FTPd		CT 1050 1716172681 FTPd
	PVE Schizo310 1716172681 schizo310		PVE Schizo310 1716172681 schizo310
	CT 1022 1716172681 Grafana		CT 1022 1716172681 Grafana
	VM 1201 1716172681 manager310		VM 1201 1716172681 manager310
Line 84:		Line 84:
	CT 11001 1716172741 LibreSpeed310		CT 11001 1716172741 LibreSpeed310
	CT 1050 1716172741 FTPd		CT 1050 1716172741 FTPd
	PVE Schizo310 1716172741 schizo310		PVE Schizo310 1716172741 schizo310
	CT 1022 1716172741 Grafana		CT 1022 1716172741 Grafana
	CT 1010 1716172741 MQTT		CT 1010 1716172741 MQTT

Tinker: /* An example of the messages */

2024-05-20T02:39:40Z

An example of the messages

← Older revision		Revision as of 22:39, 19 May 2024
Line 68:		Line 68:
	<syntaxhighlight>		<syntaxhighlight>
	root@schizo310:~/bin# watcher		root@schizo310:~/bin# watcher
	VM 1001 ~~1716161760~~ pfSense310		VM 1001 1716172680 pfSense310
	CT ~~11004~~ ~~1716161761~~ ~~pubkey-test~~		CT 1050 1716172681 FTPd
	CT ~~11001~~ ~~1716161761~~ ~~LibreSpeed310~~		PVE Schizo310 1716172681 schizo310
	VM 1201 ~~1716161761~~ manager310		CT 1022 1716172681 Grafana
	CT ~~11002~~ ~~1716161761~~ ~~NetCat-Tst-CLI~~		VM 1201 1716172681 manager310
	~~PVE Schizo310 1716161761~~ ~~schizo310~~		CT 1010 1716172681 MQTT
	CT ~~1022~~ ~~1716161761~~ ~~Grafana~~		CT 1021 1716172681 InfluxDB
	VM ~~11003~~ ~~1716161761~~ ~~LMDE~~-6		CT 1020 1716172681 MariaDB
	CT ~~1050~~ ~~1716161761~~ ~~FTPd~~		CT 1030 1716172681 Apache
	CT ~~1010~~ ~~1716161761~~ ~~MQTT~~		CT 1015 1716172681 Node-Red
	VM ~~11005~~ ~~1716161761~~ ~~Mint-21-Tst~~		CT 1040 1716172681 FileStore
	CT 1040 ~~1716161761~~ FileStore		CT 11001 1716172681 LibreSpeed310
	CT ~~10022~~ ~~1716161761~~ ~~BlogNabbit~~		VM 1001 1716172740 pfSense310
	CT ~~1030~~ ~~1716161761~~ ~~Apache~~		CT 1040 1716172741 FileStore
	CT ~~1032~~ ~~1716161761~~ ~~Apache-again~~		CT 11001 1716172741 LibreSpeed310
	CT 1015 ~~1716161761~~ Node-Red		CT 1050 1716172741 FTPd
	CT ~~9060~~ ~~1716161761~~ ~~KittyCam~~		PVE Schizo310 1716172741 schizo310
	CT 1021 ~~1716161761~~ InfluxDB		CT 1022 1716172741 Grafana
	CT ~~1020~~ ~~1716161761~~ ~~MariaDB~~		CT 1010 1716172741 MQTT
			CT 1015 1716172741 Node-Red
			CT 1020 1716172741 MariaDB
			CT 1021 1716172741 InfluxDB
			VM 1201 1716172741 manager310
			CT 1030 1716172741 Apache
			^C
	</syntaxhighlight>		</syntaxhighlight>

Tinker: /* Next Steps */

2024-05-19T23:41:41Z

Next Steps

← Older revision		Revision as of 19:41, 19 May 2024
Line 92:		Line 92:
	<span style="color: rgb(153, 51, 102);" >Now to just figure out how to check the guests times against the servers time...</span>		<span style="color: rgb(153, 51, 102);" >Now to just figure out how to check the guests times against the servers time...</span>

	I suspect I'll be writing some code to ~~watch~~ the incoming data & put it in a file for handling.		I suspect I'll be writing some code to take the incoming data & put it in a file (or maybe multiple files) for handling.

	<span style="color: rgb(153, 51, 102);" >Then, if a guests time differs from the hosts time by more than a minute, it's likely crashed. Reset it.</span>		<span style="color: rgb(153, 51, 102);" >Then, if a guests time differs from the hosts time by more than a minute, it's likely crashed. Reset it.</span>

Tinker: /* On the PVE server */

2024-05-19T23:37:19Z

On the PVE server

← Older revision		Revision as of 19:37, 19 May 2024
Line 47:		Line 47:
	A continuously running service based on:		A continuously running service based on:
	* ncat -kl schizo310 9999		* ncat -kl schizo310 9999

			=== A bit of code so far... ===
			<syntaxhighlight lang="bash">
			#! /bin/bash

			while read line
			do
			IFS=';' read -ra datum <<< "$line"
			echo -e ${datum[0]} "\t" ${datum[1]} " \t" ${datum[3]} "\t" ${datum[2]}

			# echo ${datum[0]} # TYPE
			# echo ${datum[1]} # VMID
			# echo ${datum[2]} # HOSTNAME
			# echo ${datum[3]} # TIME

			done < <(ncat -kl Schizo310.TinkerNet.ca 9999)
			</syntaxhighlight>

	=== An example of the messages ===		=== An example of the messages ===
	<~~pre~~>		<syntaxhighlight>
	root@schizo310:~# ~~ncat~~ -~~kl 192.168.1.2 9999~~		root@schizo310:~/bin# watcher
	VM ~~11003 1715882341~~		VM 1001 1716161760 pfSense310
	PVE Schizo310 ~~1715882341~~		CT 11004 1716161761 pubkey-test
	CT ~~11002 1715882341~~		CT 11001 1716161761 LibreSpeed310
	CT ~~11002 1715882401~~		VM 1201 1716161761 manager310
	VM ~~11003 1715882401~~		CT 11002 1716161761 NetCat-Tst-CLI
	~~PVE Schizo310 1715882401~~		PVE Schizo310 1716161761 schizo310
	</~~pre~~>		CT 1022 1716161761 Grafana
			VM 11003 1716161761 LMDE-6
			CT 1050 1716161761 FTPd
			CT 1010 1716161761 MQTT
			VM 11005 1716161761 Mint-21-Tst
			CT 1040 1716161761 FileStore
			CT 10022 1716161761 BlogNabbit
			CT 1030 1716161761 Apache
			CT 1032 1716161761 Apache-again
			CT 1015 1716161761 Node-Red
			CT 9060 1716161761 KittyCam
			CT 1021 1716161761 InfluxDB
			CT 1020 1716161761 MariaDB
			</syntaxhighlight>

	=== Next Steps ===		=== Next Steps ===

Tinker: /* pfSense VMs */

2024-05-19T22:57:41Z

pfSense VMs

← Older revision		Revision as of 18:57, 19 May 2024
Line 88:		Line 88:
	TYPE=`/usr/local/sbin/dmidecode -s system-version`		TYPE=`/usr/local/sbin/dmidecode -s system-version`

	echo -e $TYPE"\t"$VMID"~~\t\t~~"`/bin/date +%s` \| /usr/local/bin/ncat $PVEHOST 9999		echo -e $TYPE";"$VMID";"`/bin/hostname -s`";"`/bin/date +%s` \| /usr/local/bin/ncat $PVEHOST 9999
	</pre>		</pre>

	= Windows VMs =		= Windows VMs =
	<span style="font-size: 24pt;" >¯\_(ツ)_/¯</span>		<span style="font-size: 24pt;" >¯\_(ツ)_/¯</span>

Tinker: /* Sending the info to the host */

2024-05-19T22:54:41Z

Sending the info to the host

← Older revision		Revision as of 18:54, 19 May 2024
Line 36:		Line 36:
	=== Sending the info to the host ===		=== Sending the info to the host ===
	A shell script (/root/bin/watchdog) with the above variable settings and:		A shell script (/root/bin/watchdog) with the above variable settings and:
	* <code>/usr/bin/echo -e $TYPE"\t"$VMID"~~\t\t~~"`/usr/bin/date +%s` ~~{{!}}~~ /usr/bin/ncat $PVEHOST 9999</code>		* <code>/usr/bin/echo -e $TYPE";"$VMID";"`/usr/bin/hostname -s`";"`/usr/bin/date +%s` \| /usr/bin/ncat $PVEHOST 9999</code>
	a cron job sending something like:		a cron job sending something like:
	* * * * * * /home/tinker/bin/watchdog		* * * * * * /home/tinker/bin/watchdog