Da Nerd Mage Wiki - User contributions [en-gb]

PVE LXC Containers

2026-06-11T03:58:44Z

Tinker: /* Using a VPN (OpenVPN or TailScale) on an LXC */

[[File:Successful-Containers.png{{!}}300px{{!}}right{{!}}thumb]]

So far, LXC containers are just like regular Linux (for the most part...) & the same procedures apply when building.

In fact, I have created a large set of containers on my testbed server simply by following the [[Server Building{{!}}Server Building]] outlines here on this site.

''Heck... These days, I tend to default to Debian LXC containers unless there's a VERY good reason to do otherwise...''
= Building a Container =
== Templates ==
LXC Containers start with a template.

For our example here, we're going to start with a basic Debian 11 template.

This means we need to ensure we have the template on the PVE host.

* Sign into the PVE UI & select your '''Local''' datastore
* Select the '''CT Templates''' storage
* Click the '''Templates''' button
* Select a template package (we're going with '''debian-11-standard''') then hit the '''Download''' button

== Creating the Container ==
=== Web UI method ===
Now that you have a template to start from, you can hit the '''Create CT''' button.

For now, we'll create a very basic container. I've found that most of the defaults are fine for single service applications. The only resource I've found a need to increase so far is CPU cores.  Memory can be increased later if needed (as can CPU cores) and actual data storage should be handled outside of the boot disk anyhow.

* 1st screen: choose an '''ID#''' and a '''hostname''' for the container. Then enter the password for root on this container (twice...). Then hit '''Next'''
* 2nd screen: select the template we downloaded above. Then hit '''Next'''
* 3rd screen: choose a storage location & boot disk size. Then hit '''Next'''
* 4th screen: how many CPU cores you want available to the container. Then hit '''Next'''
* 5th screen: how much memory you want available to the container. Then hit '''Next'''
* 6th screen: set up networking (you'll note it defaults to static addressing...  silly...) Then hit '''Next'''
* 7th screen: set up DNS (Why in heck this isn't considered part of networking...) Then hit '''Next'''
* 8th (final) screen: Look things over to make sure they're the way you want them.  Possibly check the '''Start after created''' box. Then hit '''Finish'''

=== CLI method ===
* <code>pct create '''VMID''' '''TEMPLATE''' --hostname '''HOSTNAME''' --password '''PASSWORD''' --rootfs '''LOCATION''':'''SIZE''' --cores '''CORES''' --memory '''RAM''' --swap '''SWAP''' --unprivileged '''<1{{!}}0>''' --features '''nesting=<1{{!}}0>''' --net0 name='''eth0''',bridge='''BRIDGE''',firewall='''<1{{!}}0>''',ip='''<(IPv4/CIDR{{!}}dhcp{{!}}manual)>'''</code>
* EXAMPLE (same as defaults in UI, but DHCP for IPv4 & unique identifiers...):
** <code>pct create '''666''' '''local:vztmpl/debian-11-standard_11.6-1_amd64.tar.zst''' --hostname '''LXC-test''' --password '''nuggit''' --rootfs '''local-lvm''':'''8''' --cores '''1''' --memory '''512''' --swap '''512''' --unprivileged '''1''' --features '''nesting=1''' --net0 name='''eth0''',bridge='''vmbr0''',firewall='''1''',ip='''dhcp'''</code>
* Many more options to be found in the man page for <code>pct</code>

== Configuring & Using the Container ==
Congratulations!

You have an LXC container.

The only user account currently configured is root.

There will be minor oddities, but basically, it's a lightweight VM.

At this point,, for all intents and purposes, it works just like a normal Linux VM. You can sign in at its console and set it up like you would a VM.

I'd suggest adding a regular user so that you can then SSH into the container.

I have a set of configurations that I do to pretty much every LXC container I build. Check [[CopyPasta#For_doing_basic_setup_of_an_LXC_(or_pretty_much_any_Linux_VM...):{{!}}this entry]].

=== Root Access via SSH ===
By default a Proxmox LXC container allows root login only with public key authentication.

ATM... To my knowledge, you either have to set up the public key during build of the LXC or mess about grabbing it from your work machine from inside the LXC itself.

To login to a container with username/password login to your Proxmox host and attach to the container with the following command.

* <code>lxc-attach --name '''VMID'''</code>
(Where VMID is the number of the container)

Edit '''sshd_config'''

* <code>vi /etc/ssh/sshd_config</code>
and change the line '''<code>#PermitRootLogin without-password</code>''' to '''<code>PermitRootLogin yes</code>'''

Restart ssh service for the changes to take effect.

* <code>service ssh restart</code>
(In theory, you should be able to [[HowTo -_ssh#Make_it_easier_to_connect{{!}}set up the public key normally]] now & then revert the ssh_config change.)

== Further Tricks ==
One useful trick is to add storage to the LXC that's not managed by PVE directly. (For instance, I'm building an LXC for use in backing up & reconfiguring local machines & VMs. I'd like to use a dedicated drive in a hot-swap bay on my server.)

Apparently, either [https://pve.proxmox.com/wiki/Linux_Container#_bind_mount_points "Bind Mount Points"] or [https://pve.proxmox.com/wiki/Linux_Container#_device_mount_points "Device Mount Points"] are the solution...

[https://forum.proxmox.com/threads/mount-external-hdd-in-lxc-container.50897/ This thread] and [https://forum.proxmox.com/threads/container-with-physical-disk.42280/ this one] have some helpful discussion...

= Pre-Built Containers =
PVE has a bunch of templates available by a company called [https://www.turnkeylinux.org/ Turnkey Linux].  I've tried a couple...  Not overly impressed so far, but YMMV.

= Stumbling Blocks & Oddities =
== Time Zone Settings ==
Apparently, LXCs default to UTC even if your server is set to your local timezone.

Fortunately, it's not difficult to correct this.

* <code>sudo dpkg-reconfigure tzdata</code>
or
* pick another method from [https://itslinuxguide.com/debian-set-timezone/ this web page].

== DHCP / DNS Difficulties ==
For some reason, If I set up a container with both IPv4 & IPv6 using '''DHCP''', resolved will ONLY use IPv6. This is kind of an issue if I want to use IPv4 DNS...

Fortunately, simply setting IPv6 to '''Static''' fixes the issue.

== Console connection from the PVE UI ==
LXC containers seem to default to '''xterm.js''' rather than the '''noVNC''' console standard with regular VMs.

This isn't a bad thing... Just surprised me.

== Standard LXC containers built upon Debian seem to block non-root users from using ping... ==
'''ping: socket: Operation not permitted'''

[https://www.suse.com/support/kb/doc/?id=000020581 Discussion & a solution]

Solution #3 seems most appropriate...

* <code>sudo chmod u+s /usr/bin/ping</code>

== LXC containers built on Debian don't seem to get usbutils ==

* <code>sudo apt install usbutils</code>

= Container Building Tips =
== Converting unpriveledged container to priveledged ==
# Backup the container
# shutdown the container
# restore from the backup and select "Priveledged" in the dialog.

== Adding NFS to an LXC ==
'''MUST BE A PRIVELEDGED CONTAINER'''

in: Options / Features enable '''Nesting''' and '''NFS'''

Then go ahead & [[Setting up NFS for file sharing{{!}}set up NFS as normal]].

== Using a VPN (OpenVPN or TailScale) on an LXC ==
LXCs do not give access to a '''/dev/tun''' device & this is needed for Tailscale to work.

If you're running the LXC on Proxmox, You can add this feature by editing the containers configuration file.

The following instructions are copied (and/or adapted) from [https://tailscale.com/kb/1130/lxc-unprivileged this page].

For example, using Proxmox to host an LXC with ID 112, the following lines would be added to /etc/pve/lxc/112.conf:

* <code>vi /etc/pve/lxc/112.conf</code>
lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file
If the LXC is already running it will need to be shut down and started again for this change to take effect.

* <code>pct reboot 112</code>

=== Side note about custom LXC templates ===
Sadly, if you generate a template from an LXC that you've done this with, this modification does NOT get applied to LXCs made using that template. You'll have to edit the conf file of any LXC you create from that template.

= PVE CLI Tools for LXC work... =
* pct (Proxmox Container Toolkit)
* pveam (Proxmox VE Appliance Manager)

Quick Notes

2026-06-10T20:03:15Z

Tinker: /* Tips: */

Sections here are really just placeholders with enough information to get started on building actual articles...

= rDNS =
This is gonna get interesting...

What does it take to make rDNS work from outside the network?
== Local Mail Service ==
(This will depend on whether rDNS works out...)

= Better Permissions Handling =
{{{!}} class="wikitable" style="float: right; width: 322px;" border="2"
{{!}}+ Proven on:
{{!}}- --="" debian=""
{{!}} style="text-align: center; width: 60px;" {{!}} [[File:Logo Debian.png{{!}}60px{{!}}link=https://www.debian.org/{{!}}center{{!}}middle{{!}}frameless]]
{{!}} style="text-align: center; width: 40px;" {{!}} 12 (bookworm)
{{!}} 
{{!}}}

* <code>sudo apt install acl</code>
* <code>sudo mkdir /FOOBAR</code>
* <code>sudo chown -R :users /FOOBAR</code>
* <code>sudo chmod -R g+ws /FOOBAR</code>
* <code>sudo setfacl -R -m g:users:rwx /FOOBAR</code>
Now, the directory '''/FOOBAR''' is writeable by anyone in the '''users''' group.

Do keep in mind that '''users''' is a default group that all accounts are a member of. You could also create specific groups and add each user to specific groups for finer granularity.
== Some Links ==
* [https://en.wikipedia.org/wiki/Setuid Wikipedia : setuid]
* [https://en.wikipedia.org/wiki/Access-control_list Wikipedia : Access-control list]
* [https://unix.stackexchange.com/questions/115631/getting-new-files-to-inherit-group-permissions-on-linux/115632#115632 StackExchange : Getting new files to inherit group permissions on Linux]
* [https://unix.stackexchange.com/questions/93868/using-setfacl-to-create-recursive-permissions-for-apache-with-rsync/93871#93871 StackExchange : Using setfacl to create recursive permissions for Apache with rsync]

= Linux USERs and GROUPs =
Eventually to be an article about how granular access control works on Linux...
= Development Testing for a Backup Server =
{{{!}} class="wikitable" style="float: right; width: 322px;" border="2"
{{!}}+ Proven on:
{{!}}- --="" debian=""
{{!}} style="text-align: center; width: 60px;" {{!}} [[File:Logo Debian.png{{!}}60px{{!}}link=https://www.debian.org/{{!}}center{{!}}middle{{!}}frameless]]
{{!}} style="text-align: center; width: 40px;" {{!}} 12 (bookworm)
{{!}} 
{{!}}}
This will eventually be fully fleshed out as instructions for building proper backup servers on separate machines including off-site backups.

Currently testing on a pair of LXCs, both configured as priveledged containers (tho this may not be needed).

'''Interesting side-quest:''' Apparently, unpriviledged containers do a bit of weirdness with bind-mount ownerships. There are [https://pve.proxmox.com/wiki/Unprivileged_LXC_containers instructions] to work around this, but I have yet to work through this... For the moment, I'm using priviledged containers to work around it.

* VMID 999001: storage
* VMID 999002: backup
I've created a user named '''draal''' with specific responsibility for handling backups.(because remote access by '''''root''''' is stupid...)

'''draal''' has an equivalent account on ALL involved machines and is set up with ssh keys to avoid the use of password-based access.

== On the PVE Server: ==
(setting up the storage for the test LXCs...)

* <code>mkdir /mnt/Storage-0</code>
* <code>mkdir /mnt/Storage-1</code>
* <code>mount -L Storage-0 /mnt/Storage-0</code>
* <code>mount -L Storage-1 /mnt/Storage-1</code>
* <code>vi /etc/fstab</code>
LABEL=Storage-0 /mnt/Storage-0 ext4 defaults 0 0
LABEL=Storage-1 /mnt/Storage-1 ext4 defaults 0 0
* <code>pct set 999001 -mp0 /mnt/Storage-0,mp=/mnt/Storage # storage</code>
* <code>pct set 999002 -mp0 /mnt/Storage-1,mp=/mnt/Storage # backup</code>

== On "storage" ==
(logged in as '''draal''')
* <code>ssh-keygen</code>
* <code>ssh-copy-id '''''draal'''''@backup</code>
* <code>sudo apt install acl rsync</code>
* <code>sudo setfacl -R -m u:'''''draal''''':rwx /mnt/Storage</code>

== On "backup" ==
(logged in as '''draal''')
* <code>ssh-keygen</code>
* <code>ssh-copy-id '''''draal'''''@storage</code>
* <code>sudo apt install acl rsync</code>
* <code>sudo setfacl -R -m u:'''''draal''''':rwx /mnt/Storage</code>

== Testing '''rsync''' commands ==
In order to test things, I've basically dumped a copy of my Documents folder into '''/mnt/Storage''' on the '''storage''' LXC. This is around 25GB of incredibley mixed & sloppy crap varying from zero-length placeholders to USB binaries...

Everything from here on is done while logged in to '''storage''' as '''draal'''.
* <code>cd /mnt/Storage</code>
* <code>rsync -Aaiz * backup:/mnt/Storage</code>
** copies all files to the backup machine
*** Took about 7 minutes... (measured by wrapping the command in a '''time''' command)
** (& if you run it again, only copies NEW or CHANGED files)
*** Every change I made took substantially under 1 second...
* <code>rsync -Aaiz --delete * backup:/mnt/Storage</code>
** Will actually synchronize. (i.e. It will '''DELETE''' any files from the destination that have been removed on the source.)
** '''DANGER...DANGER...DANGER'''
*** This may NOT be exactly what you want to do in a backup situation.
*** (But it might... Your choice...)

= Custom LXC Templates =

* Create an LXC
** My example is an NFS capable LXC with VMID 666... (You do you...)
* Configure it
** Install your preferred standard set of apps & such... (Basically, pretend it's your finished LXC & configure it.)

== From the PVE command line: ==
* remove the network interface
** <code>pct set 666 --delete net0</code>
* Do a backup
** Gonna need a bunch of available space (I mount a spare 2TB drive specifically for the purpose on the machine where I build these and '''cd''' to that drive before this step...)
** <code>vzdump 666 --mode stop --compress zstd --dumpdir .</code>
* Rename the backup to a more useful name
** (You'll need to take a look & work out the exact name of the created backup file...)
** <code>mv '''vzdump-lxc-666-2025_02_04-14_19_43.tar.zst''' Deb-12-NFS.tar.zst</code>
* Copy it to someplace you can access it from the PVE UI
** Option 1: For direct '''Upload''' (Some oddities happen... More research needed)
*** <code>scp Deb-12-NFS.tar.zst '''''user'''''@'''''mydesktopmachine''''':~</code>
** Option 2: For '''Download from URL'''
*** <code>scp Deb-12-NFS.tar.zst '''''user'''''@'''''mywebserver''''':/var/www/html/LXCtemplates/</code>
Now you can add it to your '''CT Templates''' storage on any PVE server you're working on.

'''NOTE:''' You can also directly add it to the available templates on the machine you are currently logged into...
* <code>cp Deb-12-NFS.tar.zst /var/lib/vz/template/cache/</code>

== Tips: ==
If you want to put the original back onto the network for further development, you'll need to give it a new network interface. (assuming here that you're using DHCP...)

From the PVE command line:
*<code>pct set 666 -net0 name=net0,bridge=vmbr0,ip=dhcp</code>

Then sign into the LXC itself and:

(useful tip: <code>pct enter 666</code>)
*<code>sudo dhcpcd net0</code>

= Teaching RSYNC =
A nice little [https://www.youtube.com/watch?v=QKCIi-NxJEo intro video] by Veronica

Quick Notes

2026-06-10T19:24:41Z

Tinker: /* Custom LXC Templates */

Sections here are really just placeholders with enough information to get started on building actual articles...

= rDNS =
This is gonna get interesting...

What does it take to make rDNS work from outside the network?
== Local Mail Service ==
(This will depend on whether rDNS works out...)

= Better Permissions Handling =
{{{!}} class="wikitable" style="float: right; width: 322px;" border="2"
{{!}}+ Proven on:
{{!}}- --="" debian=""
{{!}} style="text-align: center; width: 60px;" {{!}} [[File:Logo Debian.png{{!}}60px{{!}}link=https://www.debian.org/{{!}}center{{!}}middle{{!}}frameless]]
{{!}} style="text-align: center; width: 40px;" {{!}} 12 (bookworm)
{{!}} 
{{!}}}

* <code>sudo apt install acl</code>
* <code>sudo mkdir /FOOBAR</code>
* <code>sudo chown -R :users /FOOBAR</code>
* <code>sudo chmod -R g+ws /FOOBAR</code>
* <code>sudo setfacl -R -m g:users:rwx /FOOBAR</code>
Now, the directory '''/FOOBAR''' is writeable by anyone in the '''users''' group.

Do keep in mind that '''users''' is a default group that all accounts are a member of. You could also create specific groups and add each user to specific groups for finer granularity.
== Some Links ==
* [https://en.wikipedia.org/wiki/Setuid Wikipedia : setuid]
* [https://en.wikipedia.org/wiki/Access-control_list Wikipedia : Access-control list]
* [https://unix.stackexchange.com/questions/115631/getting-new-files-to-inherit-group-permissions-on-linux/115632#115632 StackExchange : Getting new files to inherit group permissions on Linux]
* [https://unix.stackexchange.com/questions/93868/using-setfacl-to-create-recursive-permissions-for-apache-with-rsync/93871#93871 StackExchange : Using setfacl to create recursive permissions for Apache with rsync]

= Linux USERs and GROUPs =
Eventually to be an article about how granular access control works on Linux...
= Development Testing for a Backup Server =
{{{!}} class="wikitable" style="float: right; width: 322px;" border="2"
{{!}}+ Proven on:
{{!}}- --="" debian=""
{{!}} style="text-align: center; width: 60px;" {{!}} [[File:Logo Debian.png{{!}}60px{{!}}link=https://www.debian.org/{{!}}center{{!}}middle{{!}}frameless]]
{{!}} style="text-align: center; width: 40px;" {{!}} 12 (bookworm)
{{!}} 
{{!}}}
This will eventually be fully fleshed out as instructions for building proper backup servers on separate machines including off-site backups.

Currently testing on a pair of LXCs, both configured as priveledged containers (tho this may not be needed).

'''Interesting side-quest:''' Apparently, unpriviledged containers do a bit of weirdness with bind-mount ownerships. There are [https://pve.proxmox.com/wiki/Unprivileged_LXC_containers instructions] to work around this, but I have yet to work through this... For the moment, I'm using priviledged containers to work around it.

* VMID 999001: storage
* VMID 999002: backup
I've created a user named '''draal''' with specific responsibility for handling backups.(because remote access by '''''root''''' is stupid...)

'''draal''' has an equivalent account on ALL involved machines and is set up with ssh keys to avoid the use of password-based access.

== On the PVE Server: ==
(setting up the storage for the test LXCs...)

* <code>mkdir /mnt/Storage-0</code>
* <code>mkdir /mnt/Storage-1</code>
* <code>mount -L Storage-0 /mnt/Storage-0</code>
* <code>mount -L Storage-1 /mnt/Storage-1</code>
* <code>vi /etc/fstab</code>
LABEL=Storage-0 /mnt/Storage-0 ext4 defaults 0 0
LABEL=Storage-1 /mnt/Storage-1 ext4 defaults 0 0
* <code>pct set 999001 -mp0 /mnt/Storage-0,mp=/mnt/Storage # storage</code>
* <code>pct set 999002 -mp0 /mnt/Storage-1,mp=/mnt/Storage # backup</code>

== On "storage" ==
(logged in as '''draal''')
* <code>ssh-keygen</code>
* <code>ssh-copy-id '''''draal'''''@backup</code>
* <code>sudo apt install acl rsync</code>
* <code>sudo setfacl -R -m u:'''''draal''''':rwx /mnt/Storage</code>

== On "backup" ==
(logged in as '''draal''')
* <code>ssh-keygen</code>
* <code>ssh-copy-id '''''draal'''''@storage</code>
* <code>sudo apt install acl rsync</code>
* <code>sudo setfacl -R -m u:'''''draal''''':rwx /mnt/Storage</code>

== Testing '''rsync''' commands ==
In order to test things, I've basically dumped a copy of my Documents folder into '''/mnt/Storage''' on the '''storage''' LXC. This is around 25GB of incredibley mixed & sloppy crap varying from zero-length placeholders to USB binaries...

Everything from here on is done while logged in to '''storage''' as '''draal'''.
* <code>cd /mnt/Storage</code>
* <code>rsync -Aaiz * backup:/mnt/Storage</code>
** copies all files to the backup machine
*** Took about 7 minutes... (measured by wrapping the command in a '''time''' command)
** (& if you run it again, only copies NEW or CHANGED files)
*** Every change I made took substantially under 1 second...
* <code>rsync -Aaiz --delete * backup:/mnt/Storage</code>
** Will actually synchronize. (i.e. It will '''DELETE''' any files from the destination that have been removed on the source.)
** '''DANGER...DANGER...DANGER'''
*** This may NOT be exactly what you want to do in a backup situation.
*** (But it might... Your choice...)

= Custom LXC Templates =

* Create an LXC
** My example is an NFS capable LXC with VMID 666... (You do you...)
* Configure it
** Install your preferred standard set of apps & such... (Basically, pretend it's your finished LXC & configure it.)

== From the PVE command line: ==
* remove the network interface
** <code>pct set 666 --delete net0</code>
* Do a backup
** Gonna need a bunch of available space (I mount a spare 2TB drive specifically for the purpose on the machine where I build these and '''cd''' to that drive before this step...)
** <code>vzdump 666 --mode stop --compress zstd --dumpdir .</code>
* Rename the backup to a more useful name
** (You'll need to take a look & work out the exact name of the created backup file...)
** <code>mv '''vzdump-lxc-666-2025_02_04-14_19_43.tar.zst''' Deb-12-NFS.tar.zst</code>
* Copy it to someplace you can access it from the PVE UI
** Option 1: For direct '''Upload''' (Some oddities happen... More research needed)
*** <code>scp Deb-12-NFS.tar.zst '''''user'''''@'''''mydesktopmachine''''':~</code>
** Option 2: For '''Download from URL'''
*** <code>scp Deb-12-NFS.tar.zst '''''user'''''@'''''mywebserver''''':/var/www/html/LXCtemplates/</code>
Now you can add it to your '''CT Templates''' storage on any PVE server you're working on.

'''NOTE:''' You can also directly add it to the available templates on the machine you are currently logged into...
* <code>cp Deb-12-NFS.tar.zst /var/lib/vz/template/cache/</code>

== Tips: ==
If you want to put the original back onto the network for further development, you'll need to give it a new network interface. (assuming here that you're using DHCP...)

From the PVE command line:
*<code>pct set 666 -net0 name=net0,bridge=vmbr0,ip=dhcp</code>

Then sign into the LXC itself and:
*<code>sudo dhcpcd net0</code>

= Teaching RSYNC =
A nice little [https://www.youtube.com/watch?v=QKCIi-NxJEo intro video] by Veronica

Quick Notes

2026-06-10T18:21:28Z

Tinker: /* From the PVE command line: */

Sections here are really just placeholders with enough information to get started on building actual articles...

= rDNS =
This is gonna get interesting...

What does it take to make rDNS work from outside the network?
== Local Mail Service ==
(This will depend on whether rDNS works out...)

= Better Permissions Handling =
{{{!}} class="wikitable" style="float: right; width: 322px;" border="2"
{{!}}+ Proven on:
{{!}}- --="" debian=""
{{!}} style="text-align: center; width: 60px;" {{!}} [[File:Logo Debian.png{{!}}60px{{!}}link=https://www.debian.org/{{!}}center{{!}}middle{{!}}frameless]]
{{!}} style="text-align: center; width: 40px;" {{!}} 12 (bookworm)
{{!}} 
{{!}}}

* <code>sudo apt install acl</code>
* <code>sudo mkdir /FOOBAR</code>
* <code>sudo chown -R :users /FOOBAR</code>
* <code>sudo chmod -R g+ws /FOOBAR</code>
* <code>sudo setfacl -R -m g:users:rwx /FOOBAR</code>
Now, the directory '''/FOOBAR''' is writeable by anyone in the '''users''' group.

Do keep in mind that '''users''' is a default group that all accounts are a member of. You could also create specific groups and add each user to specific groups for finer granularity.
== Some Links ==
* [https://en.wikipedia.org/wiki/Setuid Wikipedia : setuid]
* [https://en.wikipedia.org/wiki/Access-control_list Wikipedia : Access-control list]
* [https://unix.stackexchange.com/questions/115631/getting-new-files-to-inherit-group-permissions-on-linux/115632#115632 StackExchange : Getting new files to inherit group permissions on Linux]
* [https://unix.stackexchange.com/questions/93868/using-setfacl-to-create-recursive-permissions-for-apache-with-rsync/93871#93871 StackExchange : Using setfacl to create recursive permissions for Apache with rsync]

= Linux USERs and GROUPs =
Eventually to be an article about how granular access control works on Linux...
= Development Testing for a Backup Server =
{{{!}} class="wikitable" style="float: right; width: 322px;" border="2"
{{!}}+ Proven on:
{{!}}- --="" debian=""
{{!}} style="text-align: center; width: 60px;" {{!}} [[File:Logo Debian.png{{!}}60px{{!}}link=https://www.debian.org/{{!}}center{{!}}middle{{!}}frameless]]
{{!}} style="text-align: center; width: 40px;" {{!}} 12 (bookworm)
{{!}} 
{{!}}}
This will eventually be fully fleshed out as instructions for building proper backup servers on separate machines including off-site backups.

Currently testing on a pair of LXCs, both configured as priveledged containers (tho this may not be needed).

'''Interesting side-quest:''' Apparently, unpriviledged containers do a bit of weirdness with bind-mount ownerships. There are [https://pve.proxmox.com/wiki/Unprivileged_LXC_containers instructions] to work around this, but I have yet to work through this... For the moment, I'm using priviledged containers to work around it.

* VMID 999001: storage
* VMID 999002: backup
I've created a user named '''draal''' with specific responsibility for handling backups.(because remote access by '''''root''''' is stupid...)

'''draal''' has an equivalent account on ALL involved machines and is set up with ssh keys to avoid the use of password-based access.

== On the PVE Server: ==
(setting up the storage for the test LXCs...)

* <code>mkdir /mnt/Storage-0</code>
* <code>mkdir /mnt/Storage-1</code>
* <code>mount -L Storage-0 /mnt/Storage-0</code>
* <code>mount -L Storage-1 /mnt/Storage-1</code>
* <code>vi /etc/fstab</code>
LABEL=Storage-0 /mnt/Storage-0 ext4 defaults 0 0
LABEL=Storage-1 /mnt/Storage-1 ext4 defaults 0 0
* <code>pct set 999001 -mp0 /mnt/Storage-0,mp=/mnt/Storage # storage</code>
* <code>pct set 999002 -mp0 /mnt/Storage-1,mp=/mnt/Storage # backup</code>

== On "storage" ==
(logged in as '''draal''')
* <code>ssh-keygen</code>
* <code>ssh-copy-id '''''draal'''''@backup</code>
* <code>sudo apt install acl rsync</code>
* <code>sudo setfacl -R -m u:'''''draal''''':rwx /mnt/Storage</code>

== On "backup" ==
(logged in as '''draal''')
* <code>ssh-keygen</code>
* <code>ssh-copy-id '''''draal'''''@storage</code>
* <code>sudo apt install acl rsync</code>
* <code>sudo setfacl -R -m u:'''''draal''''':rwx /mnt/Storage</code>

== Testing '''rsync''' commands ==
In order to test things, I've basically dumped a copy of my Documents folder into '''/mnt/Storage''' on the '''storage''' LXC. This is around 25GB of incredibley mixed & sloppy crap varying from zero-length placeholders to USB binaries...

Everything from here on is done while logged in to '''storage''' as '''draal'''.
* <code>cd /mnt/Storage</code>
* <code>rsync -Aaiz * backup:/mnt/Storage</code>
** copies all files to the backup machine
*** Took about 7 minutes... (measured by wrapping the command in a '''time''' command)
** (& if you run it again, only copies NEW or CHANGED files)
*** Every change I made took substantially under 1 second...
* <code>rsync -Aaiz --delete * backup:/mnt/Storage</code>
** Will actually synchronize. (i.e. It will '''DELETE''' any files from the destination that have been removed on the source.)
** '''DANGER...DANGER...DANGER'''
*** This may NOT be exactly what you want to do in a backup situation.
*** (But it might... Your choice...)

= Custom LXC Templates =

* Create an LXC
** My example is an NFS capable LXC with VMID 666... (You do you...)
* Configure it
** Install your preferred standard set of apps & such... (Basically, pretend it's your finished LXC & configure it.)

== From the PVE command line: ==
* remove the network interface
** <code>pct set 666 --delete net0</code>
* Do a backup
** Gonna need a bunch of available space (I mount a spare 2TB drive specifically for the purpose on the machine where I build these and '''cd''' to that drive before this step...)
** <code>vzdump 666 --mode stop --compress zstd --dumpdir .</code>
* Rename the backup to a more useful name
** (You'll need to take a look & work out the exact name of the created backup file...)
** <code>mv '''vzdump-lxc-666-2025_02_04-14_19_43.tar.zst''' Deb-12-NFS.tar.zst</code>
* Copy it to someplace you can access it from the PVE UI
** Option 1: For direct '''Upload''' (Some oddities happen... More research needed)
*** <code>scp Deb-12-NFS.tar.zst '''''user'''''@'''''mydesktopmachine''''':~</code>
** Option 2: For '''Download from URL'''
*** <code>scp Deb-12-NFS.tar.zst '''''user'''''@'''''mywebserver''''':/var/www/html/LXCtemplates/</code>
Now you can add it to your '''CT Templates''' storage on any PVE server you're working on.

'''NOTE:''' You can also directly add it to the available templates on the machine you are currently logged into...
* <code>cp Deb-12-NFS.tar.zst /var/lib/vz/template/cache/</code>

= Teaching RSYNC =
A nice little [https://www.youtube.com/watch?v=QKCIi-NxJEo intro video] by Veronica

Quick Notes

2026-06-10T18:11:24Z

Tinker: /* Custom LXC Templates */

Sections here are really just placeholders with enough information to get started on building actual articles...

= rDNS =
This is gonna get interesting...

What does it take to make rDNS work from outside the network?
== Local Mail Service ==
(This will depend on whether rDNS works out...)

= Better Permissions Handling =
{{{!}} class="wikitable" style="float: right; width: 322px;" border="2"
{{!}}+ Proven on:
{{!}}- --="" debian=""
{{!}} style="text-align: center; width: 60px;" {{!}} [[File:Logo Debian.png{{!}}60px{{!}}link=https://www.debian.org/{{!}}center{{!}}middle{{!}}frameless]]
{{!}} style="text-align: center; width: 40px;" {{!}} 12 (bookworm)
{{!}} 
{{!}}}

* <code>sudo apt install acl</code>
* <code>sudo mkdir /FOOBAR</code>
* <code>sudo chown -R :users /FOOBAR</code>
* <code>sudo chmod -R g+ws /FOOBAR</code>
* <code>sudo setfacl -R -m g:users:rwx /FOOBAR</code>
Now, the directory '''/FOOBAR''' is writeable by anyone in the '''users''' group.

Do keep in mind that '''users''' is a default group that all accounts are a member of. You could also create specific groups and add each user to specific groups for finer granularity.
== Some Links ==
* [https://en.wikipedia.org/wiki/Setuid Wikipedia : setuid]
* [https://en.wikipedia.org/wiki/Access-control_list Wikipedia : Access-control list]
* [https://unix.stackexchange.com/questions/115631/getting-new-files-to-inherit-group-permissions-on-linux/115632#115632 StackExchange : Getting new files to inherit group permissions on Linux]
* [https://unix.stackexchange.com/questions/93868/using-setfacl-to-create-recursive-permissions-for-apache-with-rsync/93871#93871 StackExchange : Using setfacl to create recursive permissions for Apache with rsync]

= Linux USERs and GROUPs =
Eventually to be an article about how granular access control works on Linux...
= Development Testing for a Backup Server =
{{{!}} class="wikitable" style="float: right; width: 322px;" border="2"
{{!}}+ Proven on:
{{!}}- --="" debian=""
{{!}} style="text-align: center; width: 60px;" {{!}} [[File:Logo Debian.png{{!}}60px{{!}}link=https://www.debian.org/{{!}}center{{!}}middle{{!}}frameless]]
{{!}} style="text-align: center; width: 40px;" {{!}} 12 (bookworm)
{{!}} 
{{!}}}
This will eventually be fully fleshed out as instructions for building proper backup servers on separate machines including off-site backups.

Currently testing on a pair of LXCs, both configured as priveledged containers (tho this may not be needed).

'''Interesting side-quest:''' Apparently, unpriviledged containers do a bit of weirdness with bind-mount ownerships. There are [https://pve.proxmox.com/wiki/Unprivileged_LXC_containers instructions] to work around this, but I have yet to work through this... For the moment, I'm using priviledged containers to work around it.

* VMID 999001: storage
* VMID 999002: backup
I've created a user named '''draal''' with specific responsibility for handling backups.(because remote access by '''''root''''' is stupid...)

'''draal''' has an equivalent account on ALL involved machines and is set up with ssh keys to avoid the use of password-based access.

== On the PVE Server: ==
(setting up the storage for the test LXCs...)

* <code>mkdir /mnt/Storage-0</code>
* <code>mkdir /mnt/Storage-1</code>
* <code>mount -L Storage-0 /mnt/Storage-0</code>
* <code>mount -L Storage-1 /mnt/Storage-1</code>
* <code>vi /etc/fstab</code>
LABEL=Storage-0 /mnt/Storage-0 ext4 defaults 0 0
LABEL=Storage-1 /mnt/Storage-1 ext4 defaults 0 0
* <code>pct set 999001 -mp0 /mnt/Storage-0,mp=/mnt/Storage # storage</code>
* <code>pct set 999002 -mp0 /mnt/Storage-1,mp=/mnt/Storage # backup</code>

== On "storage" ==
(logged in as '''draal''')
* <code>ssh-keygen</code>
* <code>ssh-copy-id '''''draal'''''@backup</code>
* <code>sudo apt install acl rsync</code>
* <code>sudo setfacl -R -m u:'''''draal''''':rwx /mnt/Storage</code>

== On "backup" ==
(logged in as '''draal''')
* <code>ssh-keygen</code>
* <code>ssh-copy-id '''''draal'''''@storage</code>
* <code>sudo apt install acl rsync</code>
* <code>sudo setfacl -R -m u:'''''draal''''':rwx /mnt/Storage</code>

== Testing '''rsync''' commands ==
In order to test things, I've basically dumped a copy of my Documents folder into '''/mnt/Storage''' on the '''storage''' LXC. This is around 25GB of incredibley mixed & sloppy crap varying from zero-length placeholders to USB binaries...

Everything from here on is done while logged in to '''storage''' as '''draal'''.
* <code>cd /mnt/Storage</code>
* <code>rsync -Aaiz * backup:/mnt/Storage</code>
** copies all files to the backup machine
*** Took about 7 minutes... (measured by wrapping the command in a '''time''' command)
** (& if you run it again, only copies NEW or CHANGED files)
*** Every change I made took substantially under 1 second...
* <code>rsync -Aaiz --delete * backup:/mnt/Storage</code>
** Will actually synchronize. (i.e. It will '''DELETE''' any files from the destination that have been removed on the source.)
** '''DANGER...DANGER...DANGER'''
*** This may NOT be exactly what you want to do in a backup situation.
*** (But it might... Your choice...)

= Custom LXC Templates =

* Create an LXC
** My example is an NFS capable LXC with VMID 666... (You do you...)
* Configure it
** Install your preferred standard set of apps & such... (Basically, pretend it's your finished LXC & configure it.)

== From the PVE command line: ==
* remove the network interface
** <code>pct set 666 --delete net0</code>
* Do a backup
** Gonna need a bunch of available space (I mount a spare 2TB drive specifically for the purpose on the machine where I build these and '''cd''' to that drive before this step...)
** <code>vzdump 666 --mode stop --compress zstd --dumpdir .</code>
* Rename the backup to a more useful name
** <code>mv vzdump-lxc-666-2025_02_04-14_19_43.tar.zst Deb-12-NFS.tar.zst</code>
* Copy it to someplace you can access it from the PVE UI
** Option 1: For direct '''Upload''' (Some oddities happen... More research needed)
*** <code>scp Deb-12-NFS.tar.zst '''''user'''''@'''''mydesktopmachine''''':~</code>
** Option 2: For '''Download from URL'''
*** <code>scp Deb-12-NFS.tar.zst '''''user'''''@'''''mywebserver''''':/var/www/html/LXCtemplates/</code>
Now you can add it to your '''CT Templates''' storage on any PVE server you're working on.

'''NOTE:''' You can also directly add it to the available templates on the machine you are currently logged into...
* <code>cp Deb-12-NFS.tar.zst /var/lib/vz/template/cache/</code>

= Teaching RSYNC =
A nice little [https://www.youtube.com/watch?v=QKCIi-NxJEo intro video] by Veronica

Quick Notes

2026-06-10T18:09:17Z

Tinker: /* From the PVE command line: */

Sections here are really just placeholders with enough information to get started on building actual articles...

= rDNS =
This is gonna get interesting...

What does it take to make rDNS work from outside the network?
== Local Mail Service ==
(This will depend on whether rDNS works out...)

= Better Permissions Handling =
{{{!}} class="wikitable" style="float: right; width: 322px;" border="2"
{{!}}+ Proven on:
{{!}}- --="" debian=""
{{!}} style="text-align: center; width: 60px;" {{!}} [[File:Logo Debian.png{{!}}60px{{!}}link=https://www.debian.org/{{!}}center{{!}}middle{{!}}frameless]]
{{!}} style="text-align: center; width: 40px;" {{!}} 12 (bookworm)
{{!}} 
{{!}}}

* <code>sudo apt install acl</code>
* <code>sudo mkdir /FOOBAR</code>
* <code>sudo chown -R :users /FOOBAR</code>
* <code>sudo chmod -R g+ws /FOOBAR</code>
* <code>sudo setfacl -R -m g:users:rwx /FOOBAR</code>
Now, the directory '''/FOOBAR''' is writeable by anyone in the '''users''' group.

Do keep in mind that '''users''' is a default group that all accounts are a member of. You could also create specific groups and add each user to specific groups for finer granularity.
== Some Links ==
* [https://en.wikipedia.org/wiki/Setuid Wikipedia : setuid]
* [https://en.wikipedia.org/wiki/Access-control_list Wikipedia : Access-control list]
* [https://unix.stackexchange.com/questions/115631/getting-new-files-to-inherit-group-permissions-on-linux/115632#115632 StackExchange : Getting new files to inherit group permissions on Linux]
* [https://unix.stackexchange.com/questions/93868/using-setfacl-to-create-recursive-permissions-for-apache-with-rsync/93871#93871 StackExchange : Using setfacl to create recursive permissions for Apache with rsync]

= Linux USERs and GROUPs =
Eventually to be an article about how granular access control works on Linux...
= Development Testing for a Backup Server =
{{{!}} class="wikitable" style="float: right; width: 322px;" border="2"
{{!}}+ Proven on:
{{!}}- --="" debian=""
{{!}} style="text-align: center; width: 60px;" {{!}} [[File:Logo Debian.png{{!}}60px{{!}}link=https://www.debian.org/{{!}}center{{!}}middle{{!}}frameless]]
{{!}} style="text-align: center; width: 40px;" {{!}} 12 (bookworm)
{{!}} 
{{!}}}
This will eventually be fully fleshed out as instructions for building proper backup servers on separate machines including off-site backups.

Currently testing on a pair of LXCs, both configured as priveledged containers (tho this may not be needed).

'''Interesting side-quest:''' Apparently, unpriviledged containers do a bit of weirdness with bind-mount ownerships. There are [https://pve.proxmox.com/wiki/Unprivileged_LXC_containers instructions] to work around this, but I have yet to work through this... For the moment, I'm using priviledged containers to work around it.

* VMID 999001: storage
* VMID 999002: backup
I've created a user named '''draal''' with specific responsibility for handling backups.(because remote access by '''''root''''' is stupid...)

'''draal''' has an equivalent account on ALL involved machines and is set up with ssh keys to avoid the use of password-based access.

== On the PVE Server: ==
(setting up the storage for the test LXCs...)

* <code>mkdir /mnt/Storage-0</code>
* <code>mkdir /mnt/Storage-1</code>
* <code>mount -L Storage-0 /mnt/Storage-0</code>
* <code>mount -L Storage-1 /mnt/Storage-1</code>
* <code>vi /etc/fstab</code>
LABEL=Storage-0 /mnt/Storage-0 ext4 defaults 0 0
LABEL=Storage-1 /mnt/Storage-1 ext4 defaults 0 0
* <code>pct set 999001 -mp0 /mnt/Storage-0,mp=/mnt/Storage # storage</code>
* <code>pct set 999002 -mp0 /mnt/Storage-1,mp=/mnt/Storage # backup</code>

== On "storage" ==
(logged in as '''draal''')
* <code>ssh-keygen</code>
* <code>ssh-copy-id '''''draal'''''@backup</code>
* <code>sudo apt install acl rsync</code>
* <code>sudo setfacl -R -m u:'''''draal''''':rwx /mnt/Storage</code>

== On "backup" ==
(logged in as '''draal''')
* <code>ssh-keygen</code>
* <code>ssh-copy-id '''''draal'''''@storage</code>
* <code>sudo apt install acl rsync</code>
* <code>sudo setfacl -R -m u:'''''draal''''':rwx /mnt/Storage</code>

== Testing '''rsync''' commands ==
In order to test things, I've basically dumped a copy of my Documents folder into '''/mnt/Storage''' on the '''storage''' LXC. This is around 25GB of incredibley mixed & sloppy crap varying from zero-length placeholders to USB binaries...

Everything from here on is done while logged in to '''storage''' as '''draal'''.
* <code>cd /mnt/Storage</code>
* <code>rsync -Aaiz * backup:/mnt/Storage</code>
** copies all files to the backup machine
*** Took about 7 minutes... (measured by wrapping the command in a '''time''' command)
** (& if you run it again, only copies NEW or CHANGED files)
*** Every change I made took substantially under 1 second...
* <code>rsync -Aaiz --delete * backup:/mnt/Storage</code>
** Will actually synchronize. (i.e. It will '''DELETE''' any files from the destination that have been removed on the source.)
** '''DANGER...DANGER...DANGER'''
*** This may NOT be exactly what you want to do in a backup situation.
*** (But it might... Your choice...)

= Custom LXC Templates =

* Create an LXC
** My example is an NFS capable LXC with VMID 666...
* Configure it
** Install your preferred standard set of apps & such...

== From the PVE command line: ==
* remove the network interface
** <code>pct set 666 --delete net0</code>
* Do a backup
** Gonna need a bunch of available space (I mount a spare 2TB drive specifically for the purpose on the machine where I build these and '''cd''' to that drive before this step...)
** <code>vzdump 666 --mode stop --compress zstd --dumpdir .</code>
* Rename the backup to a more useful name
** <code>mv vzdump-lxc-666-2025_02_04-14_19_43.tar.zst Deb-12-NFS.tar.zst</code>
* Copy it to someplace you can access it from the PVE UI
** Option 1: For direct '''Upload''' (Some oddities happen... More research needed)
*** <code>scp Deb-12-NFS.tar.zst '''''user'''''@'''''mydesktopmachine''''':~</code>
** Option 2: For '''Download from URL'''
*** <code>scp Deb-12-NFS.tar.zst '''''user'''''@'''''mywebserver''''':/var/www/html/LXCtemplates/</code>
Now you can add it to your '''CT Templates''' storage on any PVE server you're working on.

'''NOTE:''' You can also directly add it to the available templates on the machine you are currently logged into...
* <code>cp Deb-12-NFS.tar.zst /var/lib/vz/template/cache/</code>

= Teaching RSYNC =
A nice little [https://www.youtube.com/watch?v=QKCIi-NxJEo intro video] by Veronica

CopyPasta

2026-04-12T23:18:13Z

Tinker: /* For doing basic setup of an LXC (or pretty much any Linux VM...) (Heck, I run these on physical machines too.): */

= For doing basic setup of an LXC (or pretty much any Linux VM...) (Heck, I run these on physical machines too.): =
I run this set of commands '''as root''' on nearly every new VM or LXC I spin up. That way, I have a consistant environment to work in with all the tools I rely on.

Note: You CAN copyPasta groups of commands all at once. BUT: in this case, everything indented (after <code>su - '''yourname'''</code>) needs to be pasted AFTER that command has run.

* <code>apt update</code>
* <code>apt -y upgrade</code>
* <code>apt -y install sudo rsync vim curl mosquitto-clients</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/RootStuff/DOT.bashrc</nowiki> -O .bashrc</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/RootStuff/DOT.vimrc</nowiki> -O .vimrc</code>
* <code>mkdir bin</code>
* <code>source .bashrc</code>
* <code>apt -y install gnupg tmux htop ### Optional...</code>
* <code>apt -y install qemu-guest-agent ### Only on VMs. (Wish it worked for LXCs as well tho...)</code>
* <code>adduser '''yourname'''</code>
* <code>adduser '''yourname''' sudo</code>
* <code>su - '''yourname'''</code>
** <code>wget <nowiki>https://www.nerdmage.ca/Downloads/UserStuff/DOT.bashrc</nowiki> -O .bashrc</code>
** <code>wget <nowiki>https://www.nerdmage.ca/Downloads/UserStuff/DOT.vimrc</nowiki> -O .vimrc</code>
** <code>mkdir bin</code>
** <code>source .bashrc</code>
** <code>ssh-keygen ### Optional... But rather handy.</code>

== Bonus item: if your LXC has NFS enabled (or you're working on a VM or bare-metal machine) ==
* <code>sudo apt install nfs-kernel-server</code>
* <code>sudo vi /etc/fstab</code>
** Copy in your default set of NFS mounts (don't forget to actually create any needed mount points) I tend to mount all of the typical subfolders of my <code>/home/'''USER'''</code> folder from a network share...
* <code>sudo systemctl daemon-reload</code>
* <code>sudo mount -a</code>

=New System Cleanup & Prep=
Freshly installed systems usually have a bunch of stuff installed that you will never use.

For myself, I don't generally have any interest in the games. I also don't use some of the Internet apps that seem important to the developers/maintainers.

These instructions are based on running Debian with the Cinnamon Desktop Environment...
==Remove all the Games==

* <code>sudo apt purge gnome-2048 aisleriot atomix gnome-chess five-or-more hitori iagno gnome-klotski lightsoff gnome-mahjongg gnome-mines gnome-nibbles quadrapassel four-in-a-row gnome-robots gnome-sudoku swell-foop tali gnome-taquin gnome-tetravex -y</code>
* <code>sudo apt autoremove -y</code>

==Remove obsolete and/or silly Internet apps==

* <code>sudo apt purge hexchat pidgin transmission-gtk thunderbird -y</code>
* <code>sudo apt autoremove -y</code>

== Some others I don't use (from LMDE7) ==

* <code>sudo apt purge hypnotix rhythmbox -y</code>
* <code>sudo apt autoremove -y</code>

== Get rid of Firefox ==
* <code>sudo apt purge firefox*</code>

= Simplified installation techniques & instructions =
Some cross-platform software seems to have Linux installation instructions written by people who have never actually used Linux.

(Or, at least, people who believe in making life difficult...)

So...

* [[Installing Brave Browser]] (Properly... Not FLATPAK)
* [[Installing TP-Link Omada SDN Controller on a Debian-based LXC]]
Also, many installation instructions are long-winded or confusing...

So...

* [[Installing Docker on a Debian-based system]]

= Building/Installing things from source =
You may have noticed that the various distro repositories tend to have outdated versions of some (most) packages. While this makes perfect sense, sometimes you want the newest features & fixes. (Or maybe you just want to enable something that the repo managers figured wouldn't be useful...)

Annoyingly, it is rather common for build instructions to suck really badly.

So...

* [[Arp-Scan from Source{{!}}Arp-Scan]]
* [[FreeCAD from Source{{!}}FreeCAD]]
* [[KiCAD from Source{{!}}KiCAD]]
* [[Mosquitto from Source{{!}}Mosquitto]]

= Useful console display for Proxmox Virtual Environment =
I find it nice to have stats & such on the console of a server. This way I can just look & see what's happening with the machine.

* <code>apt update</code>
* <code>apt -y upgrade</code>
* <code>apt -y install tmux htop</code>
* <code>cd /usr/local/bin</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/PVEStuff/bin/TMUX-console</nowiki> -O TMUX-console</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/PVEStuff/bin/TM-guestlist</nowiki> -O TM-guestlist</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/PVEStuff/bin/TM-clusterstatus</nowiki> -O TM-clusterstatus</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/PVEStuff/bin/TM-systemstatus</nowiki> -O TM-systemstatus</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/PVEStuff/bin/TM-UPSstatus</nowiki> -O TM-UPSstatus</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/PVEStuff/bin/TM-FixThis</nowiki> -O TM-FixThis</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/PVEStuff/bin/TM-ShowMe</nowiki> -O TM-ShowMe</code>
* <code>chmod +x TM*</code>

You will need to edit '''TM-UPSstatus''' to match local configurations (i.e. UPS name).

Running <code>TMUX-console</code> at the console will create a formatted screen of useful system information that you can reach by attaching to the '''tmux''' session from a terminal (SSH) session.

CopyPasta

2026-04-12T20:05:46Z

Tinker: /* Some others I don't use (from LMDE7) */

= For doing basic setup of an LXC (or pretty much any Linux VM...) (Heck, I run these on physical machines too.): =
I run this set of commands '''as root''' on nearly every new VM or LXC I spin up. That way, I have a consistant environment to work in with all the tools I rely on.

Note: You CAN copyPasta groups of commands all at once. BUT: in this case, everything indented (after <code>su - '''yourname'''</code>) needs to be pasted AFTER that command has run.

* <code>apt update</code>
* <code>apt -y upgrade</code>
* <code>apt -y install sudo rsync vim curl mosquitto-clients</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/RootStuff/DOT.bashrc</nowiki> -O .bashrc</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/RootStuff/DOT.vimrc</nowiki> -O .vimrc</code>
* <code>mkdir bin</code>
* <code>source .bashrc</code>
* <code>apt -y install gnupg tmux htop ### Optional...</code>
* <code>apt -y install qemu-guest-agent ### Only on VMs. (Wish it worked for LXCs as well tho...)</code>
* <code>adduser '''yourname'''</code>
* <code>adduser '''yourname''' sudo</code>
* <code>su - '''yourname'''</code>
** <code>wget <nowiki>https://www.nerdmage.ca/Downloads/UserStuff/DOT.bashrc</nowiki> -O .bashrc</code>
** <code>wget <nowiki>https://www.nerdmage.ca/Downloads/UserStuff/DOT.vimrc</nowiki> -O .vimrc</code>
** <code>mkdir bin</code>
** <code>source .bashrc</code>
** <code>ssh-keygen ### Optional... But rather handy.</code>

=New System Cleanup & Prep=
Freshly installed systems usually have a bunch of stuff installed that you will never use.

For myself, I don't generally have any interest in the games. I also don't use some of the Internet apps that seem important to the developers/maintainers.

These instructions are based on running Debian with the Cinnamon Desktop Environment...
==Remove all the Games==

* <code>sudo apt purge gnome-2048 aisleriot atomix gnome-chess five-or-more hitori iagno gnome-klotski lightsoff gnome-mahjongg gnome-mines gnome-nibbles quadrapassel four-in-a-row gnome-robots gnome-sudoku swell-foop tali gnome-taquin gnome-tetravex -y</code>
* <code>sudo apt autoremove -y</code>

==Remove obsolete and/or silly Internet apps==

* <code>sudo apt purge hexchat pidgin transmission-gtk thunderbird -y</code>
* <code>sudo apt autoremove -y</code>

== Some others I don't use (from LMDE7) ==

* <code>sudo apt purge hypnotix rhythmbox -y</code>
* <code>sudo apt autoremove -y</code>

== Get rid of Firefox ==
* <code>sudo apt purge firefox*</code>

= Simplified installation techniques & instructions =
Some cross-platform software seems to have Linux installation instructions written by people who have never actually used Linux.

(Or, at least, people who believe in making life difficult...)

So...

* [[Installing Brave Browser]] (Properly... Not FLATPAK)
* [[Installing TP-Link Omada SDN Controller on a Debian-based LXC]]
Also, many installation instructions are long-winded or confusing...

So...

* [[Installing Docker on a Debian-based system]]

= Building/Installing things from source =
You may have noticed that the various distro repositories tend to have outdated versions of some (most) packages. While this makes perfect sense, sometimes you want the newest features & fixes. (Or maybe you just want to enable something that the repo managers figured wouldn't be useful...)

Annoyingly, it is rather common for build instructions to suck really badly.

So...

* [[Arp-Scan from Source{{!}}Arp-Scan]]
* [[FreeCAD from Source{{!}}FreeCAD]]
* [[KiCAD from Source{{!}}KiCAD]]
* [[Mosquitto from Source{{!}}Mosquitto]]

= Useful console display for Proxmox Virtual Environment =
I find it nice to have stats & such on the console of a server. This way I can just look & see what's happening with the machine.

* <code>apt update</code>
* <code>apt -y upgrade</code>
* <code>apt -y install tmux htop</code>
* <code>cd /usr/local/bin</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/PVEStuff/bin/TMUX-console</nowiki> -O TMUX-console</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/PVEStuff/bin/TM-guestlist</nowiki> -O TM-guestlist</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/PVEStuff/bin/TM-clusterstatus</nowiki> -O TM-clusterstatus</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/PVEStuff/bin/TM-systemstatus</nowiki> -O TM-systemstatus</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/PVEStuff/bin/TM-UPSstatus</nowiki> -O TM-UPSstatus</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/PVEStuff/bin/TM-FixThis</nowiki> -O TM-FixThis</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/PVEStuff/bin/TM-ShowMe</nowiki> -O TM-ShowMe</code>
* <code>chmod +x TM*</code>

You will need to edit '''TM-UPSstatus''' to match local configurations (i.e. UPS name).

Running <code>TMUX-console</code> at the console will create a formatted screen of useful system information that you can reach by attaching to the '''tmux''' session from a terminal (SSH) session.

CopyPasta

2026-04-12T20:01:19Z

Tinker: /* Some others I don't use (from LMDE7) */

= For doing basic setup of an LXC (or pretty much any Linux VM...) (Heck, I run these on physical machines too.): =
I run this set of commands '''as root''' on nearly every new VM or LXC I spin up. That way, I have a consistant environment to work in with all the tools I rely on.

Note: You CAN copyPasta groups of commands all at once. BUT: in this case, everything indented (after <code>su - '''yourname'''</code>) needs to be pasted AFTER that command has run.

* <code>apt update</code>
* <code>apt -y upgrade</code>
* <code>apt -y install sudo rsync vim curl mosquitto-clients</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/RootStuff/DOT.bashrc</nowiki> -O .bashrc</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/RootStuff/DOT.vimrc</nowiki> -O .vimrc</code>
* <code>mkdir bin</code>
* <code>source .bashrc</code>
* <code>apt -y install gnupg tmux htop ### Optional...</code>
* <code>apt -y install qemu-guest-agent ### Only on VMs. (Wish it worked for LXCs as well tho...)</code>
* <code>adduser '''yourname'''</code>
* <code>adduser '''yourname''' sudo</code>
* <code>su - '''yourname'''</code>
** <code>wget <nowiki>https://www.nerdmage.ca/Downloads/UserStuff/DOT.bashrc</nowiki> -O .bashrc</code>
** <code>wget <nowiki>https://www.nerdmage.ca/Downloads/UserStuff/DOT.vimrc</nowiki> -O .vimrc</code>
** <code>mkdir bin</code>
** <code>source .bashrc</code>
** <code>ssh-keygen ### Optional... But rather handy.</code>

=New System Cleanup & Prep=
Freshly installed systems usually have a bunch of stuff installed that you will never use.

For myself, I don't generally have any interest in the games. I also don't use some of the Internet apps that seem important to the developers/maintainers.

These instructions are based on running Debian with the Cinnamon Desktop Environment...
==Remove all the Games==

* <code>sudo apt purge gnome-2048 aisleriot atomix gnome-chess five-or-more hitori iagno gnome-klotski lightsoff gnome-mahjongg gnome-mines gnome-nibbles quadrapassel four-in-a-row gnome-robots gnome-sudoku swell-foop tali gnome-taquin gnome-tetravex -y</code>
* <code>sudo apt autoremove -y</code>

==Remove obsolete and/or silly Internet apps==

* <code>sudo apt purge hexchat pidgin transmission-gtk thunderbird -y</code>
* <code>sudo apt autoremove -y</code>

==Some others I don't use (from LMDE7)==

* <code>sudo apt purge matrix hypnotix rhythmbox -y</code>
* <code>sudo apt autoremove -y</code>

== Get rid of Firefox ==
* <code>sudo apt purge firefox*</code>

= Simplified installation techniques & instructions =
Some cross-platform software seems to have Linux installation instructions written by people who have never actually used Linux.

(Or, at least, people who believe in making life difficult...)

So...

* [[Installing Brave Browser]] (Properly... Not FLATPAK)
* [[Installing TP-Link Omada SDN Controller on a Debian-based LXC]]
Also, many installation instructions are long-winded or confusing...

So...

* [[Installing Docker on a Debian-based system]]

= Building/Installing things from source =
You may have noticed that the various distro repositories tend to have outdated versions of some (most) packages. While this makes perfect sense, sometimes you want the newest features & fixes. (Or maybe you just want to enable something that the repo managers figured wouldn't be useful...)

Annoyingly, it is rather common for build instructions to suck really badly.

So...

* [[Arp-Scan from Source{{!}}Arp-Scan]]
* [[FreeCAD from Source{{!}}FreeCAD]]
* [[KiCAD from Source{{!}}KiCAD]]
* [[Mosquitto from Source{{!}}Mosquitto]]

= Useful console display for Proxmox Virtual Environment =
I find it nice to have stats & such on the console of a server. This way I can just look & see what's happening with the machine.

* <code>apt update</code>
* <code>apt -y upgrade</code>
* <code>apt -y install tmux htop</code>
* <code>cd /usr/local/bin</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/PVEStuff/bin/TMUX-console</nowiki> -O TMUX-console</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/PVEStuff/bin/TM-guestlist</nowiki> -O TM-guestlist</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/PVEStuff/bin/TM-clusterstatus</nowiki> -O TM-clusterstatus</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/PVEStuff/bin/TM-systemstatus</nowiki> -O TM-systemstatus</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/PVEStuff/bin/TM-UPSstatus</nowiki> -O TM-UPSstatus</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/PVEStuff/bin/TM-FixThis</nowiki> -O TM-FixThis</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/PVEStuff/bin/TM-ShowMe</nowiki> -O TM-ShowMe</code>
* <code>chmod +x TM*</code>

You will need to edit '''TM-UPSstatus''' to match local configurations (i.e. UPS name).

Running <code>TMUX-console</code> at the console will create a formatted screen of useful system information that you can reach by attaching to the '''tmux''' session from a terminal (SSH) session.

CopyPasta

2026-04-12T20:01:00Z

Tinker: /* Remove obsolete and/or silly Internet apps */

= For doing basic setup of an LXC (or pretty much any Linux VM...) (Heck, I run these on physical machines too.): =
I run this set of commands '''as root''' on nearly every new VM or LXC I spin up. That way, I have a consistant environment to work in with all the tools I rely on.

Note: You CAN copyPasta groups of commands all at once. BUT: in this case, everything indented (after <code>su - '''yourname'''</code>) needs to be pasted AFTER that command has run.

* <code>apt update</code>
* <code>apt -y upgrade</code>
* <code>apt -y install sudo rsync vim curl mosquitto-clients</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/RootStuff/DOT.bashrc</nowiki> -O .bashrc</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/RootStuff/DOT.vimrc</nowiki> -O .vimrc</code>
* <code>mkdir bin</code>
* <code>source .bashrc</code>
* <code>apt -y install gnupg tmux htop ### Optional...</code>
* <code>apt -y install qemu-guest-agent ### Only on VMs. (Wish it worked for LXCs as well tho...)</code>
* <code>adduser '''yourname'''</code>
* <code>adduser '''yourname''' sudo</code>
* <code>su - '''yourname'''</code>
** <code>wget <nowiki>https://www.nerdmage.ca/Downloads/UserStuff/DOT.bashrc</nowiki> -O .bashrc</code>
** <code>wget <nowiki>https://www.nerdmage.ca/Downloads/UserStuff/DOT.vimrc</nowiki> -O .vimrc</code>
** <code>mkdir bin</code>
** <code>source .bashrc</code>
** <code>ssh-keygen ### Optional... But rather handy.</code>

=New System Cleanup & Prep=
Freshly installed systems usually have a bunch of stuff installed that you will never use.

For myself, I don't generally have any interest in the games. I also don't use some of the Internet apps that seem important to the developers/maintainers.

These instructions are based on running Debian with the Cinnamon Desktop Environment...
==Remove all the Games==

* <code>sudo apt purge gnome-2048 aisleriot atomix gnome-chess five-or-more hitori iagno gnome-klotski lightsoff gnome-mahjongg gnome-mines gnome-nibbles quadrapassel four-in-a-row gnome-robots gnome-sudoku swell-foop tali gnome-taquin gnome-tetravex -y</code>
* <code>sudo apt autoremove -y</code>

==Remove obsolete and/or silly Internet apps==

* <code>sudo apt purge hexchat pidgin transmission-gtk thunderbird -y</code>
* <code>sudo apt autoremove -y</code>

==Some others I don't use (from LMDE7)==

* <code>sudo apt purge matrix hypnotix rhythmbox -y
* <code>sudo apt autoremove -y</code>

== Get rid of Firefox ==
* <code>sudo apt purge firefox*</code>

= Simplified installation techniques & instructions =
Some cross-platform software seems to have Linux installation instructions written by people who have never actually used Linux.

(Or, at least, people who believe in making life difficult...)

So...

* [[Installing Brave Browser]] (Properly... Not FLATPAK)
* [[Installing TP-Link Omada SDN Controller on a Debian-based LXC]]
Also, many installation instructions are long-winded or confusing...

So...

* [[Installing Docker on a Debian-based system]]

= Building/Installing things from source =
You may have noticed that the various distro repositories tend to have outdated versions of some (most) packages. While this makes perfect sense, sometimes you want the newest features & fixes. (Or maybe you just want to enable something that the repo managers figured wouldn't be useful...)

Annoyingly, it is rather common for build instructions to suck really badly.

So...

* [[Arp-Scan from Source{{!}}Arp-Scan]]
* [[FreeCAD from Source{{!}}FreeCAD]]
* [[KiCAD from Source{{!}}KiCAD]]
* [[Mosquitto from Source{{!}}Mosquitto]]

= Useful console display for Proxmox Virtual Environment =
I find it nice to have stats & such on the console of a server. This way I can just look & see what's happening with the machine.

* <code>apt update</code>
* <code>apt -y upgrade</code>
* <code>apt -y install tmux htop</code>
* <code>cd /usr/local/bin</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/PVEStuff/bin/TMUX-console</nowiki> -O TMUX-console</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/PVEStuff/bin/TM-guestlist</nowiki> -O TM-guestlist</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/PVEStuff/bin/TM-clusterstatus</nowiki> -O TM-clusterstatus</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/PVEStuff/bin/TM-systemstatus</nowiki> -O TM-systemstatus</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/PVEStuff/bin/TM-UPSstatus</nowiki> -O TM-UPSstatus</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/PVEStuff/bin/TM-FixThis</nowiki> -O TM-FixThis</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/PVEStuff/bin/TM-ShowMe</nowiki> -O TM-ShowMe</code>
* <code>chmod +x TM*</code>

You will need to edit '''TM-UPSstatus''' to match local configurations (i.e. UPS name).

Running <code>TMUX-console</code> at the console will create a formatted screen of useful system information that you can reach by attaching to the '''tmux''' session from a terminal (SSH) session.

PVE Tips

2026-03-29T19:40:09Z

Tinker: /* Additional things to add to PVE */

= xterm.js in place of noVNC =
So...

An issue with noVNC...

It kinda sucks for CLI-only VMs. It doesn't allow for copy-pasta.

Solution, [https://silicon.blog/2023/01/12/how-to-enable-copy-and-paste-function-on-your-proxmox-web-console-without-install-additional-software-in-your-vm/ configure the VM to use xterm.js] instead.

The grub edits in that article don't seem to work properly tho...

Instead:

<pre>GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,115200 earlyprintk=ttyS0,115200 consoleblank=0"
GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --speed=115200"</pre>

(initially from [https://www.reddit.com/r/Proxmox/comments/15dhkh7/newbie_issue_unable_to_enable_xtermjs_in_debian/ here])

For some reason, Proxmox leaves noVNC as the console in the UI window & it does weird stuff to the console window when you open it. MRIN...

[https://forum.proxmox.com/threads/support-for-xterm-js-console-by-default-for-specific-vm.114734/ FFS...]

To open the console of a VM set up this way, either click the "'''>_ Console'''" button or right-click the VM in the left pane & select "'''>_ Console'''" from the menu.

= Manually installing REAL certs =

I keep copies of my certs on a local [[Building Web Servers{{!}}webserver]] that acts as a [[Proxy Server Notes{{!}}proxy]] to feed HTTPS connections to various local servers. This makes it fairly trivial to pull certs to various machines that need them.  Also, I'm using wildcard certs, which simplifies life.

'''Step 1:''' Log into the servers shell as root. (either through the Web UI or SSH)

'''Step 2:''' Follow along with this list of commands (adjusting appropriately...)

* <code>cd /etc/pve/nodes/</code>
* <code>cd '''NODENAME'''</code>
* <code>cp pveproxy-ssl.key pveproxy-ssl.key.BAK</code>
* <code>cp pveproxy-ssl.pem pveproxy-ssl.pem.BAK</code>
* <code>scp '''USER@WEBSERVER''':/etc/apache2/certs/fullchain_'''YOURDOMAIN.TLD'''.crt pveproxy-ssl.pem</code>
* <code>scp '''USER@WEBSERVER''':/etc/apache2/certs/'''YOURDOMAIN.TLD'''.key pveproxy-ssl.key</code>
* <code>systemctl restart pveproxy</code>
Note: "/etc/apache2/certs" must be readable by '''USER''' & '''USER''' must also have SSH access to '''WEBSERVER'''

'''Step 3:''' Enjoy browsing to the Web UI of your server without being bitched at by your browser.

This will also work just fine with machine-specific certs.  You'll need to adjust the filenames of your '''crt''' & '''key''' files.

(these instructions are currently based on a non-clustered server...)

= Convert a physical computer to a proxmox VM =

* [https://pve.proxmox.com/wiki/Advanced_Migration_Techniques_to_Proxmox_VE From the Proxmox documentation]
* [https://forum.proxmox.com/threads/convert-physical-computer-to-proxmox.124490/ From the Proxmox forums]

= Want a GUI on the console screen? =

* <code>apt update</code>
* <code>apt upgrade</code>
* <code>apt install task-cinnamon-desktop</code>

== Take this further... ==
* Maybe add in Synergy
* [[CopyPasta#New System_Cleanup_&_Prep{{!}}Cleanup the bloat]]?
* Set up the screensaver to work as a clock (I do this on my crash-test server...)

= Remove Proxmox Subscription Notice (Tested to 8.4) =

Seems to work on 9.1.6 also...

[https://mclarendatasystems.com/remove-proxmox51-subscription-notice/ Source]

* <code>sed -Ezi.bak "s/(function ?$orig_cmd$ \{)/\1\n\torig_cmd;\n\treturn;/g" /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js && systemctl restart pveproxy.service</code>

= Additional things to add to PVE =
* [[TailScale{{!}}TailScale]]
* Automated Cert updates
* NFS mount utilities & stuff...

PVE Tips

2026-03-29T17:02:06Z

Tinker: /* Additional things to add to PVE */

PVE Tips

2026-03-29T16:27:07Z

Tinker: /* Remove Proxmox Subscription Notice (Tested to 8.4) */

PVE Tips

2026-03-29T15:46:01Z

Tinker: /* Take this further... */

PVE Tips

2026-03-29T15:43:56Z

Tinker: /* Remove Proxmox Subscription Notice (Tested to 8.4) */

= xterm.js in place of noVNC =
So...

An issue with noVNC...

It kinda sucks for CLI-only VMs. It doesn't allow for copy-pasta.

Solution, [https://silicon.blog/2023/01/12/how-to-enable-copy-and-paste-function-on-your-proxmox-web-console-without-install-additional-software-in-your-vm/ configure the VM to use xterm.js] instead.

The grub edits in that article don't seem to work properly tho...

Instead:

<pre>GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,115200 earlyprintk=ttyS0,115200 consoleblank=0"
GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --speed=115200"</pre>

(initially from [https://www.reddit.com/r/Proxmox/comments/15dhkh7/newbie_issue_unable_to_enable_xtermjs_in_debian/ here])

For some reason, Proxmox leaves noVNC as the console in the UI window & it does weird stuff to the console window when you open it. MRIN...

[https://forum.proxmox.com/threads/support-for-xterm-js-console-by-default-for-specific-vm.114734/ FFS...]

To open the console of a VM set up this way, either click the "'''>_ Console'''" button or right-click the VM in the left pane & select "'''>_ Console'''" from the menu.

= Manually installing REAL certs =

I keep copies of my certs on a local [[Building Web Servers{{!}}webserver]] that acts as a [[Proxy Server Notes{{!}}proxy]] to feed HTTPS connections to various local servers. This makes it fairly trivial to pull certs to various machines that need them.  Also, I'm using wildcard certs, which simplifies life.

'''Step 1:''' Log into the servers shell as root. (either through the Web UI or SSH)

'''Step 2:''' Follow along with this list of commands (adjusting appropriately...)

* <code>cd /etc/pve/nodes/</code>
* <code>cd '''NODENAME'''</code>
* <code>cp pveproxy-ssl.key pveproxy-ssl.key.BAK</code>
* <code>cp pveproxy-ssl.pem pveproxy-ssl.pem.BAK</code>
* <code>scp '''USER@WEBSERVER''':/etc/apache2/certs/fullchain_'''YOURDOMAIN.TLD'''.crt pveproxy-ssl.pem</code>
* <code>scp '''USER@WEBSERVER''':/etc/apache2/certs/'''YOURDOMAIN.TLD'''.key pveproxy-ssl.key</code>
* <code>systemctl restart pveproxy</code>
Note: "/etc/apache2/certs" must be readable by '''USER''' & '''USER''' must also have SSH access to '''WEBSERVER'''

'''Step 3:''' Enjoy browsing to the Web UI of your server without being bitched at by your browser.

This will also work just fine with machine-specific certs.  You'll need to adjust the filenames of your '''crt''' & '''key''' files.

(these instructions are currently based on a non-clustered server...)

= Convert a physical computer to a proxmox VM =

* [https://pve.proxmox.com/wiki/Advanced_Migration_Techniques_to_Proxmox_VE From the Proxmox documentation]
* [https://forum.proxmox.com/threads/convert-physical-computer-to-proxmox.124490/ From the Proxmox forums]

= Want a GUI on the console screen? =

* <code>apt update</code>
* <code>apt upgrade</code>
* <code>apt install task-cinnamon-desktop</code>

== Take this further... ==
* Maybe add in Synergy
* [[CopyPasta#New System_Cleanup_&_Prep{{!}}Cleanup the bloat]]?

= Remove Proxmox Subscription Notice (Tested to 8.4) =

Seems to work on 9.1.6 also...

[https://mclarendatasystems.com/remove-proxmox51-subscription-notice/ Source]

* <code>sed -Ezi.bak "s/(function ?$orig_cmd$ \{)/\1\n\torig_cmd;\n\treturn;/g" /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js && systemctl restart pveproxy.service</code>

PVE Tips

2026-03-29T15:33:37Z

Tinker: /* Take this further... */

= xterm.js in place of noVNC =
So...

An issue with noVNC...

It kinda sucks for CLI-only VMs. It doesn't allow for copy-pasta.

Solution, [https://silicon.blog/2023/01/12/how-to-enable-copy-and-paste-function-on-your-proxmox-web-console-without-install-additional-software-in-your-vm/ configure the VM to use xterm.js] instead.

The grub edits in that article don't seem to work properly tho...

Instead:

<pre>GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,115200 earlyprintk=ttyS0,115200 consoleblank=0"
GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --speed=115200"</pre>

(initially from [https://www.reddit.com/r/Proxmox/comments/15dhkh7/newbie_issue_unable_to_enable_xtermjs_in_debian/ here])

For some reason, Proxmox leaves noVNC as the console in the UI window & it does weird stuff to the console window when you open it. MRIN...

[https://forum.proxmox.com/threads/support-for-xterm-js-console-by-default-for-specific-vm.114734/ FFS...]

To open the console of a VM set up this way, either click the "'''>_ Console'''" button or right-click the VM in the left pane & select "'''>_ Console'''" from the menu.

= Manually installing REAL certs =

I keep copies of my certs on a local [[Building Web Servers{{!}}webserver]] that acts as a [[Proxy Server Notes{{!}}proxy]] to feed HTTPS connections to various local servers. This makes it fairly trivial to pull certs to various machines that need them.  Also, I'm using wildcard certs, which simplifies life.

'''Step 1:''' Log into the servers shell as root. (either through the Web UI or SSH)

'''Step 2:''' Follow along with this list of commands (adjusting appropriately...)

* <code>cd /etc/pve/nodes/</code>
* <code>cd '''NODENAME'''</code>
* <code>cp pveproxy-ssl.key pveproxy-ssl.key.BAK</code>
* <code>cp pveproxy-ssl.pem pveproxy-ssl.pem.BAK</code>
* <code>scp '''USER@WEBSERVER''':/etc/apache2/certs/fullchain_'''YOURDOMAIN.TLD'''.crt pveproxy-ssl.pem</code>
* <code>scp '''USER@WEBSERVER''':/etc/apache2/certs/'''YOURDOMAIN.TLD'''.key pveproxy-ssl.key</code>
* <code>systemctl restart pveproxy</code>
Note: "/etc/apache2/certs" must be readable by '''USER''' & '''USER''' must also have SSH access to '''WEBSERVER'''

'''Step 3:''' Enjoy browsing to the Web UI of your server without being bitched at by your browser.

This will also work just fine with machine-specific certs.  You'll need to adjust the filenames of your '''crt''' & '''key''' files.

(these instructions are currently based on a non-clustered server...)

= Convert a physical computer to a proxmox VM =

* [https://pve.proxmox.com/wiki/Advanced_Migration_Techniques_to_Proxmox_VE From the Proxmox documentation]
* [https://forum.proxmox.com/threads/convert-physical-computer-to-proxmox.124490/ From the Proxmox forums]

= Want a GUI on the console screen? =

* <code>apt update</code>
* <code>apt upgrade</code>
* <code>apt install task-cinnamon-desktop</code>

== Take this further... ==
* Maybe add in Synergy
* [[CopyPasta#New System_Cleanup_&_Prep{{!}}Cleanup the bloat]]?

= Remove Proxmox Subscription Notice (Tested to 8.4) =

Seems to work on 9.0.x also...

[https://mclarendatasystems.com/remove-proxmox51-subscription-notice/ Source]

* <code>sed -Ezi.bak "s/(function ?$orig_cmd$ \{)/\1\n\torig_cmd;\n\treturn;/g" /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js && systemctl restart pveproxy.service</code>

PVE Tips

2026-03-29T15:32:19Z

Tinker: /* Want a GUI on the console screen? */

= xterm.js in place of noVNC =
So...

An issue with noVNC...

It kinda sucks for CLI-only VMs. It doesn't allow for copy-pasta.

Solution, [https://silicon.blog/2023/01/12/how-to-enable-copy-and-paste-function-on-your-proxmox-web-console-without-install-additional-software-in-your-vm/ configure the VM to use xterm.js] instead.

The grub edits in that article don't seem to work properly tho...

Instead:

<pre>GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,115200 earlyprintk=ttyS0,115200 consoleblank=0"
GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --speed=115200"</pre>

(initially from [https://www.reddit.com/r/Proxmox/comments/15dhkh7/newbie_issue_unable_to_enable_xtermjs_in_debian/ here])

For some reason, Proxmox leaves noVNC as the console in the UI window & it does weird stuff to the console window when you open it. MRIN...

[https://forum.proxmox.com/threads/support-for-xterm-js-console-by-default-for-specific-vm.114734/ FFS...]

To open the console of a VM set up this way, either click the "'''>_ Console'''" button or right-click the VM in the left pane & select "'''>_ Console'''" from the menu.

= Manually installing REAL certs =

I keep copies of my certs on a local [[Building Web Servers{{!}}webserver]] that acts as a [[Proxy Server Notes{{!}}proxy]] to feed HTTPS connections to various local servers. This makes it fairly trivial to pull certs to various machines that need them.  Also, I'm using wildcard certs, which simplifies life.

'''Step 1:''' Log into the servers shell as root. (either through the Web UI or SSH)

'''Step 2:''' Follow along with this list of commands (adjusting appropriately...)

* <code>cd /etc/pve/nodes/</code>
* <code>cd '''NODENAME'''</code>
* <code>cp pveproxy-ssl.key pveproxy-ssl.key.BAK</code>
* <code>cp pveproxy-ssl.pem pveproxy-ssl.pem.BAK</code>
* <code>scp '''USER@WEBSERVER''':/etc/apache2/certs/fullchain_'''YOURDOMAIN.TLD'''.crt pveproxy-ssl.pem</code>
* <code>scp '''USER@WEBSERVER''':/etc/apache2/certs/'''YOURDOMAIN.TLD'''.key pveproxy-ssl.key</code>
* <code>systemctl restart pveproxy</code>
Note: "/etc/apache2/certs" must be readable by '''USER''' & '''USER''' must also have SSH access to '''WEBSERVER'''

'''Step 3:''' Enjoy browsing to the Web UI of your server without being bitched at by your browser.

This will also work just fine with machine-specific certs.  You'll need to adjust the filenames of your '''crt''' & '''key''' files.

(these instructions are currently based on a non-clustered server...)

= Convert a physical computer to a proxmox VM =

* [https://pve.proxmox.com/wiki/Advanced_Migration_Techniques_to_Proxmox_VE From the Proxmox documentation]
* [https://forum.proxmox.com/threads/convert-physical-computer-to-proxmox.124490/ From the Proxmox forums]

= Want a GUI on the console screen? =

* <code>apt update</code>
* <code>apt upgrade</code>
* <code>apt install task-cinnamon-desktop</code>

== Take this further... ==
* Maybe add in Synergy
* Cleanup the bloat?

= Remove Proxmox Subscription Notice (Tested to 8.4) =

Seems to work on 9.0.x also...

[https://mclarendatasystems.com/remove-proxmox51-subscription-notice/ Source]

* <code>sed -Ezi.bak "s/(function ?$orig_cmd$ \{)/\1\n\torig_cmd;\n\treturn;/g" /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js && systemctl restart pveproxy.service</code>

PVE Tips

2026-03-29T15:25:08Z

Tinker: /* Want a GUI on the console screen? */

= xterm.js in place of noVNC =
So...

An issue with noVNC...

It kinda sucks for CLI-only VMs. It doesn't allow for copy-pasta.

Solution, [https://silicon.blog/2023/01/12/how-to-enable-copy-and-paste-function-on-your-proxmox-web-console-without-install-additional-software-in-your-vm/ configure the VM to use xterm.js] instead.

The grub edits in that article don't seem to work properly tho...

Instead:

<pre>GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,115200 earlyprintk=ttyS0,115200 consoleblank=0"
GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --speed=115200"</pre>

(initially from [https://www.reddit.com/r/Proxmox/comments/15dhkh7/newbie_issue_unable_to_enable_xtermjs_in_debian/ here])

For some reason, Proxmox leaves noVNC as the console in the UI window & it does weird stuff to the console window when you open it. MRIN...

[https://forum.proxmox.com/threads/support-for-xterm-js-console-by-default-for-specific-vm.114734/ FFS...]

To open the console of a VM set up this way, either click the "'''>_ Console'''" button or right-click the VM in the left pane & select "'''>_ Console'''" from the menu.

= Manually installing REAL certs =

I keep copies of my certs on a local [[Building Web Servers{{!}}webserver]] that acts as a [[Proxy Server Notes{{!}}proxy]] to feed HTTPS connections to various local servers. This makes it fairly trivial to pull certs to various machines that need them.  Also, I'm using wildcard certs, which simplifies life.

'''Step 1:''' Log into the servers shell as root. (either through the Web UI or SSH)

'''Step 2:''' Follow along with this list of commands (adjusting appropriately...)

* <code>cd /etc/pve/nodes/</code>
* <code>cd '''NODENAME'''</code>
* <code>cp pveproxy-ssl.key pveproxy-ssl.key.BAK</code>
* <code>cp pveproxy-ssl.pem pveproxy-ssl.pem.BAK</code>
* <code>scp '''USER@WEBSERVER''':/etc/apache2/certs/fullchain_'''YOURDOMAIN.TLD'''.crt pveproxy-ssl.pem</code>
* <code>scp '''USER@WEBSERVER''':/etc/apache2/certs/'''YOURDOMAIN.TLD'''.key pveproxy-ssl.key</code>
* <code>systemctl restart pveproxy</code>
Note: "/etc/apache2/certs" must be readable by '''USER''' & '''USER''' must also have SSH access to '''WEBSERVER'''

'''Step 3:''' Enjoy browsing to the Web UI of your server without being bitched at by your browser.

This will also work just fine with machine-specific certs.  You'll need to adjust the filenames of your '''crt''' & '''key''' files.

(these instructions are currently based on a non-clustered server...)

= Convert a physical computer to a proxmox VM =

* [https://pve.proxmox.com/wiki/Advanced_Migration_Techniques_to_Proxmox_VE From the Proxmox documentation]
* [https://forum.proxmox.com/threads/convert-physical-computer-to-proxmox.124490/ From the Proxmox forums]

= Want a GUI on the console screen? =

* <code>apt update</code>
* <code>apt upgrade</code>
* <code>apt install task-cinnamon-desktop</code>

== Maybe add in Synergy ==

= Remove Proxmox Subscription Notice (Tested to 8.4) =

Seems to work on 9.0.x also...

[https://mclarendatasystems.com/remove-proxmox51-subscription-notice/ Source]

* <code>sed -Ezi.bak "s/(function ?$orig_cmd$ \{)/\1\n\torig_cmd;\n\treturn;/g" /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js && systemctl restart pveproxy.service</code>

WeeWX Weather Server

2026-03-09T00:45:02Z

Tinker: /* Setting up for EcoWitt gear */

{{{!}} class="wikitable" style="float: right; width: 322px;" border="2"
{{!}}+ Proven on:
{{!}}- < --="" debian="" -->
{{!}} style="text-align: center; width: 60px;" {{!}} [[File:Logo Debian.png{{!}}60px{{!}}link=https://www.debian.org/{{!}}center{{!}}middle{{!}}frameless]]
{{!}} style="text-align: center; width: 40px;" {{!}} 12 (bullseye)
{{!}} Actually, a pretty much all default LXC running under Proxmox. Only non-default item being that I gave it 4 cores.
{{!}}}

This particular set of instructions is based on building a Debian 12 based LXC under Proxmox to handle WeeWX. On my network, I have an existing WebServer & an existing Automation Server. The intent is to have the weather information served up as a web page (Default WeeWX thing...) by the WebServer and also be used by the Automation Server.

There is also a dedicated Database Server which we will be using. (can't possibly do things the easy way...)

(Based on: [https://www.weewx.com/docs/5.1/quickstarts/debian/ WeeWX: Installation on Debian-based systems])

Originally, the old version of this article was done with a much earlier version of WeeWX & our old Accurite weather station. Things have seriously changed since those days...

== Configure apt ==
* <code>sudo apt install -y wget gnupg</code>
* <code>wget -qO - <nowiki>https://weewx.com/keys.html</nowiki> | sudo gpg --dearmor --output /etc/apt/trusted.gpg.d/weewx.gpg</code>
* <code>echo "deb [arch=all] <nowiki>https://weewx.com/apt/python3</nowiki> buster main" | sudo tee /etc/apt/sources.list.d/weewx.list</code>

== Install WeeWX ==

* <code>sudo apt update</code>
* <code>sudo apt install weewx</code>
During this step, you will be asked for all the details about your install.

Fill everything in as best you can, but you can indeed [[#OTHER CUSTOMISATIONS{{!}}change things later]] too.

If you are using weather station hardware that is supported by default, go ahead and select that hardware when asked. In my case, I'm currently running an EcoWitt weather station & that will require a little more effort. For now, I'm selecting '''Simulator''' as my hardware type.

Once the machine has finished churning away at the install, WeeWX will be running. You can verify this by running:

* <code>sudo systemctl status weewx</code>

== Setting up for EcoWitt gear ==
You'll be installing the [https://github.com/hoetzgit/weewx-gw1000 '''Ecowitt Gateway (formerly GW1000) Driver''']

(Something happened to the original developer & all his stuff has vanished from the Interwebs...)

* <code>wget -O weewx-gw1000.zip <nowiki>https://github.com/hoetzgit/weewx-gw1000/archive/master.zip</nowiki></code>
* <code>sudo apt install python3-six</code>
* <code>sudo weectl extension install weewx-gw1000.zip</code>
Verify it can talk to your EcoWitt gateway:

*<code>PYTHONPATH=/usr/share/weewx python3 /etc/weewx/bin/user/gw1000.py --test-driver --ip-address='''device_ip_address'''</code>
(Where you replace '''device_ip_address''' with the actual address of your gateway...)

You should observe loop packets being emitted on a regular basis. Once finished press ctrl-c to exit.

'''Note''': You will only see loop packets and not archive records when running the driver directly. This is because you are seeing output not from WeeWX, but rather directly from the driver.

Configure WeeWX to use the driver:

*<code>sudo weectl station reconfigure --driver=user.gw1000</code>
restart WeeWx

*<code>sudo systemctl restart weewx</code>
('''Note''': I've verified this with '''GW1100''', '''GW1200 and GW3000''' gateways so far...)

== Customising WeeWX ==

=== Broadcasting over MQTT ===
Install paho-mqtt

* <code>sudo apt -y install python3-paho-mqtt</code>

Download the WeeWX MQTT Extension:

* <code>wget -O weewx-mqtt.zip <nowiki>https://github.com/matthewwall/weewx-mqtt/archive/master.zip</nowiki></code>

Install the extension:

* <code>sudo weectl extension install weewx-mqtt.zip</code>

Then edit weewx.conf to configure things for your own MQTT server

* <code>sudo vim /etc/weewx/weewx.conf</code>

Scroll down & find the '''<nowiki>[[MQTT]]</nowiki>''' block, then edit it to suit your network.

<nowiki>[[MQTT]]</nowiki>
server_url = mqtt://'''Machine.Domain.TLD''':1883
topic = '''weather'''
unit_system = METRIC

restart WeeWx

* <code>sudo systemctl restart weewx</code>

=== Connecting to a remote database ===
By default, WeeWX uses a local instance of SQLite.

But, you may wish to archive its data on your primary database.

(Maybe even go nuts & set up multiple weather stations, all sending their data to a central database server...)

==== Setting up the database (MySQL / MariaDB) ====
On the Database Server:

(NOTE: Your Database Server must be configured to allow remote access.)

*<code>sudo mysql -u root -p</code>

CREATE USER 'weewx'@''''Machine.Domain.TLD'''' IDENTIFIED BY 'weewx';
CREATE DATABASE weewx;
GRANT select, update, create, delete, insert, drop ON weewx.* TO weewx@''''Machine.Domain.TLD'''';

('''Machine.Domain.TLD''', of course, being the address of your WeeWX Server...)

==== Configuring WeeWX to use this database ====
*<code>sudo vi /etc/weewx/weewx.conf</code>

in <nowiki>[[wx_binding]]</nowiki>

change

<code>database = archive_sqlite</code>

to

<code>database = archive_mysql</code>

in <nowiki>[[MySQL]]</nowiki>

change

<code>host = localhost</code>

to

<code>host = '''Machine.Domain.TLD'''</code>

('''Machine.Domain.TLD''', of course, being the address of your Database Server...)

restart WeeWx

* <code>sudo systemctl restart weewx</code>

=== Making the weather visible via Web Browser ===
There are 3 options for locally serving up the weather as HTML...

==== 1: Install a web server on the same machine (or VM...or LXC...) ====

Install your choice of web server and simply point the root at '''/var/www/html/weewx'''

==== 2: Use FTP to upload to your existing WebServer ====

(Your Web Server will need to have FTP enabled...)

*<code>sudo vi /etc/weewx/weewx.conf</code>

Find the <nowiki>[[FTP]]</nowiki> block & edit it to suit.

<nowiki>[StdReport]</nowiki>
<nowiki>[[FTP]]</nowiki>
skin = Ftp
enable = true
server = '''Machine.Domain.TLD'''
path = /var/www/html/weewx
user = '''USERNAME'''
password = '''PASSWORD'''

==== 3: Use RSYNC to upload to your existing WebServer ====

(You will need to set up passwordless SSH between this machine & your web server...)

*<code>sudo vi /etc/weewx/weewx.conf</code>

Find the <nowiki>[[RSYNC]]</nowiki> block & edit it to suit.

<nowiki>[StdReport]</nowiki>
<nowiki>[[RSYNC]]</nowiki>
skin = Rsync
enable = true
server = '''Machine.Domain.TLD'''
path = /var/www/html/weewx
user = '''USERNAME'''

(Make sure your chosen user has write permissions...)

=== OTHER CUSTOMISATIONS ===
There is a configuration file at:
* <code>/etc/weewx/weewx.conf</code>
that holds most (if not all) of the little details & tweaks for your weather server.

==== Useful examples ====
'''archive_interval''' defines how often WeeWX updates its reports (Including via MQTT if you're using that...)

==Administration==
Checking status:
* <code>sudo systemctl status weewx</code>

Starting:
* <code>sudo systemctl start weewx</code>

Stopping:
* <code>sudo systemctl stop weewx</code>

Restarting:
* <code>sudo systemctl restart weewx</code>

Watching the system log
* <code>sudo journalctl -u weewx</code>

* <code></code>

Automation - Home-Assistant

2026-03-04T21:04:39Z

Tinker: /* (Their instructions...) */

Automation - Home-Assistant

2026-03-04T21:04:20Z

Tinker: /* (Their instructions...) */

Automation - Home-Assistant

2026-03-04T20:01:07Z

Tinker: /* (Their instructions...) */

Automation - Home-Assistant

2026-03-04T19:57:44Z

Tinker: /* (Their instructions...) */

Automation - Home-Assistant

2026-03-04T19:50:57Z

Tinker: /* Installation */

Automation - Home-Assistant

2026-03-04T19:49:42Z

Tinker: /* ccckkk */

= Installation =
While it would be rather nice to be able to simply install Home Assistant straight onto a LXC, it seems that the maintainers feel otherwise.

The only straightforward install that actually seems to work is installing their own version of Linux onto either bare metal or a VM.

As a bonus silly point, rather than just setting up a VM as usual, it is recommended that you run a community-supplied script on your Proxmox host to build the VM.
== fffuuu ==

*Run this command in the Proxmox shell:<pre class="code svelte-x57wqa" data-lang="bash"><code class="svelte-x57wqa">bash -c "$(wget -qLO - https://github.com/community-scripts/ProxmoxVE/raw/main/vm/haos-vm.sh)"</code></pre><div class="copy-container svelte-clssou"> </div>
*Follow the on-screen prompts. Use default settings unless you need custom configurations (e.g., network bridge, VLAN).
*The script downloads the latest HAOS qcow2 image, creates the VM, imports the disk, configures UEFI, and starts the VM.
*Once complete, check the VM’s Summary tab in the Proxmox GUI to find the IP address assigned by DHCP.
*Access Home Assistant at <code class="svelte-x57wqa">http://[VM_IP]:8123</code> to complete setup. 

== ccckkk ==
* Run this command in the Proxmox shell:
<pre class="code" data-lang="bash">
bash -c "$(wget -qLO - https://github.com/community-scripts/ProxmoxVE/raw/main/vm/haos-vm.sh)"
</pre>
* Follow the on-screen prompts. Use default settings unless you need custom configurations (e.g., network bridge, VLAN).
* The script downloads the latest HAOS qcow2 image, creates the VM, imports the disk, configures UEFI, and starts the VM.
* Once complete, check the VM’s Summary tab in the Proxmox GUI to find the IP address assigned by DHCP.
* Access Home Assistant at <code >http://'''[VM_IP]''':8123</code> to complete setup.

Automation - Home-Assistant

2026-03-04T19:44:31Z

Tinker: /* ccckkk */

= Installation =
While it would be rather nice to be able to simply install Home Assistant straight onto a LXC, it seems that the maintainers feel otherwise.

The only straightforward install that actually seems to work is installing their own version of Linux onto either bare metal or a VM.

As a bonus silly point, rather than just setting up a VM as usual, it is recommended that you run a community-supplied script on your Proxmox host to build the VM.
== fffuuu ==

*Run this command in the Proxmox shell:<pre class="code svelte-x57wqa" data-lang="bash"><code class="svelte-x57wqa">bash -c "$(wget -qLO - https://github.com/community-scripts/ProxmoxVE/raw/main/vm/haos-vm.sh)"</code></pre><div class="copy-container svelte-clssou"> </div>
*Follow the on-screen prompts. Use default settings unless you need custom configurations (e.g., network bridge, VLAN).
*The script downloads the latest HAOS qcow2 image, creates the VM, imports the disk, configures UEFI, and starts the VM.
*Once complete, check the VM’s Summary tab in the Proxmox GUI to find the IP address assigned by DHCP.
*Access Home Assistant at <code class="svelte-x57wqa">http://[VM_IP]:8123</code> to complete setup. 

== ccckkk ==
* Run this command in the Proxmox shell:
<pre class="code svelte-x57wqa" data-lang="bash"><code >bash -c "$(wget -qLO - https://github.com/community-scripts/ProxmoxVE/raw/main/vm/haos-vm.sh)"</code></pre><div class="copy-container svelte-clssou"> </div>
* Follow the on-screen prompts. Use default settings unless you need custom configurations (e.g., network bridge, VLAN).
* The script downloads the latest HAOS qcow2 image, creates the VM, imports the disk, configures UEFI, and starts the VM.
* Once complete, check the VM’s Summary tab in the Proxmox GUI to find the IP address assigned by DHCP.
* Access Home Assistant at <code >http://[VM_IP]:8123</code> to complete setup.

Automation - Home-Assistant

2026-03-04T19:40:54Z

Tinker:

Automation - Home-Assistant

2026-03-04T19:35:35Z

Tinker:

Automation - Home-Assistant

2026-03-04T19:32:49Z

Tinker: Created page with "fffff"

fffff

Installing TP-Link Omada SDN Controller on a Debian-based LXC

2026-01-18T21:31:26Z

Tinker: /* Restarting the Omada Controller from CLI */

Installing TP-Link Omada SDN Controller on a Debian-based LXC

2026-01-18T21:26:11Z

Tinker: /* Restarting the Omada Controller from CLI */

Installing TP-Link Omada SDN Controller on a Debian-based LXC

2026-01-18T21:22:39Z

Tinker: /* Restarting the Omada Controller from CLI */

Installing TP-Link Omada SDN Controller on a Debian-based LXC

2026-01-18T21:22:28Z

Tinker: /* Notes & tips */

Installing TP-Link Omada SDN Controller on a Debian-based LXC

2026-01-18T21:20:22Z

Tinker: /* Problem(s) */

Installing TP-Link Omada SDN Controller on a Debian-based LXC

2026-01-18T21:19:18Z

Tinker: /* CERTS!!! */

Automation - Node-Red

2025-11-23T02:18:51Z

Tinker: /* Another Note: If you find that you've outgrown Node-Reds available environment space... */

{{{!}} class="wikitable" style="float: right; width: 322px;" border="2"
{{!}}+ Proven on:
{{!}}- --="" debian=""
{{!}} style="text-align: center; width: 60px;" {{!}} [[File:Logo Debian.png{{!}}60px{{!}}link=https://www.debian.org/{{!}}center{{!}}middle{{!}}frameless]]
{{!}} style="text-align: center; width: 40px;" {{!}} 12 (bookworm)
{{!}}

 
{{!}}- --="" lmde=""
{{!}} style="text-align: center; width: 65.675px;" {{!}} [[File:Logo LMDE.png{{!}}60px{{!}}link=https://linuxmint.com/edition.php?id=279{{!}}center{{!}}middle{{!}}frameless]]
{{!}} 4
{{!}}

 
{{!}}- --="" linux="" mint=""
{{!}} style="text-align: center; width: 67.5125px;" {{!}} [[File:Logo Mint.png{{!}}60px{{!}}link=https://linuxmint.com{{!}}center{{!}}middle{{!}}frameless]]
{{!}} 19.3 / 20.3
{{!}}

 
{{!}}- --="" ubuntu=""
{{!}} style="text-align: center; width: 65.7px;" {{!}} [[File:Logo Ubuntu.png{{!}}60px{{!}}link=https://ubuntu.com/{{!}}center{{!}}middle{{!}}frameless]]
{{!}} 20.04.3
{{!}} issues
 
{{!}}- --="" sparky="" linux=""
{{!}} style="text-align: center; width: 65.7px;" {{!}} [[File:Logo Sparky.png{{!}}60px{{!}}link=https://sparkylinux.org/{{!}}center{{!}}middle{{!}}frameless]]
{{!}} 5.11
{{!}}

 
{{!}}}
[https://nodered.org/ Node-Red Web Site]

'''As always...'''

Start with:

*<code>sudo apt update</code>
*<code>sudo apt upgrade</code>

== Installing Node-Red ==
=== Start by installing nodejs and the javascript package manager (npm) ===

'''FFFfffuuuuuu...'''

nodejs & npm versions are problematic.

For some reason, the versions available in all of the repositories tend to be out of date AND all attempts to update them fail in weird & wonderful ways.

[[Getting up-to-date NodeJS & NPM{{!}}I have found a procedure that seems to work]]

=== Use npm to install Node-RED ===

* <code>sudo npm install -g node-red</code>
[[File:Node-red install output (2022-05-14).png{{!}}300px{{!}}left{{!}}thumb{{!}}Latest WTF?]]
<div style="clear: both"></div>

=== Run it... ===

* <code>node-red</code>

=== Adding Autostart capability using SystemD ===
==== Then download three required files to their correct locations ====

* <code>sudo wget <nowiki>https://raw.githubusercontent.com/node-red/raspbian-deb-package/master/resources/nodered.service</nowiki> -O /lib/systemd/system/nodered.service</code>
* <code>sudo wget <nowiki>https://raw.githubusercontent.com/node-red/raspbian-deb-package/master/resources/node-red-start</nowiki> -O /usr/bin/node-red-start</code>
* <code>sudo wget <nowiki>https://raw.githubusercontent.com/node-red/raspbian-deb-package/master/resources/node-red-stop</nowiki> -O /usr/bin/node-red-stop</code>

===== '''Note:''' These 3 files are kinda rpi specific... =====

To run as a user other than Pi, you need to edit the nodered.service file. To edit this use sudo to edit the file <code>/lib/systemd/system/nodered.service</code> and change the lines 11, 12 and 13.

*<code>sudo vi /lib/systemd/system/nodered.service</code>

Replace the user name "pi" with a user on your actual system...

===== '''Another Note:''' If you find that you've outgrown Node-Reds available environment space... =====

'''i.e.:''' it starts crashing with a message like: <code>'''FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory'''</code> in the logs...

You can try editing line 16 of <code>/lib/systemd/system/nodered.service</code> and change the number 256 to something larger. (within as-yet undetermined limits...)

Apparently, another symptom you need to do this is when you start randomly losing connection to the dashboard.

==== make the two scripts executable ====

* <code>sudo chmod +x /usr/bin/node-red-st*</code>

==== reload the systemd daemon ====

* <code>sudo systemctl daemon-reload</code>

==== enable Node-RED to run automatically ====

* <code>sudo systemctl enable nodered.service</code>

==== Start Node-RED ====

* <code>sudo systemctl start nodered.service</code>

== Securing Node-RED ==

* [https://nodered.org/docs/security#usernamepassword-based-authentication Username/password based authentication]
** The suggested<code>'''adminAuth''' section is already there... edit it</code>
** <code>You can create multiple users by duplicating the user object inside '''users: []'''</code>
* [https://nodered.org/docs/node-red-admin Command-line Administration](how to get '''node-red-admin''')

== Useful management stuff ==
=== Learn about the current service status ===

* <code>sudo systemctl status nodered.service</code>

=== (Re-)Start the Node-RED background service ===

* <code>sudo systemctl restart nodered.service</code>

=== Stop the Node-RED background service ===

* <code>sudo systemctl stop nodered.service</code>

=== Disable the Node-RED background service ===

* sudo systemctl disable nodered.service

=== Get the service log since the last boot ===

* <code>sudo journalctl -f -u nodered.service -b -o cat</code>

=== Also ===
Node-RED can then be started and stopped by using the commands:

* <code>node-red-start</code>
and

* <code>node-red-stop</code>

=== Updating ===
Apparently, just run the install again...

* <code>sudo npm install -g node-red</code>
(pretty simple...)

== Debugging Node-Red issues ==
=== ECONNRESET caused by MQTT ===
<code>Error: Client network socket disconnected before secure TLS connection was established Error: read ECONNRESET Error: read ECONNRESET Error: Client network socket disconnected before secure TLS connection was established</code>

Hhhmmm...

=== ECONNREFUSED caused by alexa-home-conf ===
<code>[error] [alexa-home-conf:919d2660.bd9218] Error: connect ECONNREFUSED 34.240.81.189:8883</code>

Strange response from the creator: https://github.com/hardillb/node-red-contrib-alexa-home-skill/issues/51

Doesn't actually explain what's going on, but it's allegedly harmless. My question is: Why in fuck does it want to connect to a non-functional "alternate endpoint"?


== Graceful restarts (maybe) ==
This is likely not needed when running under systemd...

[https://nodered.org/docs/getting-started/running From the Node-Red website]

You can start a script as a daemon. But first install <code>forever</code>

<code>sudo npm install -g --unsafe-perm forever</code>

Then issue command:

<code>forever start /usr/local/lib/node_modules/node-red/red.js</code>

== Things to do with Node-RED once it's installed ==

WIP:  These pages will appear as I get things updated from what's [[TinkerWiki:AutomationServer -_Node-Red#Things_to_do_with_Node-RED_once_it.27s_installed{{!}}on the older wiki]]...
* [[Node-RED_-_with_Google_Home{{!}}Node-RED with Google Home]]
* [[Node-RED_-_with_InfluxDB{{!}}Node-RED with InfluxDB]]
* [[Node-RED_-_with_Grafana{{!}}Node-RED with Grafana]]

== Some useful Node-Red tips ==

* [https://flows.nodered.org/flow/7c67709fecce1ad5992b695028e57646 Enable / Disable a Flow]
* '''Search:''' [https://www.google.com/search?lei=2ZMyXJPAD8i7jwTCvrfwCw&q=node%20red%20function%20multiple%20inputs&ved=2ahUKEwiGhrqErNrfAhXk5IMKHeUMAlAQsKwBKAJ6BAgBEAM&biw=1260&bih=652 node red function multiple inputs]
* Node-Red stores it's data in '''<code>~/.node-red</code>'''
* [https://discourse.nodered.org/t/palette-export-or-backup/18364 Palette Export or Backup]

== Some interesting Node-Red Add-Ons ==

* [https://flows.nodered.org/node/node-red-contrib-machine-learning Machine learning package for node-red]
* node-red-contrib-boolean-logic
** Create "Buffer" from "Invert"
* node-red-contrib-google-home-notify
* node-red-dashboard
* node-red-node-ping
** There seem to be other versions... possibly better...
* node-red-contrib-plex

== Moving Node-Red to a new server ==
=== When creating a NEW server ===
'''NOTE''': If you are updating to a much newer version of Node-Red, this is not likely the best technique...

Pretty easy actually...

Build the server & install everything as normal.

Then stop the node-red service and replace ~/.npm & ~/node-red with copies from the old server.

=== When Renaming a server ===
No longer seems to be an issue with current version of Node-Red...

(i.e.: Moving or cloning a VM...)

Seems to lose the flows. :(

They're not actually lost. :)

The flow files are named with the server name.

You can select a flow file in <code>~/.node-red/settings.js</code>

Uncomment the line that says: <code>//flowfile: 'flows.json',</code> and change '''flows.json''' to the name of your actual flow file. (in the same folder...)

Then restart node-red


<code>sudo systemctl restart nodered.service</code>

If you are using other services on the server (i.e.: MQTT), you may run into further weirdness like having to tell nodes where to find these services. More research still required on this...

[[Category:ServerBuilding]]
[[Category:AutomationServers]]

Database - MariaDB

2025-10-12T23:33:01Z

Tinker: /* Allowing non-localhost access */

{{{!}} class="wikitable" style="float: right; width: 322px;" border="2"
{{!}}+ Proven on:
{{!}}- --="" debian=""
{{!}} style="text-align: center; width: 60px;" {{!}} [[File:Logo Debian.png{{!}}60px{{!}}link=https://www.debian.org/{{!}}center{{!}}middle{{!}}frameless]]
{{!}} style="text-align: center; width: 40px;" {{!}} 11 (bullseye) 12 (bookworm)
{{!}}

 
{{!}}}

Here in the NerdMage's Lair, we have switched from mySQL to MariaDB lately. This has been prompted by the apparent abandonment of mySQL. (Yes, this IS a primarily Debian environment...)

'''Something to note:''' When working with MariaDB, you will occasionally see utilities & such referring to mySQL. This is normal as MariaDB is a replacement for mySQL & the developers have not recreated things that work for both. (in many cases, this means that commands you would expect to be based on '''mariadb''' will be '''mysql''' instead.)

= Installation =

* <code>sudo apt install mariadb-server mariadb-client php-mysql</code>

= Initial Configuration =

* <code>sudo mysql_secure_installation</code>

<pre style="color: rgb(22, 145, 121);">
NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY!

In order to log into MariaDB to secure it, we'll need the current
password for the root user. If you've just installed MariaDB, and
haven't set the root password yet, you should just press enter here.

Enter current password for root (enter for none):
</pre>

Press '''Enter'''

<pre style="color: rgb(22, 145, 121);">
OK, successfully used password, moving on...

Setting the root password or using the unix_socket ensures that nobody
can log into the MariaDB root user without the proper authorisation.

You already have your root account protected, so you can safely answer 'n'.

Switch to unix_socket authentication [Y/n]
</pre>

Select '''N'''

<pre style="color: rgb(22, 145, 121);">
... skipping.

You already have your root account protected, so you can safely answer 'n'.

Change the root password? [Y/n]
</pre>

Select '''N'''

<pre style="color: rgb(22, 145, 121);">
... skipping.

By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them. This is intended only for testing, and to make the installation
go a bit smoother. You should remove them before moving into a
production environment.

Remove anonymous users? [Y/n]
</pre>

Select '''Y'''

<pre style="color: rgb(22, 145, 121);">
... Success!

Normally, root should only be allowed to connect from 'localhost'. This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n]
</pre>

Select '''Y'''

<pre style="color: rgb(22, 145, 121);">
... Success!

By default, MariaDB comes with a database named 'test' that anyone can
access. This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n]
</pre>

Select '''Y'''

(Don't think I've EVER used the test database... EVER...)

<pre style="color: rgb(22, 145, 121);">
- Dropping test database...
... Success!
- Removing privileges on test database...
... Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n]
</pre>

Select '''Y'''

This is a silly question.

<pre style="color: rgb(22, 145, 121);">
... Success!

Cleaning up...

All done! If you've completed all of the above steps, your MariaDB
installation should now be secure.

Thanks for using MariaDB!
</pre>

= Allowing non-localhost access =
By default, MariaDB (like MySQL) binds to 127.0.0.1...

This is ugly, but works for now:

* <code>sudo vi /etc/mysql/mariadb.conf.d/50-server.cnf</code>
& change

<code>bind-address = 127.0.0.1</code>

to

<code>bind-address = '''Machine.Domain.TLD'''</code>

(where '''Machine.Domain.TLD''' is the FQDN or IP address of the database server itself...)

or... you could live on the edge & simply do:

<code>bind-address = 0.0.0.0</code>

= Set up at least one database user =

* <code>sudo mysql -u root</code>
<pre>
CREATE USER 'someone'@'localhost' IDENTIFIED BY 'password';
FLUSH PRIVILEGES;
EXIT;
</pre>

('''Hint:''' This'd be a good time to create yourself as that user with your non-admin password of choice...)

('''BONUS Hint:''' Change '''''localhost''''' in that CREATE USER line with ''''''%'''''' if this will be a remotely accessible server...)
== Remotely accessed Database Server ==
If this is to be a remotely accessed Database Server, ''''localhost'''' will actually be the name/address of the remote machine you'll access '''from'''.

Possibly even ''''%'''' as a wildcard.

'''A NOTE:''' If you have a user @ a specific host, then add the same user @ '%', MariaDB will not let the wildcard user have priveleges...

= Some Reference Materials =

*[https://www.w3schools.com/sql/default.asp SQL Tutorial]
*[http://g2pc1.bu.edu/~qzpeng/manual/MySQL%20Commands.htm Commands]
*[https://www.freecodecamp.org/news/basic-sql-commands/ Basic SQL Commands - The List of Database Queries and Statements You Should Know]
*[https://linuxize.com/post/how-to-manage-mysql-databases-and-users-from-the-command-line/ How to Manage MySQL Databases and Users from the Command Line]
*[https://www.hostinger.com/tutorials/?s=mysql Some handy tutorials]
**[https://www.hostinger.com/tutorials/mysql-show-users/ How to Show Users in MySQL on Linux]

= Some Useful Things =
(These have yet to be fully tested on '''MariaDB''' as opposed to '''mySQL'''...)

*Sign into MySQL
**<code>sudo mysql -u root -p</code>
*Restart MySQL
**<code>sudo systemctl restart mariadb.service</code>
*Change a user password
**<code>ALTER USER 'userName'@'localhost' IDENTIFIED BY 'New-Password-Here';</code>
*Create a new database
**<code>CREATE DATABASE db_name;</code>
***You will need to grant access to any new database to the appropriate user(s). This is covered more clearly on pages where you create specific databases...
***<code>GRANT ALL ON db_name.* TO 'user'@'localhost';</code>
***(as one example)
*Trash a complete database '''(WARNING! DANGER! DAMAGE LIKELY!)'''
**<code>DROP DATABASE wp_Someblog;</code>
*List existing databases
**<code>SHOW databases;</code>
*List existing users
**<code>SELECT user,host FROM mysql.user;</code>

== Move a database from one server to another ==

=== On the originating server: ===

*<code>sudo mysql -u root -p</code>
**<code>GRANT ALL ON *.* TO 'user'@'localhost';</code>
**<code>FLUSH PRIVILEGES;</code>
**<code>EXIT;</code>
*<code>mysqldump -u user -p DBname > DBname.sql</code>
*<code>scp DBname.sql user@Machine.Domain.TLD:~</code>

=== On the destination server: ===
Create the user & database (following the instructions for the application requiring the database)

*<code>mysql -u user -p DBname < DBname.sql</code>

=== If using a separate DB Server... ===
Create the user & database on the DB Server, then log onto the application server &do the following

(You may have to backtrack & SCP the file to the application server... :P )

*<code>mysql -u user -p -h Machine.Domain.TLD DBname < DBname.sql</code>

Automation - MQTT

2025-10-12T22:53:58Z

Tinker: /* password Authentication */

{{{!}} class="wikitable" style="float: right; width: 322px;" border="2"
{{!}}+ Proven on:
{{!}}- 
{{!}} style="text-align: center; width: 60px;" {{!}} [[File:Logo Debian.png{{!}}60px{{!}}link=https://www.debian.org/{{!}}center{{!}}middle{{!}}frameless]]
{{!}} style="text-align: center; width: 40px;" {{!}} 12 (bookworm)
{{!}} ([[#Debian / Version Caveat{{!}}caveat]])
 
{{!}}- 
{{!}} style="text-align: center; width: 65.675px;" {{!}} [[File:Logo LMDE.png{{!}}60px{{!}}link=https://linuxmint.com/edition.php?id=279{{!}}center{{!}}middle{{!}}frameless]]
{{!}} 4
{{!}} ([[#Debian / Version Caveat{{!}}caveat]])
 
{{!}}- 
{{!}} style="text-align: center; width: 67.5125px;" {{!}} [[File:Logo Mint.png{{!}}60px{{!}}link=https://linuxmint.com{{!}}center{{!}}middle{{!}}frameless]]
{{!}} 19.3 / 20.3
{{!}}
 
{{!}}- 
{{!}} style="text-align: center; width: 65.7px;" {{!}} [[File:Logo Ubuntu.png{{!}}60px{{!}}link=https://ubuntu.com/{{!}}center{{!}}middle{{!}}frameless]]
{{!}} 20.04.3
{{!}}
 
{{!}}- 
{{!}} style="text-align: center; width: 65.7px;" {{!}} [[File:Logo Sparky.png{{!}}60px{{!}}link=https://sparkylinux.org/{{!}}center{{!}}middle{{!}}frameless]]
{{!}} 5.11
{{!}} ([[#Debian / Version Caveat{{!}}caveat]])
 
{{!}}}

[https://mosquitto.org/ Mosquitto MQTT Broker]

{{{!}} class="wikitable" style="border-collapse: collapse; width: 33%; left;"
{{!}}- style="text-align: center;"
! style="width: 50%;" colspan="2" {{!}} As always...
{{!}}-
{{!}} {{!}}
Start with:
{{!}} {{!}}
*<code>sudo apt update</code>
*<code>sudo apt upgrade</code>
{{!}}}

= Installing Mosquitto =

*<code>sudo apt-get install mosquitto</code>
Pretty simple, eh?

(Tho... If you want the latest & gratest and you're feeling adventurous... There's a [[Mosquitto from Source|CopyPasta page here]].)
== Ensure that Mosquitto broker is running ==

*<code>sudo service mosquitto status</code>
expected result is <code>Active: active (running)</code>

== Note: Ubuntu repositories have an outdated version ==

If you want to know which version you've installed...

*<code>sudo mosquitto</code>
(Ignore the error message "Error: Address already in use". It's already running as a service.)

= Install client tools for testing etc =

(Do this on any machine expected to manually use MQTT)

*<code>sudo apt install mosquitto-clients</code>

== Testing ==

=== In a terminal: ===

*<code>mosquitto_sub -h localhost -t "mqtt" -v</code>

=== In another terminal: ===

*<code>mosquitto_pub -h localhost -t "mqtt" -m "Hello MQTT"</code>
Now the message “<code>mqtt Hello MQTT</code>” will be displayed in the first terminal where the topic “'''mqtt'''” is subscribed.

Subscribing to '''<code>#</code>''' gives you a subscription to everything except for topics that start with a '''<code>$</code>''' (these are normally control topics anyway).

= Debian / Version Caveat =
(Ignore this if you built from source. It's apparently a repo thing...)

During an install on a raw Debian system, I discovered that Mosquitto refused connection when I tried to access it with anything other than "localhost" as the hostname...

Apparently, mosquitto 2.0 binds only to the loopback interface unless specifically told otherwise.

& Debian installs v2.0 or higher...

But for now, It's a simple matter of editing the config file for Mosquitto.

* <code>sudo vi /etc/mosquitto/mosquitto.conf</code>

& add in:

listener 1883
allow_anonymous true

Then,

* <code>sudo service mosquitto restart</code>

= Securing the broker =
[https://mosquitto.org/documentation/authentication-methods/ Reference]

Mosquitto supports password authentication...

Simplest is:

== password Authentication ==

You can create a password file by:
* <code>sudo mosquitto_passwd -c /etc/mosquitto/SecretSquirrels '''USERNAME'''</code>
Then edit the configuration:
* <code>sudo vi /etc/mosquitto/mosquitto.conf</code>
and replace <code>allow_anonymous true</code> with <code>password_file /etc/mosquitto/SecretSquirrels</code>

Then restart the broker:

* <code>sudo service mosquitto restart</code>
From this point, you have to provide '''USERNAME''' & '''PASSWORD''' to access it.

=== Note: ===
The password file must be able to be read by whatever user Mosquitto is running as. On Linux/POSIX systems this will typically be the mosquitto user, and /etc/mosquitto/password_file is a good place for the file itself.

[[Category:ServerBuilding]]
[[Category:AutomationServers]]

CopyPasta

2025-10-11T19:11:43Z

Tinker: /* Get rid of Firefox-ESR */

= For doing basic setup of an LXC (or pretty much any Linux VM...) (Heck, I run these on physical machines too.): =
I run this set of commands '''as root''' on nearly every new VM or LXC I spin up. That way, I have a consistant environment to work in with all the tools I rely on.

Note: You CAN copyPasta groups of commands all at once. BUT: in this case, everything indented (after <code>su - '''yourname'''</code>) needs to be pasted AFTER that command has run.

* <code>apt update</code>
* <code>apt -y upgrade</code>
* <code>apt -y install sudo rsync vim curl mosquitto-clients</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/RootStuff/DOT.bashrc</nowiki> -O .bashrc</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/RootStuff/DOT.vimrc</nowiki> -O .vimrc</code>
* <code>mkdir bin</code>
* <code>source .bashrc</code>
* <code>apt -y install gnupg tmux htop ### Optional...</code>
* <code>apt -y install qemu-guest-agent ### Only on VMs. (Wish it worked for LXCs as well tho...)</code>
* <code>adduser '''yourname'''</code>
* <code>adduser '''yourname''' sudo</code>
* <code>su - '''yourname'''</code>
** <code>wget <nowiki>https://www.nerdmage.ca/Downloads/UserStuff/DOT.bashrc</nowiki> -O .bashrc</code>
** <code>wget <nowiki>https://www.nerdmage.ca/Downloads/UserStuff/DOT.vimrc</nowiki> -O .vimrc</code>
** <code>mkdir bin</code>
** <code>source .bashrc</code>
** <code>ssh-keygen ### Optional... But rather handy.</code>

=New System Cleanup & Prep=
Freshly installed systems usually have a bunch of stuff installed that you will never use.

For myself, I don't generally have any interest in the games. I also don't use some of the Internet apps that seem important to the developers/maintainers.

These instructions are based on running Debian with the Cinnamon Desktop Environment...
==Remove all the Games==

* <code>sudo apt purge gnome-2048 aisleriot atomix gnome-chess five-or-more hitori iagno gnome-klotski lightsoff gnome-mahjongg gnome-mines gnome-nibbles quadrapassel four-in-a-row gnome-robots gnome-sudoku swell-foop tali gnome-taquin gnome-tetravex -y</code>
* <code>sudo apt autoremove -y</code>

==Remove obsolete and/or silly Internet apps==

* <code>sudo apt purge hexchat pidgin transmission-gtk thunderbird -y</code>
* <code>sudo apt autoremove -y</code>

== Get rid of Firefox ==
* <code>sudo apt purge firefox*</code>

= Simplified installation techniques & instructions =
Some cross-platform software seems to have Linux installation instructions written by people who have never actually used Linux.

(Or, at least, people who believe in making life difficult...)

So...

* [[Installing Brave Browser]] (Properly... Not FLATPAK)
* [[Installing TP-Link Omada SDN Controller on a Debian-based LXC]]
Also, many installation instructions are long-winded or confusing...

So...

* [[Installing Docker on a Debian-based system]]

= Building/Installing things from source =
You may have noticed that the various distro repositories tend to have outdated versions of some (most) packages. While this makes perfect sense, sometimes you want the newest features & fixes. (Or maybe you just want to enable something that the repo managers figured wouldn't be useful...)

Annoyingly, it is rather common for build instructions to suck really badly.

So...

* [[Arp-Scan from Source{{!}}Arp-Scan]]
* [[FreeCAD from Source{{!}}FreeCAD]]
* [[KiCAD from Source{{!}}KiCAD]]
* [[Mosquitto from Source{{!}}Mosquitto]]

= Useful console display for Proxmox Virtual Environment =
I find it nice to have stats & such on the console of a server. This way I can just look & see what's happening with the machine.

* <code>apt update</code>
* <code>apt -y upgrade</code>
* <code>apt -y install tmux htop</code>
* <code>cd /usr/local/bin</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/PVEStuff/bin/TMUX-console</nowiki> -O TMUX-console</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/PVEStuff/bin/TM-guestlist</nowiki> -O TM-guestlist</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/PVEStuff/bin/TM-clusterstatus</nowiki> -O TM-clusterstatus</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/PVEStuff/bin/TM-systemstatus</nowiki> -O TM-systemstatus</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/PVEStuff/bin/TM-UPSstatus</nowiki> -O TM-UPSstatus</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/PVEStuff/bin/TM-FixThis</nowiki> -O TM-FixThis</code>
* <code>wget <nowiki>https://www.nerdmage.ca/Downloads/PVEStuff/bin/TM-ShowMe</nowiki> -O TM-ShowMe</code>
* <code>chmod +x TM*</code>

You will need to edit '''TM-UPSstatus''' to match local configurations (i.e. UPS name).

Running <code>TMUX-console</code> at the console will create a formatted screen of useful system information that you can reach by attaching to the '''tmux''' session from a terminal (SSH) session.

YT-DLP Scripting

2025-10-07T14:58:35Z

Tinker: /* One script with multiple personalities */

= About the weirdness that is YouTube naming conventions =
== When looking at a channel page: ==

<nowiki>https://www.youtube.com/@</nowiki>'''Tinker001'''

The '''bold purple''' part is the channel name

== When watching a video: ==

<nowiki>https://www.youtube.com/watch?v=</nowiki>'''y8o7qkmiDso'''

The '''bold purple''' part is the video name

== When watching a playlist: ==

<nowiki>https://www.youtube.com/watch?v=</nowiki>'''oizvS01ovHE'''&list='''PLLG8IgBGD4eUG2n6p3GGfkQ-KPceDuuWj'''&pp=gAQB

The '''bold purple''' part is the playlist name

Bonus fun bit: The '''bold green''' part is actually the name of the video being watched inside that playlist.

= Cookies! =
For the potentially required cookie file, see [https://github.com/yt-dlp/yt-dlp/wiki/Extractors#exporting-youtube-cookies Exporting YouTube cookies]

= One script with multiple personalities =
Yup...

Looks even more complicated. In fact, it might even be considered kinda silly.

(assumes your base for downloading YouTube videos is at '''/mnt/Download_Space/YTDL/''' and contains subfolders named '''_Individual_''' and '''_Playlists_''')

Save this script as '''YTall'''

Then make symbolic links to it.
* <code>ln -s YTall YTc</code>
* <code>ln -s YTall YTv</code>
* <code>ln -s YTall YTl</code>
Call it as '''YTc''' for doing whole channels, '''YTv''' for individual videos, & '''YTl''' for playlists.

<syntaxhighlight lang="Bash" line copy>
#!/bin/bash

BASENAME=$(basename "$0")

echo -e "~~~~~~~ $BASENAME $@ ~~~~~~~ "\\n

case $BASENAME in
"YTv")
TYPE="VideoHash"
DESC="$TYPE = The name YouTube has given to the individual video"
DESC+="\n\\t\\t (This is gonna take some work to explain...)"
DESC+="\n\\t\\t You can specify multiple videos"
;;
"YTc")
TYPE="ChannelName"
DESC="$TYPE = The name of the channel $as defined by YouTube$"
DESC+="\n\\t\\t You can specify multiple channels"
;;
"YTl")
TYPE="ListHash"
DESC="$TYPE = The name YouTube has given to the playlist"
DESC+="\n\\t\\t (This is gonna take some work to explain...)"
DESC+="\n\\t\\t You can specify multiple lists"
;;
*)
TYPE="poop"
DESC="$TYPE = Some crap"
;;
esac

if [ $# -eq 0 ] || [ $1 = "-u" ] || [ $1 = "-?" ]; then
echo -e USAGE: $(basename "$0") \[...option$s$...\] $TYPE $TYPE ...
echo
echo -e OPTIONS:
echo -e \\t -BOB = Grant Style Folder Structure
echo -e \\t -a = Download audio only $into MP3 files$
echo
echo -e \\t -s = simulate this run
echo -e \\t -c = use a cookie file exported from your browser to identify yourself.
echo -e \\t -r RATE = Limit bandwidth
echo -e \\t\\t Maximum download rate in bytes per second,
echo -e \\t\\t\\t e.g. 50K or 4.2M
echo -e \\t -p MIN MAX = Sleep interval $pause$ between downloads $in seconds$
echo -e \\t\\t $will result in a random pause between MIN \& MAX seconds long$
echo -e \\t -e DATE = Earliest video upload date to grab.
echo -e \\t\\t see https://github.com/yt-dlp/yt-dlp?tab=readme-ov-file#video-selection for format info
echo -e \\t\\t $Simplest is YYYYMMDD but fancier options exist$
echo -e \\t $DESC
echo
echo -e The following \"ERROR\"s will also appear in 00000000-ERRORS
echo -e $Unfortunately, without indications of which files caused them...$
echo
echo -e \"ERROR: unable to download video data: HTTP Error 403: Forbidden\"
echo -e \\t Means you may need to supply cookies.
echo -e \\t Often caused by things like age restrictions.
echo -e \\t May also be caused by YouTube noticing you\'ve been downloading a bunch...
case $BASENAME in
"YTc")
echo
echo -e \"ERROR: \[youtube\] oizvS01ovH0: Video unavailable\"
echo -e \\t Most likely caused by a typo in the VideoHash
;;
"YTv")
echo
echo -e \"ERROR: \[youtube:tab\] \@ChannelName: This channel does not have a videos tab\"
echo -e \"ERROR: \[youtube:tab\] \@ChannelName: This channel does not have a shorts tab\"
echo -e \"ERROR: \[youtube:tab\] \@ChannelName: This channel does not have a streams tab\"
echo -e \\t Is simply indicating that the channel doesn\'t have that type of content.
echo -e \\t $All 3 means the channel doesn\'t actually have any content... :P $
;;
"YTl")
echo
echo -e \"mv: cannot stat \'*\': No such file or directory\"
echo -e \\t Don\'t panic. This one\'s just a bit of residue from the playlist folder cleanup
;;
esac
echo

exit
fi

BASEdest=/mnt/Download_Space/YTDL/
COOKIEfile=/mnt/Download_Space/YTDL/www.youtube.com_cookies.txt

SIMULATE=""
WITHcookies=""
LIMITrate=""
PAUSEbetween=""
TIMEspan=""
BOBstyle=""
AUDIOonly=""
FILETYPE="-t mp4"

while (( "$#" ));
do
case $1 in
"-c")
echo Using Cookies: $COOKIEfile
WITHcookies="--cookies "$COOKIEfile
shift
;;

"-s")
echo Simulating
SIMULATE="-s"
shift
;;

"-r")
LIMITrate="-r $2"
echo Rate Limiting $2
shift 2
;;

"-p")
PAUSEbetween="--sleep-interval $2 --max-sleep-interval $3"
echo Random pause between $2 \& $3 seconds
shift 3
;;

"-e")
TIMEspan="--dateafter $2"
echo Time Span $2 to current
shift 2
;;

"-BOB")
BOBstyle="True"
echo Using Grant Style Folder Structure
shift
;;

"-a")
AUDIOonly="--extract-audio --audio-format mp3 --audio-quality 0"
FILETYPE="-t mp3"
echo Extracting audio only
shift
;;

*)
case $BASENAME in
"YTv") # Video
echo Working on video: $1
echo ========================================
WeirdID=$1
URL=https://www.youtube.com/watch\?v=$1
TYPES=""
DEST="$BASEdest""_Individual_/"
FORMAT="%(uploader_id)s - %(upload_date)s - %(title)s.%(ext)s"
;;

"YTc") # Channel
echo Working on Channel: $1
echo ========================================
WeirdID=$1
URL=https://www.youtube.com/\@$1
TYPES="videos shorts streams"
# TYPES="videos shorts streams playlists podcasts courses"
DEST="$BASEdest""$1"
FORMAT="%(upload_date)s - %(title)s.%(ext)s"
;;

"YTl") # Playlist
echo Working on playlist: $1
echo ========================================
WeirdID=$1
URL=https://www.youtube.com/playlist\?list=$1
TYPES=""
DEST="$BASEdest""_Playlists_/"
FORMAT="%(playlist_index)s_%(playlist_count)s %(uploader_id)s - %(title)s.%(ext)s"
;;
esac

ERRfile=$DEST/00000000-ERRORS
ARCHfile=$DEST/11111111-ARCHIVE

OPTIONS="$SIMULATE $LIMITrate $TIMEspan $WITHcookies $PAUSEbetween $AUDIOonly"
OPTIONS+=" --no-overwrites --write-description $FILETYPE"

echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
echo $URL
echo $WeirdID
echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"

if wget --spider --quiet $URL > /dev/null 2>&1; then
# Weirdly, for individual videos, YouTube says the URL is valid even if it doesn't exist when asking about videos...
# Then it gives an error while attempting the download.
# Not fatal, just annoying...

echo -en "(storing in $DEST"

case $BASENAME in
"YTv")
echo ")"
echo VID $WeirdID >> $ARCHfile
cd $DEST
pwd
/usr/local/bin/yt-dlp $OPTIONS --download-archive $ARCHfile -o "$FORMAT" $URL 2> >(/usr/bin/tee -a $ERRfile)
rm -f *.part
;;
"YTl")
echo "$WeirdID)"
echo LIST $WeirdID >> $ARCHfile
cd $DEST
pwd
if [ -d $DEST/$WeirdID ]; then
echo $DEST exists
else
echo no $DEST$WeirdID... Building it...
mkdir $DEST$WeirdID
fi
cd $WeirdID
pwd
/usr/local/bin/yt-dlp $OPTIONS --download-archive $ARCHfile -o "$FORMAT" $URL 2> >(/usr/bin/tee -a $ERRfile)
rm -f *.part

NAME=`find . -maxdepth 1 -regextype posix-extended -regex './0+_.*' -type f `

Listname=`echo ${NAME::-12} | cut -d'@' -f 2`

if [ -d ../"$Listname" ]; then
echo "$Listname" already exists
else
mkdir ../"$Listname"
fi

mv "$NAME" ../"$Listname/000 $Listname.description"
mv * ../"$Listname"
cd ..
rmdir $WeirdID
;;
"YTc")
echo ")"
if [ -d $DEST ]; then
echo $DEST exists
else
echo no $DEST... Building it...
mkdir $DEST
if [ -z $BOBstyle ]; then
for ACK in $TYPES
do
mkdir $DEST/$ACK
done
fi
fi

cd $DEST
pwd

if [ -z $BOBstyle ]; then
echo -e "(using subfolders...)"
else
echo -e "(using Grant format...)"
fi
echo

for ACK in $TYPES
do
echo ">>>>>>>> " $URL/$ACK
if [ -z $BOBstyle ]; then
cd $ACK
echo CHAN $WeirdID $ACK >> $ARCHfile
fi

/usr/local/bin/yt-dlp $OPTIONS --download-archive $ARCHfile -o "$FORMAT" $URL/$ACK 2> >(/usr/bin/tee -a $ERRfile)

# Clear out .part files
rm -f *.part

if [ -z $BOBstyle ]; then
cd ..
fi
done
;;
esac
else
echo YouTube says this channel does not exist.
fi
shift
;;

esac
done
</syntaxhighlight>

= Even more options? =
'''WIP'''

= Errors noted, but not researched =
In console output:
* <nowiki>[download] 57.7% of 15.24MiB at 65.83KiB/s ETA 01:40[download] Got error: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer')). Retrying (1/10)...</nowiki>

In 00000000-ERRORS:
* <nowiki>WARNING: [youtube] ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')). Retrying (1/3)...</nowiki>

YT-DLP Scripting

2025-10-07T04:26:46Z

Tinker: /* Errors noted, but not researched */

= About the weirdness that is YouTube naming conventions =
== When looking at a channel page: ==

<nowiki>https://www.youtube.com/@</nowiki>'''Tinker001'''

The '''bold purple''' part is the channel name

== When watching a video: ==

<nowiki>https://www.youtube.com/watch?v=</nowiki>'''y8o7qkmiDso'''

The '''bold purple''' part is the video name

== When watching a playlist: ==

<nowiki>https://www.youtube.com/watch?v=</nowiki>'''oizvS01ovHE'''&list='''PLLG8IgBGD4eUG2n6p3GGfkQ-KPceDuuWj'''&pp=gAQB

The '''bold purple''' part is the playlist name

Bonus fun bit: The '''bold green''' part is actually the name of the video being watched inside that playlist.

= Cookies! =
For the potentially required cookie file, see [https://github.com/yt-dlp/yt-dlp/wiki/Extractors#exporting-youtube-cookies Exporting YouTube cookies]

= One script with multiple personalities =
Yup...

Looks even more complicated. In fact, it might even be considered kinda silly.

(assumes your base for downloading YouTube videos is at '''/mnt/Download_Space/YTDL/''' and contains subfolders named '''_Individual_''' and '''_Playlists_''')

Save this script as '''YTall'''

Then make symbolic links to it.
* <code>ln -s YTall YTc</code>
* <code>ln -s YTall YTv</code>
* <code>ln -s YTall YTl</code>
Call it as '''YTc''' for doing whole channels, '''YTv''' for individual videos, & '''YTl''' for playlists.

<syntaxhighlight lang="Bash" line copy>
#!/bin/bash

BASENAME=$(basename "$0")

echo -e "~~~~~~~ $BASENAME $@ ~~~~~~~ "\\n

case $BASENAME in
"YTv")
TYPE="VideoHash"
DESC="$TYPE = The name YouTube has given to the individual video"
DESC+="\n\\t\\t (This is gonna take some work to explain...)"
DESC+="\n\\t\\t You can specify multiple videos"
;;
"YTc")
TYPE="ChannelName"
DESC="$TYPE = The name of the channel $as defined by YouTube$"
DESC+="\n\\t\\t You can specify multiple channels"
;;
"YTl")
TYPE="ListHash"
DESC="$TYPE = The name YouTube has given to the playlist"
DESC+="\n\\t\\t (This is gonna take some work to explain...)"
DESC+="\n\\t\\t You can specify multiple lists"
;;
*)
TYPE="poop"
DESC="$TYPE = Some crap"
;;
esac

if [ $# -eq 0 ] || [ $1 = "-u" ] || [ $1 = "-?" ]; then
echo -e USAGE: $(basename "$0") \[...option$s$...\] $TYPE $TYPE ...
echo
echo -e OPTIONS:
echo -e \\t -BOB = Grant Style Folder Structure
echo -e \\t -a = Download audio only $into MP3 files$
echo
echo -e \\t -s = simulate this run
echo -e \\t -c = use a cookie file exported from your browser to identify yourself.
echo -e \\t -r RATE = Limit bandwidth
echo -e \\t\\t Maximum download rate in bytes per second,
echo -e \\t\\t\\t e.g. 50K or 4.2M
echo -e \\t -p MIN MAX = Sleep interval $pause$ between downloads $in seconds$
echo -e \\t\\t $will result in a random pause between MIN \& MAX seconds long$
echo -e \\t -e DATE = Earliest video upload date to grab.
echo -e \\t\\t see https://github.com/yt-dlp/yt-dlp?tab=readme-ov-file#video-selection for format info
echo -e \\t\\t $Simplest is YYYYMMDD but fancier options exist$
echo -e \\t $DESC
echo
echo -e The following \"ERROR\"s will also appear in 00000000-ERRORS
echo -e $Unfortunately, without indications of which files caused them...$
echo
echo -e \"ERROR: unable to download video data: HTTP Error 403: Forbidden\"
echo -e \\t Means you may need to supply cookies.
echo -e \\t Often caused by things like age restrictions.
echo -e \\t May also be caused by YouTube noticing you\'ve been downloading a bunch...
case $BASENAME in
"YTc")
echo
echo -e \"ERROR: \[youtube\] oizvS01ovH0: Video unavailable\"
echo -e \\t Most likely caused by a typo in the VideoHash
;;
"YTv")
echo
echo -e \"ERROR: \[youtube:tab\] \@ChannelName: This channel does not have a videos tab\"
echo -e \"ERROR: \[youtube:tab\] \@ChannelName: This channel does not have a shorts tab\"
echo -e \"ERROR: \[youtube:tab\] \@ChannelName: This channel does not have a streams tab\"
echo -e \\t Is simply indicating that the channel doesn\'t have that type of content.
echo -e \\t $All 3 means the channel doesn\'t actually have any content... :P $
;;
"YTl")
echo
echo -e \"mv: cannot stat \'*\': No such file or directory\"
echo -e \\t Don\'t panic. This one\'s just a bit of residue from the playlist folder cleanup
;;
esac
echo

exit
fi

BASEdest=/mnt/Download_Space/YTDL/
COOKIEfile=/mnt/Download_Space/YTDL/www.youtube.com_cookies.txt

SIMULATE=""
WITHcookies=""
LIMITrate=""
PAUSEbetween=""
TIMEspan=""
BOBstyle=""
AUDIOonly=""
FORMAT="-t mp4"

while (( "$#" ));
do
case $1 in
"-c")
echo Using Cookies: $COOKIEfile
WITHcookies="--cookies "$COOKIEfile
shift
;;

"-s")
echo Simulating
SIMULATE="-s"
shift
;;

"-r")
LIMITrate="-r $2"
echo Rate Limiting $2
shift 2
;;

"-p")
PAUSEbetween="--sleep-interval $2 --max-sleep-interval $3"
echo Random pause between $2 \& $3 seconds
shift 3
;;

"-e")
TIMEspan="--dateafter $2"
echo Time Span $2 to current
shift 2
;;

"-BOB")
BOBstyle="True"
echo Using Grant Style Folder Structure
shift
;;

"-a")
AUDIOonly="--extract-audio --audio-format mp3 --audio-quality 0"
FORMAT="-t mp3"
echo Extracting audio only
shift
;;

*)
case $BASENAME in
"YTv") # Video
echo Working on video: $1
echo ========================================
WeirdID=$1
URL=https://www.youtube.com/watch\?v=$1
TYPES=""
DEST="$BASEdest""_Individual_/"
FORMAT="%(uploader_id)s - %(upload_date)s - %(title)s.%(ext)s"
;;

"YTc") # Channel
echo Working on Channel: $1
echo ========================================
WeirdID=$1
URL=https://www.youtube.com/\@$1
TYPES="videos shorts streams"
# TYPES="videos shorts streams playlists podcasts courses"
DEST="$BASEdest""$1"
FORMAT="%(upload_date)s - %(title)s.%(ext)s"
;;

"YTl") # Playlist
echo Working on playlist: $1
echo ========================================
WeirdID=$1
URL=https://www.youtube.com/playlist\?list=$1
TYPES=""
DEST="$BASEdest""_Playlists_/"
FORMAT="%(playlist_index)s_%(playlist_count)s %(uploader_id)s - %(title)s.%(ext)s"
;;
esac

ERRfile=$DEST/00000000-ERRORS
ARCHfile=$DEST/11111111-ARCHIVE

OPTIONS="$SIMULATE $LIMITrate $TIMEspan $WITHcookies $PAUSEbetween $AUDIOonly"
OPTIONS+=" --no-overwrites --write-description $FORMAT"

echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
echo $URL
echo $WeirdID
echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"

if wget --spider --quiet $URL > /dev/null 2>&1; then
# Weirdly, for individual videos, YouTube says the URL is valid even if it doesn't exist when asking about videos...
# Then it gives an error while attempting the download.
# Not fatal, just annoying...

echo -en "(storing in $DEST"

case $BASENAME in
"YTv")
echo ")"
echo VID $WeirdID >> $ARCHfile
cd $DEST
pwd
/usr/local/bin/yt-dlp $OPTIONS --download-archive $ARCHfile -o "$FORMAT" $URL 2> >(/usr/bin/tee -a $ERRfile)
rm -f *.part
;;
"YTl")
echo "$WeirdID)"
echo LIST $WeirdID >> $ARCHfile
cd $DEST
pwd
if [ -d $DEST/$WeirdID ]; then
echo $DEST exists
else
echo no $DEST$WeirdID... Building it...
mkdir $DEST$WeirdID
fi
cd $WeirdID
pwd
/usr/local/bin/yt-dlp $OPTIONS --download-archive $ARCHfile -o "$FORMAT" $URL 2> >(/usr/bin/tee -a $ERRfile)
rm -f *.part

NAME=`find . -maxdepth 1 -regextype posix-extended -regex './0+_.*' -type f `

Listname=`echo ${NAME::-12} | cut -d'@' -f 2`

if [ -d ../"$Listname" ]; then
echo "$Listname" already exists
else
mkdir ../"$Listname"
fi

mv "$NAME" ../"$Listname/000 $Listname.description"
mv * ../"$Listname"
cd ..
rmdir $WeirdID
;;
"YTc")
echo ")"
if [ -d $DEST ]; then
echo $DEST exists
else
echo no $DEST... Building it...
mkdir $DEST
if [ -z $BOBstyle ]; then
for ACK in $TYPES
do
mkdir $DEST/$ACK
done
fi
fi

cd $DEST
pwd

if [ -z $BOBstyle ]; then
echo -e "(using subfolders...)"
else
echo -e "(using Grant format...)"
fi
echo

for ACK in $TYPES
do
echo ">>>>>>>> " $URL/$ACK
if [ -z $BOBstyle ]; then
cd $ACK
echo CHAN $WeirdID $ACK >> $ARCHfile
fi

/usr/local/bin/yt-dlp $OPTIONS --download-archive $ARCHfile -o "$FORMAT" $URL/$ACK 2> >(/usr/bin/tee -a $ERRfile)

# Clear out .part files
rm -f *.part

if [ -z $BOBstyle ]; then
cd ..
fi
done
;;
esac
else
echo YouTube says this channel does not exist.
fi
shift
;;

esac
done
</syntaxhighlight>

= Even more options? =
'''WIP'''

= Errors noted, but not researched =
In console output:
* <nowiki>[download] 57.7% of 15.24MiB at 65.83KiB/s ETA 01:40[download] Got error: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer')). Retrying (1/10)...</nowiki>

In 00000000-ERRORS:
* <nowiki>WARNING: [youtube] ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')). Retrying (1/3)...</nowiki>

YT-DLP Scripting

2025-10-07T04:25:35Z

Tinker: /* Even more options? */

= About the weirdness that is YouTube naming conventions =
== When looking at a channel page: ==

<nowiki>https://www.youtube.com/@</nowiki>'''Tinker001'''

The '''bold purple''' part is the channel name

== When watching a video: ==

<nowiki>https://www.youtube.com/watch?v=</nowiki>'''y8o7qkmiDso'''

The '''bold purple''' part is the video name

== When watching a playlist: ==

<nowiki>https://www.youtube.com/watch?v=</nowiki>'''oizvS01ovHE'''&list='''PLLG8IgBGD4eUG2n6p3GGfkQ-KPceDuuWj'''&pp=gAQB

The '''bold purple''' part is the playlist name

Bonus fun bit: The '''bold green''' part is actually the name of the video being watched inside that playlist.

= Cookies! =
For the potentially required cookie file, see [https://github.com/yt-dlp/yt-dlp/wiki/Extractors#exporting-youtube-cookies Exporting YouTube cookies]

= One script with multiple personalities =
Yup...

Looks even more complicated. In fact, it might even be considered kinda silly.

(assumes your base for downloading YouTube videos is at '''/mnt/Download_Space/YTDL/''' and contains subfolders named '''_Individual_''' and '''_Playlists_''')

Save this script as '''YTall'''

Then make symbolic links to it.
* <code>ln -s YTall YTc</code>
* <code>ln -s YTall YTv</code>
* <code>ln -s YTall YTl</code>
Call it as '''YTc''' for doing whole channels, '''YTv''' for individual videos, & '''YTl''' for playlists.

<syntaxhighlight lang="Bash" line copy>
#!/bin/bash

BASENAME=$(basename "$0")

echo -e "~~~~~~~ $BASENAME $@ ~~~~~~~ "\\n

case $BASENAME in
"YTv")
TYPE="VideoHash"
DESC="$TYPE = The name YouTube has given to the individual video"
DESC+="\n\\t\\t (This is gonna take some work to explain...)"
DESC+="\n\\t\\t You can specify multiple videos"
;;
"YTc")
TYPE="ChannelName"
DESC="$TYPE = The name of the channel $as defined by YouTube$"
DESC+="\n\\t\\t You can specify multiple channels"
;;
"YTl")
TYPE="ListHash"
DESC="$TYPE = The name YouTube has given to the playlist"
DESC+="\n\\t\\t (This is gonna take some work to explain...)"
DESC+="\n\\t\\t You can specify multiple lists"
;;
*)
TYPE="poop"
DESC="$TYPE = Some crap"
;;
esac

if [ $# -eq 0 ] || [ $1 = "-u" ] || [ $1 = "-?" ]; then
echo -e USAGE: $(basename "$0") \[...option$s$...\] $TYPE $TYPE ...
echo
echo -e OPTIONS:
echo -e \\t -BOB = Grant Style Folder Structure
echo -e \\t -a = Download audio only $into MP3 files$
echo
echo -e \\t -s = simulate this run
echo -e \\t -c = use a cookie file exported from your browser to identify yourself.
echo -e \\t -r RATE = Limit bandwidth
echo -e \\t\\t Maximum download rate in bytes per second,
echo -e \\t\\t\\t e.g. 50K or 4.2M
echo -e \\t -p MIN MAX = Sleep interval $pause$ between downloads $in seconds$
echo -e \\t\\t $will result in a random pause between MIN \& MAX seconds long$
echo -e \\t -e DATE = Earliest video upload date to grab.
echo -e \\t\\t see https://github.com/yt-dlp/yt-dlp?tab=readme-ov-file#video-selection for format info
echo -e \\t\\t $Simplest is YYYYMMDD but fancier options exist$
echo -e \\t $DESC
echo
echo -e The following \"ERROR\"s will also appear in 00000000-ERRORS
echo -e $Unfortunately, without indications of which files caused them...$
echo
echo -e \"ERROR: unable to download video data: HTTP Error 403: Forbidden\"
echo -e \\t Means you may need to supply cookies.
echo -e \\t Often caused by things like age restrictions.
echo -e \\t May also be caused by YouTube noticing you\'ve been downloading a bunch...
case $BASENAME in
"YTc")
echo
echo -e \"ERROR: \[youtube\] oizvS01ovH0: Video unavailable\"
echo -e \\t Most likely caused by a typo in the VideoHash
;;
"YTv")
echo
echo -e \"ERROR: \[youtube:tab\] \@ChannelName: This channel does not have a videos tab\"
echo -e \"ERROR: \[youtube:tab\] \@ChannelName: This channel does not have a shorts tab\"
echo -e \"ERROR: \[youtube:tab\] \@ChannelName: This channel does not have a streams tab\"
echo -e \\t Is simply indicating that the channel doesn\'t have that type of content.
echo -e \\t $All 3 means the channel doesn\'t actually have any content... :P $
;;
"YTl")
echo
echo -e \"mv: cannot stat \'*\': No such file or directory\"
echo -e \\t Don\'t panic. This one\'s just a bit of residue from the playlist folder cleanup
;;
esac
echo

exit
fi

BASEdest=/mnt/Download_Space/YTDL/
COOKIEfile=/mnt/Download_Space/YTDL/www.youtube.com_cookies.txt

SIMULATE=""
WITHcookies=""
LIMITrate=""
PAUSEbetween=""
TIMEspan=""
BOBstyle=""
AUDIOonly=""
FORMAT="-t mp4"

while (( "$#" ));
do
case $1 in
"-c")
echo Using Cookies: $COOKIEfile
WITHcookies="--cookies "$COOKIEfile
shift
;;

"-s")
echo Simulating
SIMULATE="-s"
shift
;;

"-r")
LIMITrate="-r $2"
echo Rate Limiting $2
shift 2
;;

"-p")
PAUSEbetween="--sleep-interval $2 --max-sleep-interval $3"
echo Random pause between $2 \& $3 seconds
shift 3
;;

"-e")
TIMEspan="--dateafter $2"
echo Time Span $2 to current
shift 2
;;

"-BOB")
BOBstyle="True"
echo Using Grant Style Folder Structure
shift
;;

"-a")
AUDIOonly="--extract-audio --audio-format mp3 --audio-quality 0"
FORMAT="-t mp3"
echo Extracting audio only
shift
;;

*)
case $BASENAME in
"YTv") # Video
echo Working on video: $1
echo ========================================
WeirdID=$1
URL=https://www.youtube.com/watch\?v=$1
TYPES=""
DEST="$BASEdest""_Individual_/"
FORMAT="%(uploader_id)s - %(upload_date)s - %(title)s.%(ext)s"
;;

"YTc") # Channel
echo Working on Channel: $1
echo ========================================
WeirdID=$1
URL=https://www.youtube.com/\@$1
TYPES="videos shorts streams"
# TYPES="videos shorts streams playlists podcasts courses"
DEST="$BASEdest""$1"
FORMAT="%(upload_date)s - %(title)s.%(ext)s"
;;

"YTl") # Playlist
echo Working on playlist: $1
echo ========================================
WeirdID=$1
URL=https://www.youtube.com/playlist\?list=$1
TYPES=""
DEST="$BASEdest""_Playlists_/"
FORMAT="%(playlist_index)s_%(playlist_count)s %(uploader_id)s - %(title)s.%(ext)s"
;;
esac

ERRfile=$DEST/00000000-ERRORS
ARCHfile=$DEST/11111111-ARCHIVE

OPTIONS="$SIMULATE $LIMITrate $TIMEspan $WITHcookies $PAUSEbetween $AUDIOonly"
OPTIONS+=" --no-overwrites --write-description $FORMAT"

echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
echo $URL
echo $WeirdID
echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"

if wget --spider --quiet $URL > /dev/null 2>&1; then
# Weirdly, for individual videos, YouTube says the URL is valid even if it doesn't exist when asking about videos...
# Then it gives an error while attempting the download.
# Not fatal, just annoying...

echo -en "(storing in $DEST"

case $BASENAME in
"YTv")
echo ")"
echo VID $WeirdID >> $ARCHfile
cd $DEST
pwd
/usr/local/bin/yt-dlp $OPTIONS --download-archive $ARCHfile -o "$FORMAT" $URL 2> >(/usr/bin/tee -a $ERRfile)
rm -f *.part
;;
"YTl")
echo "$WeirdID)"
echo LIST $WeirdID >> $ARCHfile
cd $DEST
pwd
if [ -d $DEST/$WeirdID ]; then
echo $DEST exists
else
echo no $DEST$WeirdID... Building it...
mkdir $DEST$WeirdID
fi
cd $WeirdID
pwd
/usr/local/bin/yt-dlp $OPTIONS --download-archive $ARCHfile -o "$FORMAT" $URL 2> >(/usr/bin/tee -a $ERRfile)
rm -f *.part

NAME=`find . -maxdepth 1 -regextype posix-extended -regex './0+_.*' -type f `

Listname=`echo ${NAME::-12} | cut -d'@' -f 2`

if [ -d ../"$Listname" ]; then
echo "$Listname" already exists
else
mkdir ../"$Listname"
fi

mv "$NAME" ../"$Listname/000 $Listname.description"
mv * ../"$Listname"
cd ..
rmdir $WeirdID
;;
"YTc")
echo ")"
if [ -d $DEST ]; then
echo $DEST exists
else
echo no $DEST... Building it...
mkdir $DEST
if [ -z $BOBstyle ]; then
for ACK in $TYPES
do
mkdir $DEST/$ACK
done
fi
fi

cd $DEST
pwd

if [ -z $BOBstyle ]; then
echo -e "(using subfolders...)"
else
echo -e "(using Grant format...)"
fi
echo

for ACK in $TYPES
do
echo ">>>>>>>> " $URL/$ACK
if [ -z $BOBstyle ]; then
cd $ACK
echo CHAN $WeirdID $ACK >> $ARCHfile
fi

/usr/local/bin/yt-dlp $OPTIONS --download-archive $ARCHfile -o "$FORMAT" $URL/$ACK 2> >(/usr/bin/tee -a $ERRfile)

# Clear out .part files
rm -f *.part

if [ -z $BOBstyle ]; then
cd ..
fi
done
;;
esac
else
echo YouTube says this channel does not exist.
fi
shift
;;

esac
done
</syntaxhighlight>

= Even more options? =
'''WIP'''

= Errors noted, but not researched =
In console output:
* <nowiki>[download] 57.7% of 15.24MiB at 65.83KiB/s ETA 01:40[download] Got error: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer')). Retrying (1/10)...</nowiki>

In 00000000-ERRORS:
* <nowiki>WARNING: [youtube] ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')). Retrying (1/3)...</niwiki>

YT-DLP Scripting

2025-10-06T06:38:49Z

Tinker: /* One script with multiple personalities */

= About the weirdness that is YouTube naming conventions =
== When looking at a channel page: ==

<nowiki>https://www.youtube.com/@</nowiki>'''Tinker001'''

The '''bold purple''' part is the channel name

== When watching a video: ==

<nowiki>https://www.youtube.com/watch?v=</nowiki>'''y8o7qkmiDso'''

The '''bold purple''' part is the video name

== When watching a playlist: ==

<nowiki>https://www.youtube.com/watch?v=</nowiki>'''oizvS01ovHE'''&list='''PLLG8IgBGD4eUG2n6p3GGfkQ-KPceDuuWj'''&pp=gAQB

The '''bold purple''' part is the playlist name

Bonus fun bit: The '''bold green''' part is actually the name of the video being watched inside that playlist.

= Cookies! =
For the potentially required cookie file, see [https://github.com/yt-dlp/yt-dlp/wiki/Extractors#exporting-youtube-cookies Exporting YouTube cookies]

= One script with multiple personalities =
Yup...

Looks even more complicated. In fact, it might even be considered kinda silly.

(assumes your base for downloading YouTube videos is at '''/mnt/Download_Space/YTDL/''' and contains subfolders named '''_Individual_''' and '''_Playlists_''')

Save this script as '''YTall'''

Then make symbolic links to it.
* <code>ln -s YTall YTc</code>
* <code>ln -s YTall YTv</code>
* <code>ln -s YTall YTl</code>
Call it as '''YTc''' for doing whole channels, '''YTv''' for individual videos, & '''YTl''' for playlists.

<syntaxhighlight lang="Bash" line copy>
#!/bin/bash

BASENAME=$(basename "$0")

echo -e "~~~~~~~ $BASENAME $@ ~~~~~~~ "\\n

case $BASENAME in
"YTv")
TYPE="VideoHash"
DESC="$TYPE = The name YouTube has given to the individual video"
DESC+="\n\\t\\t (This is gonna take some work to explain...)"
DESC+="\n\\t\\t You can specify multiple videos"
;;
"YTc")
TYPE="ChannelName"
DESC="$TYPE = The name of the channel $as defined by YouTube$"
DESC+="\n\\t\\t You can specify multiple channels"
;;
"YTl")
TYPE="ListHash"
DESC="$TYPE = The name YouTube has given to the playlist"
DESC+="\n\\t\\t (This is gonna take some work to explain...)"
DESC+="\n\\t\\t You can specify multiple lists"
;;
*)
TYPE="poop"
DESC="$TYPE = Some crap"
;;
esac

if [ $# -eq 0 ] || [ $1 = "-u" ] || [ $1 = "-?" ]; then
echo -e USAGE: $(basename "$0") \[...option$s$...\] $TYPE $TYPE ...
echo
echo -e OPTIONS:
echo -e \\t -BOB = Grant Style Folder Structure
echo -e \\t -a = Download audio only $into MP3 files$
echo
echo -e \\t -s = simulate this run
echo -e \\t -c = use a cookie file exported from your browser to identify yourself.
echo -e \\t -r RATE = Limit bandwidth
echo -e \\t\\t Maximum download rate in bytes per second,
echo -e \\t\\t\\t e.g. 50K or 4.2M
echo -e \\t -p MIN MAX = Sleep interval $pause$ between downloads $in seconds$
echo -e \\t\\t $will result in a random pause between MIN \& MAX seconds long$
echo -e \\t -e DATE = Earliest video upload date to grab.
echo -e \\t\\t see https://github.com/yt-dlp/yt-dlp?tab=readme-ov-file#video-selection for format info
echo -e \\t\\t $Simplest is YYYYMMDD but fancier options exist$
echo -e \\t $DESC
echo
echo -e The following \"ERROR\"s will also appear in 00000000-ERRORS
echo -e $Unfortunately, without indications of which files caused them...$
echo
echo -e \"ERROR: unable to download video data: HTTP Error 403: Forbidden\"
echo -e \\t Means you may need to supply cookies.
echo -e \\t Often caused by things like age restrictions.
echo -e \\t May also be caused by YouTube noticing you\'ve been downloading a bunch...
case $BASENAME in
"YTc")
echo
echo -e \"ERROR: \[youtube\] oizvS01ovH0: Video unavailable\"
echo -e \\t Most likely caused by a typo in the VideoHash
;;
"YTv")
echo
echo -e \"ERROR: \[youtube:tab\] \@ChannelName: This channel does not have a videos tab\"
echo -e \"ERROR: \[youtube:tab\] \@ChannelName: This channel does not have a shorts tab\"
echo -e \"ERROR: \[youtube:tab\] \@ChannelName: This channel does not have a streams tab\"
echo -e \\t Is simply indicating that the channel doesn\'t have that type of content.
echo -e \\t $All 3 means the channel doesn\'t actually have any content... :P $
;;
"YTl")
echo
echo -e \"mv: cannot stat \'*\': No such file or directory\"
echo -e \\t Don\'t panic. This one\'s just a bit of residue from the playlist folder cleanup
;;
esac
echo

exit
fi

BASEdest=/mnt/Download_Space/YTDL/
COOKIEfile=/mnt/Download_Space/YTDL/www.youtube.com_cookies.txt

SIMULATE=""
WITHcookies=""
LIMITrate=""
PAUSEbetween=""
TIMEspan=""
BOBstyle=""
AUDIOonly=""
FORMAT="-t mp4"

while (( "$#" ));
do
case $1 in
"-c")
echo Using Cookies: $COOKIEfile
WITHcookies="--cookies "$COOKIEfile
shift
;;

"-s")
echo Simulating
SIMULATE="-s"
shift
;;

"-r")
LIMITrate="-r $2"
echo Rate Limiting $2
shift 2
;;

"-p")
PAUSEbetween="--sleep-interval $2 --max-sleep-interval $3"
echo Random pause between $2 \& $3 seconds
shift 3
;;

"-e")
TIMEspan="--dateafter $2"
echo Time Span $2 to current
shift 2
;;

"-BOB")
BOBstyle="True"
echo Using Grant Style Folder Structure
shift
;;

"-a")
AUDIOonly="--extract-audio --audio-format mp3 --audio-quality 0"
FORMAT="-t mp3"
echo Extracting audio only
shift
;;

*)
case $BASENAME in
"YTv") # Video
echo Working on video: $1
echo ========================================
WeirdID=$1
URL=https://www.youtube.com/watch\?v=$1
TYPES=""
DEST="$BASEdest""_Individual_/"
FORMAT="%(uploader_id)s - %(upload_date)s - %(title)s.%(ext)s"
;;

"YTc") # Channel
echo Working on Channel: $1
echo ========================================
WeirdID=$1
URL=https://www.youtube.com/\@$1
TYPES="videos shorts streams"
# TYPES="videos shorts streams playlists podcasts courses"
DEST="$BASEdest""$1"
FORMAT="%(upload_date)s - %(title)s.%(ext)s"
;;

"YTl") # Playlist
echo Working on playlist: $1
echo ========================================
WeirdID=$1
URL=https://www.youtube.com/playlist\?list=$1
TYPES=""
DEST="$BASEdest""_Playlists_/"
FORMAT="%(playlist_index)s_%(playlist_count)s %(uploader_id)s - %(title)s.%(ext)s"
;;
esac

ERRfile=$DEST/00000000-ERRORS
ARCHfile=$DEST/11111111-ARCHIVE

OPTIONS="$SIMULATE $LIMITrate $TIMEspan $WITHcookies $PAUSEbetween $AUDIOonly"
OPTIONS+=" --no-overwrites --write-description $FORMAT"

echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
echo $URL
echo $WeirdID
echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"

if wget --spider --quiet $URL > /dev/null 2>&1; then
# Weirdly, for individual videos, YouTube says the URL is valid even if it doesn't exist when asking about videos...
# Then it gives an error while attempting the download.
# Not fatal, just annoying...

echo -en "(storing in $DEST"

case $BASENAME in
"YTv")
echo ")"
echo VID $WeirdID >> $ARCHfile
cd $DEST
pwd
/usr/local/bin/yt-dlp $OPTIONS --download-archive $ARCHfile -o "$FORMAT" $URL 2> >(/usr/bin/tee -a $ERRfile)
rm -f *.part
;;
"YTl")
echo "$WeirdID)"
echo LIST $WeirdID >> $ARCHfile
cd $DEST
pwd
if [ -d $DEST/$WeirdID ]; then
echo $DEST exists
else
echo no $DEST$WeirdID... Building it...
mkdir $DEST$WeirdID
fi
cd $WeirdID
pwd
/usr/local/bin/yt-dlp $OPTIONS --download-archive $ARCHfile -o "$FORMAT" $URL 2> >(/usr/bin/tee -a $ERRfile)
rm -f *.part

NAME=`find . -maxdepth 1 -regextype posix-extended -regex './0+_.*' -type f `

Listname=`echo ${NAME::-12} | cut -d'@' -f 2`

if [ -d ../"$Listname" ]; then
echo "$Listname" already exists
else
mkdir ../"$Listname"
fi

mv "$NAME" ../"$Listname/000 $Listname.description"
mv * ../"$Listname"
cd ..
rmdir $WeirdID
;;
"YTc")
echo ")"
if [ -d $DEST ]; then
echo $DEST exists
else
echo no $DEST... Building it...
mkdir $DEST
if [ -z $BOBstyle ]; then
for ACK in $TYPES
do
mkdir $DEST/$ACK
done
fi
fi

cd $DEST
pwd

if [ -z $BOBstyle ]; then
echo -e "(using subfolders...)"
else
echo -e "(using Grant format...)"
fi
echo

for ACK in $TYPES
do
echo ">>>>>>>> " $URL/$ACK
if [ -z $BOBstyle ]; then
cd $ACK
echo CHAN $WeirdID $ACK >> $ARCHfile
fi

/usr/local/bin/yt-dlp $OPTIONS --download-archive $ARCHfile -o "$FORMAT" $URL/$ACK 2> >(/usr/bin/tee -a $ERRfile)

# Clear out .part files
rm -f *.part

if [ -z $BOBstyle ]; then
cd ..
fi
done
;;
esac
else
echo YouTube says this channel does not exist.
fi
shift
;;

esac
done
</syntaxhighlight>

= Even more options? =
'''WIP'''

YT-DLP Scripting

2025-10-06T05:08:23Z

Tinker:

= About the weirdness that is YouTube naming conventions =
== When looking at a channel page: ==

<nowiki>https://www.youtube.com/@</nowiki>'''Tinker001'''

The '''bold purple''' part is the channel name

== When watching a video: ==

<nowiki>https://www.youtube.com/watch?v=</nowiki>'''y8o7qkmiDso'''

The '''bold purple''' part is the video name

== When watching a playlist: ==

<nowiki>https://www.youtube.com/watch?v=</nowiki>'''oizvS01ovHE'''&list='''PLLG8IgBGD4eUG2n6p3GGfkQ-KPceDuuWj'''&pp=gAQB

The '''bold purple''' part is the playlist name

Bonus fun bit: The '''bold green''' part is actually the name of the video being watched inside that playlist.

= Cookies! =
For the potentially required cookie file, see [https://github.com/yt-dlp/yt-dlp/wiki/Extractors#exporting-youtube-cookies Exporting YouTube cookies]

= One script with multiple personalities =
Yup...

Looks even more complicated. In fact, it might even be considered kinda silly.

(assumes your base for downloading YouTube videos is at '''/mnt/Download_Space/YTDL/''' and contains subfolders named '''_Individual_''' and '''_Playlists_''')

Save this script as '''YTall'''

Then make symbolic links to it.
* <code>ln -s YTall YTc</code>
* <code>ln -s YTall YTv</code>
* <code>ln -s YTall YTl</code>
Call it as '''YTc''' for doing whole channels, '''YTv''' for individual videos, & '''YTl''' for playlists.

<syntaxhighlight lang="Bash" line copy>
#!/bin/bash

BASENAME=$(basename "$0")

echo -e "~~~~~~~ $BASENAME $@ ~~~~~~~ "\\n

case $BASENAME in
"YTv")
TYPE="VideoHash"
DESC="$TYPE = The name YouTube has given to the individual video"
DESC+="\n\\t\\t (This is gonna take some work to explain...)"
DESC+="\n\\t\\t You can specify multiple videos"
;;
"YTc")
TYPE="ChannelName"
DESC="$TYPE = The name of the channel $as defined by YouTube$"
DESC+="\n\\t\\t You can specify multiple channels"
;;
"YTl")
TYPE="ListHash"
DESC="$TYPE = The name YouTube has given to the playlist"
DESC+="\n\\t\\t (This is gonna take some work to explain...)"
DESC+="\n\\t\\t You can specify multiple lists"
;;
*)
TYPE="poop"
DESC="$TYPE = Some crap"
;;
esac

if [ $# -eq 0 ] || [ $1 = "-u" ] || [ $1 = "-?" ]; then
echo -e USAGE: $(basename "$0") \[...option$s$...\] $TYPE $TYPE ...
echo
echo -e OPTIONS:
echo -e \\t -BOB = Grant Style Folder Structure
echo
echo -e \\t -s = simulate this run
echo -e \\t -c = use a cookie file exported from your browser to identify yourself.
echo -e \\t -r RATE = Limit bandwidth
echo -e \\t\\t Maximum download rate in bytes per second,
echo -e \\t\\t\\t e.g. 50K or 4.2M
echo -e \\t -p MIN MAX = Sleep interval $pause$ between downloads $in seconds$
echo -e \\t\\t $will result in a random pause between MIN \& MAX seconds long$
echo -e \\t -e DATE = Earliest video upload date to grab.
echo -e \\t\\t see https://github.com/yt-dlp/yt-dlp?tab=readme-ov-file#video-selection for format info
echo -e \\t\\t $Simplest is YYYYMMDD but fancier options exist$
echo -e \\t $DESC
echo
echo -e The following \"ERROR\"s will also appear in 00000000-ERRORS
echo -e $Unfortunately, without indications of which files caused them...$
echo
echo -e \"ERROR: unable to download video data: HTTP Error 403: Forbidden\"
echo -e \\t Means you may need to supply cookies.
echo -e \\t Often caused by things like age restrictions.
echo -e \\t May also be caused by YouTube noticing you\'ve been downloading a bunch...
case $BASENAME in
"YTc")
echo
echo -e \"ERROR: \[youtube\] oizvS01ovH0: Video unavailable\"
echo -e \\t Most likely caused by a typo in the VideoHash
;;
"YTv")
echo
echo -e \"ERROR: \[youtube:tab\] \@ChannelName: This channel does not have a videos tab\"
echo -e \"ERROR: \[youtube:tab\] \@ChannelName: This channel does not have a shorts tab\"
echo -e \"ERROR: \[youtube:tab\] \@ChannelName: This channel does not have a streams tab\"
echo -e \\t Is simply indicating that the channel doesn\'t have that type of content.
echo -e \\t $All 3 means the channel doesn\'t actually have any content... :P $
;;
"YTl")
echo
echo -e \"mv: cannot stat \'*\': No such file or directory\"
echo -e \\t Don\'t panic. This one\'s just a bit of residue from the playlist folder cleanup
;;
esac
echo

exit
fi

BASEdest=/mnt/Download_Space/YTDL/
COOKIEfile=/mnt/Download_Space/YTDL/www.youtube.com_cookies.txt

SIMULATE=""
WITHcookies=""
LIMITrate=""
PAUSEbetween=""
TIMEspan=""
BOBstyle=""

while (( "$#" ));
do
case $1 in
"-c")
echo Using Cookies: $COOKIEfile
WITHcookies="--cookies "$COOKIEfile
shift
;;

"-s")
echo Simulating
SIMULATE="-s"
shift
;;

"-r")
LIMITrate="-r $2"
echo Rate Limiting $2
shift 2
;;

"-p")
PAUSEbetween="--sleep-interval $2 --max-sleep-interval $3"
echo Random pause between $2 \& $3 seconds
shift 3
;;

"-e")
TIMEspan="--dateafter $2"
echo Time Span $2 to current
shift 2
;;

"-BOB")
BOBstyle="True"
echo Using Grant Style Folder Structure
shift
;;

*)
case $BASENAME in
"YTv") # Video
echo Working on video: $1
echo ========================================
WeirdID=$1
URL=https://www.youtube.com/watch\?v=$1
TYPES=""
DEST="$BASEdest""_Individual_/"
FORMAT="%(uploader_id)s - %(upload_date)s - %(title)s.%(ext)s"
;;

"YTc") # Channel
echo Working on Channel: $1
echo ========================================
WeirdID=$1
URL=https://www.youtube.com/\@$1
TYPES="videos shorts streams"
# TYPES="videos shorts streams playlists podcasts courses"
DEST="$BASEdest""$1"
FORMAT="%(upload_date)s - %(title)s.%(ext)s"
;;

"YTl") # Playlist
echo Working on playlist: $1
echo ========================================
WeirdID=$1
URL=https://www.youtube.com/playlist\?list=$1
TYPES=""
DEST="$BASEdest""_Playlists_/"
FORMAT="%(playlist_index)s_%(playlist_count)s %(uploader_id)s - %(title)s.%(ext)s"
;;
esac

ERRfile=$DEST/00000000-ERRORS
ARCHfile=$DEST/11111111-ARCHIVE

OPTIONS="$SIMULATE $LIMITrate $TIMEspan $WITHcookies $PAUSEbetween"
OPTIONS+=" --no-overwrites --write-description -t mp4"

echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
echo $URL
echo $WeirdID
echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"

if wget --spider --quiet $URL > /dev/null 2>&1; then
# Weirdly, for individual videos, YouTube says the URL is valid even if it doesn't exist when asking about videos...
# Then it gives an error while attempting the download.
# Not fatal, just annoying...

echo -en "(storing in $DEST"

case $BASENAME in
"YTv")
echo ")"
echo VID $WeirdID >> $ARCHfile
cd $DEST
pwd
/usr/local/bin/yt-dlp $OPTIONS --download-archive $ARCHfile -o "$FORMAT" $URL 2> >(/usr/bin/tee -a $ERRfile)
rm -f *.part
;;
"YTl")
echo "$WeirdID)"
echo LIST $WeirdID >> $ARCHfile
cd $DEST
pwd
if [ -d $DEST/$WeirdID ]; then
echo $DEST exists
else
echo no $DEST$WeirdID... Building it...
mkdir $DEST$WeirdID
fi
cd $WeirdID
pwd
/usr/local/bin/yt-dlp $OPTIONS --download-archive $ARCHfile -o "$FORMAT" $URL 2> >(/usr/bin/tee -a $ERRfile)
rm -f *.part

NAME=`find . -maxdepth 1 -regextype posix-extended -regex './0+_.*' -type f `

Listname=`echo ${NAME::-12} | cut -d'@' -f 2`

if [ -d ../"$Listname" ]; then
echo "$Listname" already exists
else
mkdir ../"$Listname"
fi

mv "$NAME" ../"$Listname/000 $Listname.description"
mv * ../"$Listname"
cd ..
rmdir $WeirdID
;;
"YTc")
echo ")"
if [ -d $DEST ]; then
echo $DEST exists
else
echo no $DEST... Building it...
mkdir $DEST
if [ -z $BOBstyle ]; then
for ACK in $TYPES
do
mkdir $DEST/$ACK
done
fi
fi

cd $DEST
pwd

if [ -z $BOBstyle ]; then
echo -e "(using subfolders...)"
else
echo -e "(using Grant format...)"
fi
echo

for ACK in $TYPES
do
echo ">>>>>>>> " $URL/$ACK
if [ -z $BOBstyle ]; then
cd $ACK
echo CHAN $WeirdID $ACK >> $ARCHfile
fi

/usr/local/bin/yt-dlp $OPTIONS --download-archive $ARCHfile -o "$FORMAT" $URL/$ACK 2> >(/usr/bin/tee -a $ERRfile)

# Clear out .part files
rm -f *.part

if [ -z $BOBstyle ]; then
cd ..
fi
done
;;
esac
else
echo YouTube says this channel does not exist.
fi
shift
;;

esac
done
</syntaxhighlight>

= Even more options? =
'''WIP'''

YT-DLP Scripting

2025-10-06T05:05:57Z

Tinker: /* Gettin' Fancy... One script with multiple personalities */

YT-DLP Scripting

2025-10-06T05:05:23Z

Tinker: /* YTvid */

YT-DLP Scripting

2025-10-06T05:05:04Z

Tinker: /* YTchan */

= About the weirdness that is YouTube naming conventions =
When looking at a channel page:

<nowiki>https://www.youtube.com/@</nowiki>'''Tinker001'''

The '''bold purple''' part is the channel name
----
When watching a video:

<nowiki>https://www.youtube.com/watch?v=</nowiki>'''y8o7qkmiDso'''

The '''bold purple''' part is the video name
----
When watching a playlist:

<nowiki>https://www.youtube.com/watch?v=</nowiki>'''oizvS01ovHE'''&list='''PLLG8IgBGD4eUG2n6p3GGfkQ-KPceDuuWj'''&pp=gAQB

The '''bold purple''' part is the playlist name

Bonus fun bit: The '''bold green''' part is actually the name of the video being watched inside that playlist.

----
For the potentially required cookie file, see [https://github.com/yt-dlp/yt-dlp/wiki/Extractors#exporting-youtube-cookies Exporting YouTube cookies]

= YTvid =

A script for downloading separate videos...

(assumes your base for downloading YouTube videos is at '''/mnt/Download_Space/YTDL/''' and creates a subfolder named '''_Individual_''')

For the potentially required cookie file, see [https://github.com/yt-dlp/yt-dlp/wiki/Extractors#exporting-youtube-cookies Exporting YouTube cookies]

<syntaxhighlight lang="Bash" line copy>
#!/bin/bash

echo -e "~~~~~~~ " $0 $@ "~~~~~~~ "\\n

if [ $# -eq 0 ] || [ $1 = "-u" ] || [ $1 = "-?" ]; then
echo -e USAGE: YTvid \[...option$s$...\] VideoHash VideoHash ...
echo
echo -e OPTIONS:
echo -e \\t -BOB = Grant Style Folder Structure
echo
echo -e \\t -s = simulate this run
echo -e \\t -c = use a cookie file exported from your browser to identify yourself.
echo -e \\t -r RATE = Limit bandwidth
echo -e \\t\\t Maximum download rate in bytes per second,
echo -e \\t\\t\\t e.g. 50K or 4.2M
echo -e \\t -p MIN MAX = Sleep interval $pause$ between downloads $in seconds$
echo -e \\t\\t $will result in a random pause between MIN \& MAX seconds long$
echo -e \\t -e DATE = Earliest video upload date to grab.
echo -e \\t\\t see https://github.com/yt-dlp/yt-dlp?tab=readme-ov-file#video-selection for format info
echo -e \\t\\t $Simplest is YYYYMMDD but fancier options exist$
echo -e \\t VideoHash = The name YouTube has given to the individual video
echo -e \\t\\t $This is gonna take some work to explain...$
echo -e \\t\\t You can specify multiple videos
echo
echo -e The following \"ERROR\"s will also appear in 00000000-ERRORS
echo -e $Unfortunately, without indications of which files caused them...$
echo
echo -e \"ERROR: unable to download video data: HTTP Error 403: Forbidden\"
echo -e \\t Means you may need to supply cookies.
echo -e \\t Often caused by things like age restrictions.
echo -e \\t May also be caused by YouTube noticing you\'ve been downloading a bunch...
echo
echo -e \"ERROR: \[youtube\] oizvS01ovH0: Video unavailable\"
echo -e \\t Most likely caused by a typo in the VideoHash
echo

exit
fi

COOKIEfile=/mnt/Download_Space/YTDL/www.youtube.com_cookies.txt

SIMULATE=""
WITHcookies=""
LIMITrate=""
PAUSEbetween=""
TIMEspan=""
BOBstyle=""

while (( "$#" ));
do
if [ ${1} = "-c" ]; then
echo Using Cookies: $COOKIEfile
WITHcookies="--cookies "$COOKIEfile
shift
elif [ ${1} = "-s" ]; then
echo Simulating
SIMULATE="-s"
shift
elif [ ${1} = "-r" ]; then
shift
LIMITrate="-r $1"
shift
echo Rate Limiting $LIMITrate
elif [ ${1} = "-p" ]; then
shift
Pmin=$1
shift
Pmax=$1
PAUSEbetween="--sleep-interval $Pmin --max-sleep-interval $Pmax"
shift
echo Time Span $TIMEspan to current
elif [ ${1} = "-e" ]; then
shift
TIMEspan="--dateafter $1"
shift
echo Time Span $TIMEspan to current
elif [ ${1} = "-BOB" ]; then
echo Grant Style Folder Structure
BOBstyle="True"
shift
else
vidID=$1
shift
echo
echo Working on video: $vidID
echo ========================================

URL=https://www.youtube.com/watch\?v=$vidID

if wget --spider --quiet $URL > /dev/null 2>&1; then
# Weirdly, YouTube says the URL is valid even if it doesn't exist when asking about videos...
# Then it gives an error while attempting the download.
# Not fatal, just annoying...

DEST=/mnt/Download_Space/YTDL/_Individual_/

if [ -d $DEST ]; then
echo $DEST exists
else
echo no $DEST... Building it...
mkdir $DEST
fi

ERRfile=$DEST/00000000-ERRORS
ARCHfile=$DEST/11111111-ARCHIVE

OPTIONS="$SIMULATE $LIMITrate $TIMEspan $WITHcookies $PAUSEbetween"
OPTIONS+=" --no-overwrites --write-description -t mp4"

cd $DEST
echo -e $storing in $DEST$
echo

/usr/local/bin/yt-dlp $OPTIONS --download-archive $ARCHfile -o "%(uploader_id)s - %(upload_date)s - %(title)s.%(ext)s" $URL/$ACK 2> >(/usr/bin/tee -a $ERRfile)

else
echo YouTube says this video does not exist.
fi
fi
done

# Clear out .part files
rm -f *.part
</syntaxhighlight>

= Gettin' Fancy... One script with multiple personalities =
Yup...

Looks even more complicated. In fact, it might even be considered kinda silly.

(assumes your base for downloading YouTube videos is at '''/mnt/Download_Space/YTDL/''' and contains subfolders named '''_Individual_''' and '''_Playlists_''')

Save this script as '''YTall'''

Then make symbolic links to it.
* <code>ln -s YTall YTc</code>
* <code>ln -s YTall YTv</code>
* <code>ln -s YTall YTl</code>
Call it as '''YTc''' for doing whole channels, '''YTv''' for individual videos, & '''YTl''' for playlists.

<syntaxhighlight lang="Bash" line copy>
#!/bin/bash

BASENAME=$(basename "$0")

echo -e "~~~~~~~ $BASENAME $@ ~~~~~~~ "\\n

case $BASENAME in
"YTv")
TYPE="VideoHash"
DESC="$TYPE = The name YouTube has given to the individual video"
DESC+="\n\\t\\t (This is gonna take some work to explain...)"
DESC+="\n\\t\\t You can specify multiple videos"
;;
"YTc")
TYPE="ChannelName"
DESC="$TYPE = The name of the channel $as defined by YouTube$"
DESC+="\n\\t\\t You can specify multiple channels"
;;
"YTl")
TYPE="ListHash"
DESC="$TYPE = The name YouTube has given to the playlist"
DESC+="\n\\t\\t (This is gonna take some work to explain...)"
DESC+="\n\\t\\t You can specify multiple lists"
;;
*)
TYPE="poop"
DESC="$TYPE = Some crap"
;;
esac

if [ $# -eq 0 ] || [ $1 = "-u" ] || [ $1 = "-?" ]; then
echo -e USAGE: $(basename "$0") \[...option$s$...\] $TYPE $TYPE ...
echo
echo -e OPTIONS:
echo -e \\t -BOB = Grant Style Folder Structure
echo
echo -e \\t -s = simulate this run
echo -e \\t -c = use a cookie file exported from your browser to identify yourself.
echo -e \\t -r RATE = Limit bandwidth
echo -e \\t\\t Maximum download rate in bytes per second,
echo -e \\t\\t\\t e.g. 50K or 4.2M
echo -e \\t -p MIN MAX = Sleep interval $pause$ between downloads $in seconds$
echo -e \\t\\t $will result in a random pause between MIN \& MAX seconds long$
echo -e \\t -e DATE = Earliest video upload date to grab.
echo -e \\t\\t see https://github.com/yt-dlp/yt-dlp?tab=readme-ov-file#video-selection for format info
echo -e \\t\\t $Simplest is YYYYMMDD but fancier options exist$
echo -e \\t $DESC
echo
echo -e The following \"ERROR\"s will also appear in 00000000-ERRORS
echo -e $Unfortunately, without indications of which files caused them...$
echo
echo -e \"ERROR: unable to download video data: HTTP Error 403: Forbidden\"
echo -e \\t Means you may need to supply cookies.
echo -e \\t Often caused by things like age restrictions.
echo -e \\t May also be caused by YouTube noticing you\'ve been downloading a bunch...
case $BASENAME in
"YTc")
echo
echo -e \"ERROR: \[youtube\] oizvS01ovH0: Video unavailable\"
echo -e \\t Most likely caused by a typo in the VideoHash
;;
"YTv")
echo
echo -e \"ERROR: \[youtube:tab\] \@ChannelName: This channel does not have a videos tab\"
echo -e \"ERROR: \[youtube:tab\] \@ChannelName: This channel does not have a shorts tab\"
echo -e \"ERROR: \[youtube:tab\] \@ChannelName: This channel does not have a streams tab\"
echo -e \\t Is simply indicating that the channel doesn\'t have that type of content.
echo -e \\t $All 3 means the channel doesn\'t actually have any content... :P $
;;
"YTl")
echo
echo -e \"mv: cannot stat \'*\': No such file or directory\"
echo -e \\t Don\'t panic. This one\'s just a bit of residue from the playlist folder cleanup
;;
esac
echo

exit
fi

BASEdest=/mnt/Download_Space/YTDL/
COOKIEfile=/mnt/Download_Space/YTDL/www.youtube.com_cookies.txt

SIMULATE=""
WITHcookies=""
LIMITrate=""
PAUSEbetween=""
TIMEspan=""
BOBstyle=""

while (( "$#" ));
do
case $1 in
"-c")
echo Using Cookies: $COOKIEfile
WITHcookies="--cookies "$COOKIEfile
shift
;;

"-s")
echo Simulating
SIMULATE="-s"
shift
;;

"-r")
LIMITrate="-r $2"
echo Rate Limiting $2
shift 2
;;

"-p")
PAUSEbetween="--sleep-interval $2 --max-sleep-interval $3"
echo Random pause between $2 \& $3 seconds
shift 3
;;

"-e")
TIMEspan="--dateafter $2"
echo Time Span $2 to current
shift 2
;;

"-BOB")
BOBstyle="True"
echo Using Grant Style Folder Structure
shift
;;

*)
case $BASENAME in
"YTv") # Video
echo Working on video: $1
echo ========================================
WeirdID=$1
URL=https://www.youtube.com/watch\?v=$1
TYPES=""
DEST="$BASEdest""_Individual_/"
FORMAT="%(uploader_id)s - %(upload_date)s - %(title)s.%(ext)s"
;;

"YTc") # Channel
echo Working on Channel: $1
echo ========================================
WeirdID=$1
URL=https://www.youtube.com/\@$1
TYPES="videos shorts streams"
# TYPES="videos shorts streams playlists podcasts courses"
DEST="$BASEdest""$1"
FORMAT="%(upload_date)s - %(title)s.%(ext)s"
;;

"YTl") # Playlist
echo Working on playlist: $1
echo ========================================
WeirdID=$1
URL=https://www.youtube.com/playlist\?list=$1
TYPES=""
DEST="$BASEdest""_Playlists_/"
FORMAT="%(playlist_index)s_%(playlist_count)s %(uploader_id)s - %(title)s.%(ext)s"
;;
esac

ERRfile=$DEST/00000000-ERRORS
ARCHfile=$DEST/11111111-ARCHIVE

OPTIONS="$SIMULATE $LIMITrate $TIMEspan $WITHcookies $PAUSEbetween"
OPTIONS+=" --no-overwrites --write-description -t mp4"

echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
echo $URL
echo $WeirdID
echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"

if wget --spider --quiet $URL > /dev/null 2>&1; then
# Weirdly, for individual videos, YouTube says the URL is valid even if it doesn't exist when asking about videos...
# Then it gives an error while attempting the download.
# Not fatal, just annoying...

echo -en "(storing in $DEST"

case $BASENAME in
"YTv")
echo ")"
echo VID $WeirdID >> $ARCHfile
cd $DEST
pwd
/usr/local/bin/yt-dlp $OPTIONS --download-archive $ARCHfile -o "$FORMAT" $URL 2> >(/usr/bin/tee -a $ERRfile)
rm -f *.part
;;
"YTl")
echo "$WeirdID)"
echo LIST $WeirdID >> $ARCHfile
cd $DEST
pwd
if [ -d $DEST/$WeirdID ]; then
echo $DEST exists
else
echo no $DEST$WeirdID... Building it...
mkdir $DEST$WeirdID
fi
cd $WeirdID
pwd
/usr/local/bin/yt-dlp $OPTIONS --download-archive $ARCHfile -o "$FORMAT" $URL 2> >(/usr/bin/tee -a $ERRfile)
rm -f *.part

NAME=`find . -maxdepth 1 -regextype posix-extended -regex './0+_.*' -type f `

Listname=`echo ${NAME::-12} | cut -d'@' -f 2`

if [ -d ../"$Listname" ]; then
echo "$Listname" already exists
else
mkdir ../"$Listname"
fi

mv "$NAME" ../"$Listname/000 $Listname.description"
mv * ../"$Listname"
cd ..
rmdir $WeirdID
;;
"YTc")
echo ")"
if [ -d $DEST ]; then
echo $DEST exists
else
echo no $DEST... Building it...
mkdir $DEST
if [ -z $BOBstyle ]; then
for ACK in $TYPES
do
mkdir $DEST/$ACK
done
fi
fi

cd $DEST
pwd

if [ -z $BOBstyle ]; then
echo -e "(using subfolders...)"
else
echo -e "(using Grant format...)"
fi
echo

for ACK in $TYPES
do
echo ">>>>>>>> " $URL/$ACK
if [ -z $BOBstyle ]; then
cd $ACK
echo CHAN $WeirdID $ACK >> $ARCHfile
fi

/usr/local/bin/yt-dlp $OPTIONS --download-archive $ARCHfile -o "$FORMAT" $URL/$ACK 2> >(/usr/bin/tee -a $ERRfile)

# Clear out .part files
rm -f *.part

if [ -z $BOBstyle ]; then
cd ..
fi
done
;;
esac
else
echo YouTube says this channel does not exist.
fi
shift
;;

esac
done
</syntaxhighlight>

= Even more options? =
'''WIP'''

YT-DLP Scripting

2025-10-06T04:33:22Z

Tinker: /* About the weirdness that is YouTube naming conventions */

= About the weirdness that is YouTube naming conventions =
When looking at a channel page:

<nowiki>https://www.youtube.com/@</nowiki>'''Tinker001'''

The '''bold purple''' part is the channel name
----
When watching a video:

<nowiki>https://www.youtube.com/watch?v=</nowiki>'''y8o7qkmiDso'''

The '''bold purple''' part is the video name
----
When watching a playlist:

<nowiki>https://www.youtube.com/watch?v=</nowiki>'''oizvS01ovHE'''&list='''PLLG8IgBGD4eUG2n6p3GGfkQ-KPceDuuWj'''&pp=gAQB

The '''bold purple''' part is the playlist name

Bonus fun bit: The '''bold green''' part is actually the name of the video being watched inside that playlist.

= YTchan =

A script for downloading entire channels...

(assumes your base for downloading YouTube videos is at '''/mnt/Download_Space/YTDL/''' and creates a subfolder using the channel name(s) )

For the potentially required cookie file, see [https://github.com/yt-dlp/yt-dlp/wiki/Extractors#exporting-youtube-cookies Exporting YouTube cookies]

<syntaxhighlight lang="Bash" line copy>
#!/bin/bash

echo -e "~~~~~~~ " $0 $@ "~~~~~~~ "\\n

if [ $# -eq 0 ] || [ $1 = "-u" ] || [ $1 = "-?" ]; then
echo -e USAGE: YTchan \[...option$s$...\] ChannelName ChannelName ...
echo
echo -e OPTIONS:
echo -e \\t -BOB = Grant Style Folder Structure
echo
echo -e \\t -s = simulate this run
echo -e \\t -c = use a cookie file exported from your browser to identify yourself.
echo -e \\t -r RATE = Limit bandwidth
echo -e \\t\\t Maximum download rate in bytes per second,
echo -e \\t\\t\\t e.g. 50K or 4.2M
echo -e \\t -p MIN MAX = Sleep interval $pause$ between downloads $in seconds$
echo -e \\t\\t $will result in a random pause between MIN \& MAX seconds long$
echo -e \\t -e DATE = Earliest video upload date to grab.
echo -e \\t\\t see https://github.com/yt-dlp/yt-dlp?tab=readme-ov-file#video-selection for format info
echo -e \\t\\t $Simplest is YYYYMMDD but fancier options exist$
echo -e \\t ChannelName = The name of the channel $as defined by YouTube$
echo -e \\t\\t You can specify multiple channels
echo
echo -e The following \"ERROR\"s will also appear in 00000000-ERRORS
echo -e $Unfortunately, without indications of which files caused them...$
echo
echo -e \"ERROR: \[youtube:tab\] \@ChannelName: This channel does not have a videos tab\"
echo -e \"ERROR: \[youtube:tab\] \@ChannelName: This channel does not have a shorts tab\"
echo -e \"ERROR: \[youtube:tab\] \@ChannelName: This channel does not have a streams tab\"
echo -e \\t Is simply indicating that the channel doesn\'t have that type of content.
echo -e \\t $All 3 means the channel doesn\'t actually have any content... :P $
echo
echo -e \"ERROR: unable to download video data: HTTP Error 403: Forbidden\"
echo -e \\t Means you may need to supply cookies.
echo -e \\t Often caused by things like age restrictions.
echo -e \\t May also be caused by YouTube noticing you\'ve been downloading a bunch...
echo

exit
fi

#TYPES="videos"
TYPES="videos shorts streams"
COOKIEfile=/mnt/Download_Space/YTDL/www.youtube.com_cookies.txt

SIMULATE=""
WITHcookies=""
LIMITrate=""
PAUSEbetween=""
TIMEspan=""
BOBstyle=""

while (( "$#" ));
do
if [ ${1} = "-c" ]; then
echo Using Cookies: $COOKIEfile
WITHcookies="--cookies "$COOKIEfile
shift
elif [ ${1} = "-s" ]; then
echo Simulating
SIMULATE="-s"
shift
elif [ ${1} = "-r" ]; then
shift
LIMITrate="-r $1"
shift
echo Rate Limiting $LIMITrate
elif [ ${1} = "-p" ]; then
shift
Pmin=$1
shift
Pmax=$1
PAUSEbetween="--sleep-interval $Pmin --max-sleep-interval $Pmax"
shift
echo Time Span $TIMEspan to current
elif [ ${1} = "-e" ]; then
shift
TIMEspan="--dateafter $1"
shift
echo Time Span $TIMEspan to current
elif [ ${1} = "-BOB" ]; then
echo Grant Style Folder Structure
BOBstyle="True"
shift
else
CHANNEL=$1
shift
echo
echo Working on channel: $CHANNEL
echo ========================================

URL=https://www.youtube.com/\@$CHANNEL

if wget --spider --quiet $URL > /dev/null 2>&1; then

DEST=/mnt/Download_Space/YTDL/$CHANNEL

ERRfile=$DEST/00000000-ERRORS
ARCHfile=$DEST/11111111-ARCHIVE

if [ -d $DEST ]; then
echo $DEST exists
else
echo no $DEST... Building it...
mkdir $DEST
if [ -z $BOBstyle ]; then
for ACK in $TYPES
do
mkdir $DEST/$ACK
done
fi
fi

OPTIONS="$SIMULATE $LIMITrate $TIMEspan $WITHcookies $PAUSEbetween"
OPTIONS+=" --no-overwrites --write-description -t mp4"

cd $DEST
echo -e $storing in $DEST$
echo

for ACK in $TYPES
do
echo ">>>>>>>> " $URL/$ACK
if [ -z $BOBstyle ]; then
cd $ACK
fi
/usr/local/bin/yt-dlp $OPTIONS --download-archive $ARCHfile -o "%(upload_date)s - %(title)s.%(ext)s" $URL/$ACK 2> >(/usr/bin/tee -a $ERRfile)

# Clear out .part files
rm -f *.part

if [ -z $BOBstyle ]; then
cd ..
fi
done

else
echo YouTube says this channel does not exist.
fi
fi
done
</syntaxhighlight>

= YTvid =

A script for downloading separate videos...

(assumes your base for downloading YouTube videos is at '''/mnt/Download_Space/YTDL/''' and creates a subfolder named '''_Individual_''')

For the potentially required cookie file, see [https://github.com/yt-dlp/yt-dlp/wiki/Extractors#exporting-youtube-cookies Exporting YouTube cookies]

<syntaxhighlight lang="Bash" line copy>
#!/bin/bash

echo -e "~~~~~~~ " $0 $@ "~~~~~~~ "\\n

if [ $# -eq 0 ] || [ $1 = "-u" ] || [ $1 = "-?" ]; then
echo -e USAGE: YTvid \[...option$s$...\] VideoHash VideoHash ...
echo
echo -e OPTIONS:
echo -e \\t -BOB = Grant Style Folder Structure
echo
echo -e \\t -s = simulate this run
echo -e \\t -c = use a cookie file exported from your browser to identify yourself.
echo -e \\t -r RATE = Limit bandwidth
echo -e \\t\\t Maximum download rate in bytes per second,
echo -e \\t\\t\\t e.g. 50K or 4.2M
echo -e \\t -p MIN MAX = Sleep interval $pause$ between downloads $in seconds$
echo -e \\t\\t $will result in a random pause between MIN \& MAX seconds long$
echo -e \\t -e DATE = Earliest video upload date to grab.
echo -e \\t\\t see https://github.com/yt-dlp/yt-dlp?tab=readme-ov-file#video-selection for format info
echo -e \\t\\t $Simplest is YYYYMMDD but fancier options exist$
echo -e \\t VideoHash = The name YouTube has given to the individual video
echo -e \\t\\t $This is gonna take some work to explain...$
echo -e \\t\\t You can specify multiple videos
echo
echo -e The following \"ERROR\"s will also appear in 00000000-ERRORS
echo -e $Unfortunately, without indications of which files caused them...$
echo
echo -e \"ERROR: unable to download video data: HTTP Error 403: Forbidden\"
echo -e \\t Means you may need to supply cookies.
echo -e \\t Often caused by things like age restrictions.
echo -e \\t May also be caused by YouTube noticing you\'ve been downloading a bunch...
echo
echo -e \"ERROR: \[youtube\] oizvS01ovH0: Video unavailable\"
echo -e \\t Most likely caused by a typo in the VideoHash
echo

exit
fi

COOKIEfile=/mnt/Download_Space/YTDL/www.youtube.com_cookies.txt

SIMULATE=""
WITHcookies=""
LIMITrate=""
PAUSEbetween=""
TIMEspan=""
BOBstyle=""

while (( "$#" ));
do
if [ ${1} = "-c" ]; then
echo Using Cookies: $COOKIEfile
WITHcookies="--cookies "$COOKIEfile
shift
elif [ ${1} = "-s" ]; then
echo Simulating
SIMULATE="-s"
shift
elif [ ${1} = "-r" ]; then
shift
LIMITrate="-r $1"
shift
echo Rate Limiting $LIMITrate
elif [ ${1} = "-p" ]; then
shift
Pmin=$1
shift
Pmax=$1
PAUSEbetween="--sleep-interval $Pmin --max-sleep-interval $Pmax"
shift
echo Time Span $TIMEspan to current
elif [ ${1} = "-e" ]; then
shift
TIMEspan="--dateafter $1"
shift
echo Time Span $TIMEspan to current
elif [ ${1} = "-BOB" ]; then
echo Grant Style Folder Structure
BOBstyle="True"
shift
else
vidID=$1
shift
echo
echo Working on video: $vidID
echo ========================================

URL=https://www.youtube.com/watch\?v=$vidID

if wget --spider --quiet $URL > /dev/null 2>&1; then
# Weirdly, YouTube says the URL is valid even if it doesn't exist when asking about videos...
# Then it gives an error while attempting the download.
# Not fatal, just annoying...

DEST=/mnt/Download_Space/YTDL/_Individual_/

if [ -d $DEST ]; then
echo $DEST exists
else
echo no $DEST... Building it...
mkdir $DEST
fi

ERRfile=$DEST/00000000-ERRORS
ARCHfile=$DEST/11111111-ARCHIVE

OPTIONS="$SIMULATE $LIMITrate $TIMEspan $WITHcookies $PAUSEbetween"
OPTIONS+=" --no-overwrites --write-description -t mp4"

cd $DEST
echo -e $storing in $DEST$
echo

/usr/local/bin/yt-dlp $OPTIONS --download-archive $ARCHfile -o "%(uploader_id)s - %(upload_date)s - %(title)s.%(ext)s" $URL/$ACK 2> >(/usr/bin/tee -a $ERRfile)

else
echo YouTube says this video does not exist.
fi
fi
done

# Clear out .part files
rm -f *.part
</syntaxhighlight>

= Gettin' Fancy... One script with multiple personalities =
Yup...

Looks even more complicated. In fact, it might even be considered kinda silly.

(assumes your base for downloading YouTube videos is at '''/mnt/Download_Space/YTDL/''' and contains subfolders named '''_Individual_''' and '''_Playlists_''')

Save this script as '''YTall'''

Then make symbolic links to it.
* <code>ln -s YTall YTc</code>
* <code>ln -s YTall YTv</code>
* <code>ln -s YTall YTl</code>
Call it as '''YTc''' for doing whole channels, '''YTv''' for individual videos, & '''YTl''' for playlists.

<syntaxhighlight lang="Bash" line copy>
#!/bin/bash

BASENAME=$(basename "$0")

echo -e "~~~~~~~ $BASENAME $@ ~~~~~~~ "\\n

case $BASENAME in
"YTv")
TYPE="VideoHash"
DESC="$TYPE = The name YouTube has given to the individual video"
DESC+="\n\\t\\t (This is gonna take some work to explain...)"
DESC+="\n\\t\\t You can specify multiple videos"
;;
"YTc")
TYPE="ChannelName"
DESC="$TYPE = The name of the channel $as defined by YouTube$"
DESC+="\n\\t\\t You can specify multiple channels"
;;
"YTl")
TYPE="ListHash"
DESC="$TYPE = The name YouTube has given to the playlist"
DESC+="\n\\t\\t (This is gonna take some work to explain...)"
DESC+="\n\\t\\t You can specify multiple lists"
;;
*)
TYPE="poop"
DESC="$TYPE = Some crap"
;;
esac

if [ $# -eq 0 ] || [ $1 = "-u" ] || [ $1 = "-?" ]; then
echo -e USAGE: $(basename "$0") \[...option$s$...\] $TYPE $TYPE ...
echo
echo -e OPTIONS:
echo -e \\t -BOB = Grant Style Folder Structure
echo
echo -e \\t -s = simulate this run
echo -e \\t -c = use a cookie file exported from your browser to identify yourself.
echo -e \\t -r RATE = Limit bandwidth
echo -e \\t\\t Maximum download rate in bytes per second,
echo -e \\t\\t\\t e.g. 50K or 4.2M
echo -e \\t -p MIN MAX = Sleep interval $pause$ between downloads $in seconds$
echo -e \\t\\t $will result in a random pause between MIN \& MAX seconds long$
echo -e \\t -e DATE = Earliest video upload date to grab.
echo -e \\t\\t see https://github.com/yt-dlp/yt-dlp?tab=readme-ov-file#video-selection for format info
echo -e \\t\\t $Simplest is YYYYMMDD but fancier options exist$
echo -e \\t $DESC
echo
echo -e The following \"ERROR\"s will also appear in 00000000-ERRORS
echo -e $Unfortunately, without indications of which files caused them...$
echo
echo -e \"ERROR: unable to download video data: HTTP Error 403: Forbidden\"
echo -e \\t Means you may need to supply cookies.
echo -e \\t Often caused by things like age restrictions.
echo -e \\t May also be caused by YouTube noticing you\'ve been downloading a bunch...
case $BASENAME in
"YTc")
echo
echo -e \"ERROR: \[youtube\] oizvS01ovH0: Video unavailable\"
echo -e \\t Most likely caused by a typo in the VideoHash
;;
"YTv")
echo
echo -e \"ERROR: \[youtube:tab\] \@ChannelName: This channel does not have a videos tab\"
echo -e \"ERROR: \[youtube:tab\] \@ChannelName: This channel does not have a shorts tab\"
echo -e \"ERROR: \[youtube:tab\] \@ChannelName: This channel does not have a streams tab\"
echo -e \\t Is simply indicating that the channel doesn\'t have that type of content.
echo -e \\t $All 3 means the channel doesn\'t actually have any content... :P $
;;
"YTl")
echo
echo -e \"mv: cannot stat \'*\': No such file or directory\"
echo -e \\t Don\'t panic. This one\'s just a bit of residue from the playlist folder cleanup
;;
esac
echo

exit
fi

BASEdest=/mnt/Download_Space/YTDL/
COOKIEfile=/mnt/Download_Space/YTDL/www.youtube.com_cookies.txt

SIMULATE=""
WITHcookies=""
LIMITrate=""
PAUSEbetween=""
TIMEspan=""
BOBstyle=""

while (( "$#" ));
do
case $1 in
"-c")
echo Using Cookies: $COOKIEfile
WITHcookies="--cookies "$COOKIEfile
shift
;;

"-s")
echo Simulating
SIMULATE="-s"
shift
;;

"-r")
LIMITrate="-r $2"
echo Rate Limiting $2
shift 2
;;

"-p")
PAUSEbetween="--sleep-interval $2 --max-sleep-interval $3"
echo Random pause between $2 \& $3 seconds
shift 3
;;

"-e")
TIMEspan="--dateafter $2"
echo Time Span $2 to current
shift 2
;;

"-BOB")
BOBstyle="True"
echo Using Grant Style Folder Structure
shift
;;

*)
case $BASENAME in
"YTv") # Video
echo Working on video: $1
echo ========================================
WeirdID=$1
URL=https://www.youtube.com/watch\?v=$1
TYPES=""
DEST="$BASEdest""_Individual_/"
FORMAT="%(uploader_id)s - %(upload_date)s - %(title)s.%(ext)s"
;;

"YTc") # Channel
echo Working on Channel: $1
echo ========================================
WeirdID=$1
URL=https://www.youtube.com/\@$1
TYPES="videos shorts streams"
# TYPES="videos shorts streams playlists podcasts courses"
DEST="$BASEdest""$1"
FORMAT="%(upload_date)s - %(title)s.%(ext)s"
;;

"YTl") # Playlist
echo Working on playlist: $1
echo ========================================
WeirdID=$1
URL=https://www.youtube.com/playlist\?list=$1
TYPES=""
DEST="$BASEdest""_Playlists_/"
FORMAT="%(playlist_index)s_%(playlist_count)s %(uploader_id)s - %(title)s.%(ext)s"
;;
esac

ERRfile=$DEST/00000000-ERRORS
ARCHfile=$DEST/11111111-ARCHIVE

OPTIONS="$SIMULATE $LIMITrate $TIMEspan $WITHcookies $PAUSEbetween"
OPTIONS+=" --no-overwrites --write-description -t mp4"

echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
echo $URL
echo $WeirdID
echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"

if wget --spider --quiet $URL > /dev/null 2>&1; then
# Weirdly, for individual videos, YouTube says the URL is valid even if it doesn't exist when asking about videos...
# Then it gives an error while attempting the download.
# Not fatal, just annoying...

echo -en "(storing in $DEST"

case $BASENAME in
"YTv")
echo ")"
echo VID $WeirdID >> $ARCHfile
cd $DEST
pwd
/usr/local/bin/yt-dlp $OPTIONS --download-archive $ARCHfile -o "$FORMAT" $URL 2> >(/usr/bin/tee -a $ERRfile)
rm -f *.part
;;
"YTl")
echo "$WeirdID)"
echo LIST $WeirdID >> $ARCHfile
cd $DEST
pwd
if [ -d $DEST/$WeirdID ]; then
echo $DEST exists
else
echo no $DEST$WeirdID... Building it...
mkdir $DEST$WeirdID
fi
cd $WeirdID
pwd
/usr/local/bin/yt-dlp $OPTIONS --download-archive $ARCHfile -o "$FORMAT" $URL 2> >(/usr/bin/tee -a $ERRfile)
rm -f *.part

NAME=`find . -maxdepth 1 -regextype posix-extended -regex './0+_.*' -type f `

Listname=`echo ${NAME::-12} | cut -d'@' -f 2`

if [ -d ../"$Listname" ]; then
echo "$Listname" already exists
else
mkdir ../"$Listname"
fi

mv "$NAME" ../"$Listname/000 $Listname.description"
mv * ../"$Listname"
cd ..
rmdir $WeirdID
;;
"YTc")
echo ")"
if [ -d $DEST ]; then
echo $DEST exists
else
echo no $DEST... Building it...
mkdir $DEST
if [ -z $BOBstyle ]; then
for ACK in $TYPES
do
mkdir $DEST/$ACK
done
fi
fi

cd $DEST
pwd

if [ -z $BOBstyle ]; then
echo -e "(using subfolders...)"
else
echo -e "(using Grant format...)"
fi
echo

for ACK in $TYPES
do
echo ">>>>>>>> " $URL/$ACK
if [ -z $BOBstyle ]; then
cd $ACK
echo CHAN $WeirdID $ACK >> $ARCHfile
fi

/usr/local/bin/yt-dlp $OPTIONS --download-archive $ARCHfile -o "$FORMAT" $URL/$ACK 2> >(/usr/bin/tee -a $ERRfile)

# Clear out .part files
rm -f *.part

if [ -z $BOBstyle ]; then
cd ..
fi
done
;;
esac
else
echo YouTube says this channel does not exist.
fi
shift
;;

esac
done
</syntaxhighlight>

= Even more options? =
'''WIP'''

YT-DLP Scripting

2025-10-06T04:33:06Z

Tinker: /* YTchan */

= About the weirdness that is YouTube naming conventions =
When looking at a channel page:

<nowiki>https://www.youtube.com/@</nowiki>'''Tinker001'''

The '''bold purple''' part is the channel name
----
When watching a video:

<nowiki>https://www.youtube.com/watch?v=</nowiki>'''y8o7qkmiDso'''

The '''bold purple''' part is the video name
----
When watching a playlist:

<nowiki>https://www.youtube.com/watch?v=</nowiki>'''oizvS01ovHE'''&list='''PLLG8IgBGD4eUG2n6p3GGfkQ-KPceDuuWj'''&pp=gAQB

The '''bold purple''' part is the playlist name

Bonus fun bit: The '''bold green''' part is actually the name of the video being watched inside that playlist.

= YTchan =

A script for downloading entire channels...

(assumes your base for downloading YouTube videos is at '''/mnt/Download_Space/YTDL/''' and creates a subfolder using the channel name(s) )

For the potentially required cookie file, see [https://github.com/yt-dlp/yt-dlp/wiki/Extractors#exporting-youtube-cookies Exporting YouTube cookies]

<syntaxhighlight lang="Bash" line copy>
#!/bin/bash

echo -e "~~~~~~~ " $0 $@ "~~~~~~~ "\\n

if [ $# -eq 0 ] || [ $1 = "-u" ] || [ $1 = "-?" ]; then
echo -e USAGE: YTchan \[...option$s$...\] ChannelName ChannelName ...
echo
echo -e OPTIONS:
echo -e \\t -BOB = Grant Style Folder Structure
echo
echo -e \\t -s = simulate this run
echo -e \\t -c = use a cookie file exported from your browser to identify yourself.
echo -e \\t -r RATE = Limit bandwidth
echo -e \\t\\t Maximum download rate in bytes per second,
echo -e \\t\\t\\t e.g. 50K or 4.2M
echo -e \\t -p MIN MAX = Sleep interval $pause$ between downloads $in seconds$
echo -e \\t\\t $will result in a random pause between MIN \& MAX seconds long$
echo -e \\t -e DATE = Earliest video upload date to grab.
echo -e \\t\\t see https://github.com/yt-dlp/yt-dlp?tab=readme-ov-file#video-selection for format info
echo -e \\t\\t $Simplest is YYYYMMDD but fancier options exist$
echo -e \\t ChannelName = The name of the channel $as defined by YouTube$
echo -e \\t\\t You can specify multiple channels
echo
echo -e The following \"ERROR\"s will also appear in 00000000-ERRORS
echo -e $Unfortunately, without indications of which files caused them...$
echo
echo -e \"ERROR: \[youtube:tab\] \@ChannelName: This channel does not have a videos tab\"
echo -e \"ERROR: \[youtube:tab\] \@ChannelName: This channel does not have a shorts tab\"
echo -e \"ERROR: \[youtube:tab\] \@ChannelName: This channel does not have a streams tab\"
echo -e \\t Is simply indicating that the channel doesn\'t have that type of content.
echo -e \\t $All 3 means the channel doesn\'t actually have any content... :P $
echo
echo -e \"ERROR: unable to download video data: HTTP Error 403: Forbidden\"
echo -e \\t Means you may need to supply cookies.
echo -e \\t Often caused by things like age restrictions.
echo -e \\t May also be caused by YouTube noticing you\'ve been downloading a bunch...
echo

exit
fi

#TYPES="videos"
TYPES="videos shorts streams"
COOKIEfile=/mnt/Download_Space/YTDL/www.youtube.com_cookies.txt

SIMULATE=""
WITHcookies=""
LIMITrate=""
PAUSEbetween=""
TIMEspan=""
BOBstyle=""

while (( "$#" ));
do
if [ ${1} = "-c" ]; then
echo Using Cookies: $COOKIEfile
WITHcookies="--cookies "$COOKIEfile
shift
elif [ ${1} = "-s" ]; then
echo Simulating
SIMULATE="-s"
shift
elif [ ${1} = "-r" ]; then
shift
LIMITrate="-r $1"
shift
echo Rate Limiting $LIMITrate
elif [ ${1} = "-p" ]; then
shift
Pmin=$1
shift
Pmax=$1
PAUSEbetween="--sleep-interval $Pmin --max-sleep-interval $Pmax"
shift
echo Time Span $TIMEspan to current
elif [ ${1} = "-e" ]; then
shift
TIMEspan="--dateafter $1"
shift
echo Time Span $TIMEspan to current
elif [ ${1} = "-BOB" ]; then
echo Grant Style Folder Structure
BOBstyle="True"
shift
else
CHANNEL=$1
shift
echo
echo Working on channel: $CHANNEL
echo ========================================

URL=https://www.youtube.com/\@$CHANNEL

if wget --spider --quiet $URL > /dev/null 2>&1; then

DEST=/mnt/Download_Space/YTDL/$CHANNEL

ERRfile=$DEST/00000000-ERRORS
ARCHfile=$DEST/11111111-ARCHIVE

if [ -d $DEST ]; then
echo $DEST exists
else
echo no $DEST... Building it...
mkdir $DEST
if [ -z $BOBstyle ]; then
for ACK in $TYPES
do
mkdir $DEST/$ACK
done
fi
fi

OPTIONS="$SIMULATE $LIMITrate $TIMEspan $WITHcookies $PAUSEbetween"
OPTIONS+=" --no-overwrites --write-description -t mp4"

cd $DEST
echo -e $storing in $DEST$
echo

for ACK in $TYPES
do
echo ">>>>>>>> " $URL/$ACK
if [ -z $BOBstyle ]; then
cd $ACK
fi
/usr/local/bin/yt-dlp $OPTIONS --download-archive $ARCHfile -o "%(upload_date)s - %(title)s.%(ext)s" $URL/$ACK 2> >(/usr/bin/tee -a $ERRfile)

# Clear out .part files
rm -f *.part

if [ -z $BOBstyle ]; then
cd ..
fi
done

else
echo YouTube says this channel does not exist.
fi
fi
done
</syntaxhighlight>

= YTvid =

A script for downloading separate videos...

(assumes your base for downloading YouTube videos is at '''/mnt/Download_Space/YTDL/''' and creates a subfolder named '''_Individual_''')

For the potentially required cookie file, see [https://github.com/yt-dlp/yt-dlp/wiki/Extractors#exporting-youtube-cookies Exporting YouTube cookies]

<syntaxhighlight lang="Bash" line copy>
#!/bin/bash

echo -e "~~~~~~~ " $0 $@ "~~~~~~~ "\\n

if [ $# -eq 0 ] || [ $1 = "-u" ] || [ $1 = "-?" ]; then
echo -e USAGE: YTvid \[...option$s$...\] VideoHash VideoHash ...
echo
echo -e OPTIONS:
echo -e \\t -BOB = Grant Style Folder Structure
echo
echo -e \\t -s = simulate this run
echo -e \\t -c = use a cookie file exported from your browser to identify yourself.
echo -e \\t -r RATE = Limit bandwidth
echo -e \\t\\t Maximum download rate in bytes per second,
echo -e \\t\\t\\t e.g. 50K or 4.2M
echo -e \\t -p MIN MAX = Sleep interval $pause$ between downloads $in seconds$
echo -e \\t\\t $will result in a random pause between MIN \& MAX seconds long$
echo -e \\t -e DATE = Earliest video upload date to grab.
echo -e \\t\\t see https://github.com/yt-dlp/yt-dlp?tab=readme-ov-file#video-selection for format info
echo -e \\t\\t $Simplest is YYYYMMDD but fancier options exist$
echo -e \\t VideoHash = The name YouTube has given to the individual video
echo -e \\t\\t $This is gonna take some work to explain...$
echo -e \\t\\t You can specify multiple videos
echo
echo -e The following \"ERROR\"s will also appear in 00000000-ERRORS
echo -e $Unfortunately, without indications of which files caused them...$
echo
echo -e \"ERROR: unable to download video data: HTTP Error 403: Forbidden\"
echo -e \\t Means you may need to supply cookies.
echo -e \\t Often caused by things like age restrictions.
echo -e \\t May also be caused by YouTube noticing you\'ve been downloading a bunch...
echo
echo -e \"ERROR: \[youtube\] oizvS01ovH0: Video unavailable\"
echo -e \\t Most likely caused by a typo in the VideoHash
echo

exit
fi

COOKIEfile=/mnt/Download_Space/YTDL/www.youtube.com_cookies.txt

SIMULATE=""
WITHcookies=""
LIMITrate=""
PAUSEbetween=""
TIMEspan=""
BOBstyle=""

while (( "$#" ));
do
if [ ${1} = "-c" ]; then
echo Using Cookies: $COOKIEfile
WITHcookies="--cookies "$COOKIEfile
shift
elif [ ${1} = "-s" ]; then
echo Simulating
SIMULATE="-s"
shift
elif [ ${1} = "-r" ]; then
shift
LIMITrate="-r $1"
shift
echo Rate Limiting $LIMITrate
elif [ ${1} = "-p" ]; then
shift
Pmin=$1
shift
Pmax=$1
PAUSEbetween="--sleep-interval $Pmin --max-sleep-interval $Pmax"
shift
echo Time Span $TIMEspan to current
elif [ ${1} = "-e" ]; then
shift
TIMEspan="--dateafter $1"
shift
echo Time Span $TIMEspan to current
elif [ ${1} = "-BOB" ]; then
echo Grant Style Folder Structure
BOBstyle="True"
shift
else
vidID=$1
shift
echo
echo Working on video: $vidID
echo ========================================

URL=https://www.youtube.com/watch\?v=$vidID

if wget --spider --quiet $URL > /dev/null 2>&1; then
# Weirdly, YouTube says the URL is valid even if it doesn't exist when asking about videos...
# Then it gives an error while attempting the download.
# Not fatal, just annoying...

DEST=/mnt/Download_Space/YTDL/_Individual_/

if [ -d $DEST ]; then
echo $DEST exists
else
echo no $DEST... Building it...
mkdir $DEST
fi

ERRfile=$DEST/00000000-ERRORS
ARCHfile=$DEST/11111111-ARCHIVE

OPTIONS="$SIMULATE $LIMITrate $TIMEspan $WITHcookies $PAUSEbetween"
OPTIONS+=" --no-overwrites --write-description -t mp4"

cd $DEST
echo -e $storing in $DEST$
echo

/usr/local/bin/yt-dlp $OPTIONS --download-archive $ARCHfile -o "%(uploader_id)s - %(upload_date)s - %(title)s.%(ext)s" $URL/$ACK 2> >(/usr/bin/tee -a $ERRfile)

else
echo YouTube says this video does not exist.
fi
fi
done

# Clear out .part files
rm -f *.part
</syntaxhighlight>

= Gettin' Fancy... One script with multiple personalities =
Yup...

Looks even more complicated. In fact, it might even be considered kinda silly.

(assumes your base for downloading YouTube videos is at '''/mnt/Download_Space/YTDL/''' and contains subfolders named '''_Individual_''' and '''_Playlists_''')

Save this script as '''YTall'''

Then make symbolic links to it.
* <code>ln -s YTall YTc</code>
* <code>ln -s YTall YTv</code>
* <code>ln -s YTall YTl</code>
Call it as '''YTc''' for doing whole channels, '''YTv''' for individual videos, & '''YTl''' for playlists.

<syntaxhighlight lang="Bash" line copy>
#!/bin/bash

BASENAME=$(basename "$0")

echo -e "~~~~~~~ $BASENAME $@ ~~~~~~~ "\\n

case $BASENAME in
"YTv")
TYPE="VideoHash"
DESC="$TYPE = The name YouTube has given to the individual video"
DESC+="\n\\t\\t (This is gonna take some work to explain...)"
DESC+="\n\\t\\t You can specify multiple videos"
;;
"YTc")
TYPE="ChannelName"
DESC="$TYPE = The name of the channel $as defined by YouTube$"
DESC+="\n\\t\\t You can specify multiple channels"
;;
"YTl")
TYPE="ListHash"
DESC="$TYPE = The name YouTube has given to the playlist"
DESC+="\n\\t\\t (This is gonna take some work to explain...)"
DESC+="\n\\t\\t You can specify multiple lists"
;;
*)
TYPE="poop"
DESC="$TYPE = Some crap"
;;
esac

if [ $# -eq 0 ] || [ $1 = "-u" ] || [ $1 = "-?" ]; then
echo -e USAGE: $(basename "$0") \[...option$s$...\] $TYPE $TYPE ...
echo
echo -e OPTIONS:
echo -e \\t -BOB = Grant Style Folder Structure
echo
echo -e \\t -s = simulate this run
echo -e \\t -c = use a cookie file exported from your browser to identify yourself.
echo -e \\t -r RATE = Limit bandwidth
echo -e \\t\\t Maximum download rate in bytes per second,
echo -e \\t\\t\\t e.g. 50K or 4.2M
echo -e \\t -p MIN MAX = Sleep interval $pause$ between downloads $in seconds$
echo -e \\t\\t $will result in a random pause between MIN \& MAX seconds long$
echo -e \\t -e DATE = Earliest video upload date to grab.
echo -e \\t\\t see https://github.com/yt-dlp/yt-dlp?tab=readme-ov-file#video-selection for format info
echo -e \\t\\t $Simplest is YYYYMMDD but fancier options exist$
echo -e \\t $DESC
echo
echo -e The following \"ERROR\"s will also appear in 00000000-ERRORS
echo -e $Unfortunately, without indications of which files caused them...$
echo
echo -e \"ERROR: unable to download video data: HTTP Error 403: Forbidden\"
echo -e \\t Means you may need to supply cookies.
echo -e \\t Often caused by things like age restrictions.
echo -e \\t May also be caused by YouTube noticing you\'ve been downloading a bunch...
case $BASENAME in
"YTc")
echo
echo -e \"ERROR: \[youtube\] oizvS01ovH0: Video unavailable\"
echo -e \\t Most likely caused by a typo in the VideoHash
;;
"YTv")
echo
echo -e \"ERROR: \[youtube:tab\] \@ChannelName: This channel does not have a videos tab\"
echo -e \"ERROR: \[youtube:tab\] \@ChannelName: This channel does not have a shorts tab\"
echo -e \"ERROR: \[youtube:tab\] \@ChannelName: This channel does not have a streams tab\"
echo -e \\t Is simply indicating that the channel doesn\'t have that type of content.
echo -e \\t $All 3 means the channel doesn\'t actually have any content... :P $
;;
"YTl")
echo
echo -e \"mv: cannot stat \'*\': No such file or directory\"
echo -e \\t Don\'t panic. This one\'s just a bit of residue from the playlist folder cleanup
;;
esac
echo

exit
fi

BASEdest=/mnt/Download_Space/YTDL/
COOKIEfile=/mnt/Download_Space/YTDL/www.youtube.com_cookies.txt

SIMULATE=""
WITHcookies=""
LIMITrate=""
PAUSEbetween=""
TIMEspan=""
BOBstyle=""

while (( "$#" ));
do
case $1 in
"-c")
echo Using Cookies: $COOKIEfile
WITHcookies="--cookies "$COOKIEfile
shift
;;

"-s")
echo Simulating
SIMULATE="-s"
shift
;;

"-r")
LIMITrate="-r $2"
echo Rate Limiting $2
shift 2
;;

"-p")
PAUSEbetween="--sleep-interval $2 --max-sleep-interval $3"
echo Random pause between $2 \& $3 seconds
shift 3
;;

"-e")
TIMEspan="--dateafter $2"
echo Time Span $2 to current
shift 2
;;

"-BOB")
BOBstyle="True"
echo Using Grant Style Folder Structure
shift
;;

*)
case $BASENAME in
"YTv") # Video
echo Working on video: $1
echo ========================================
WeirdID=$1
URL=https://www.youtube.com/watch\?v=$1
TYPES=""
DEST="$BASEdest""_Individual_/"
FORMAT="%(uploader_id)s - %(upload_date)s - %(title)s.%(ext)s"
;;

"YTc") # Channel
echo Working on Channel: $1
echo ========================================
WeirdID=$1
URL=https://www.youtube.com/\@$1
TYPES="videos shorts streams"
# TYPES="videos shorts streams playlists podcasts courses"
DEST="$BASEdest""$1"
FORMAT="%(upload_date)s - %(title)s.%(ext)s"
;;

"YTl") # Playlist
echo Working on playlist: $1
echo ========================================
WeirdID=$1
URL=https://www.youtube.com/playlist\?list=$1
TYPES=""
DEST="$BASEdest""_Playlists_/"
FORMAT="%(playlist_index)s_%(playlist_count)s %(uploader_id)s - %(title)s.%(ext)s"
;;
esac

ERRfile=$DEST/00000000-ERRORS
ARCHfile=$DEST/11111111-ARCHIVE

OPTIONS="$SIMULATE $LIMITrate $TIMEspan $WITHcookies $PAUSEbetween"
OPTIONS+=" --no-overwrites --write-description -t mp4"

echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
echo $URL
echo $WeirdID
echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"

if wget --spider --quiet $URL > /dev/null 2>&1; then
# Weirdly, for individual videos, YouTube says the URL is valid even if it doesn't exist when asking about videos...
# Then it gives an error while attempting the download.
# Not fatal, just annoying...

echo -en "(storing in $DEST"

case $BASENAME in
"YTv")
echo ")"
echo VID $WeirdID >> $ARCHfile
cd $DEST
pwd
/usr/local/bin/yt-dlp $OPTIONS --download-archive $ARCHfile -o "$FORMAT" $URL 2> >(/usr/bin/tee -a $ERRfile)
rm -f *.part
;;
"YTl")
echo "$WeirdID)"
echo LIST $WeirdID >> $ARCHfile
cd $DEST
pwd
if [ -d $DEST/$WeirdID ]; then
echo $DEST exists
else
echo no $DEST$WeirdID... Building it...
mkdir $DEST$WeirdID
fi
cd $WeirdID
pwd
/usr/local/bin/yt-dlp $OPTIONS --download-archive $ARCHfile -o "$FORMAT" $URL 2> >(/usr/bin/tee -a $ERRfile)
rm -f *.part

NAME=`find . -maxdepth 1 -regextype posix-extended -regex './0+_.*' -type f `

Listname=`echo ${NAME::-12} | cut -d'@' -f 2`

if [ -d ../"$Listname" ]; then
echo "$Listname" already exists
else
mkdir ../"$Listname"
fi

mv "$NAME" ../"$Listname/000 $Listname.description"
mv * ../"$Listname"
cd ..
rmdir $WeirdID
;;
"YTc")
echo ")"
if [ -d $DEST ]; then
echo $DEST exists
else
echo no $DEST... Building it...
mkdir $DEST
if [ -z $BOBstyle ]; then
for ACK in $TYPES
do
mkdir $DEST/$ACK
done
fi
fi

cd $DEST
pwd

if [ -z $BOBstyle ]; then
echo -e "(using subfolders...)"
else
echo -e "(using Grant format...)"
fi
echo

for ACK in $TYPES
do
echo ">>>>>>>> " $URL/$ACK
if [ -z $BOBstyle ]; then
cd $ACK
echo CHAN $WeirdID $ACK >> $ARCHfile
fi

/usr/local/bin/yt-dlp $OPTIONS --download-archive $ARCHfile -o "$FORMAT" $URL/$ACK 2> >(/usr/bin/tee -a $ERRfile)

# Clear out .part files
rm -f *.part

if [ -z $BOBstyle ]; then
cd ..
fi
done
;;
esac
else
echo YouTube says this channel does not exist.
fi
shift
;;

esac
done
</syntaxhighlight>

= About the weirdness that is YouTube naming conventions =
When looking at a channel page:

<nowiki>https://www.youtube.com/@</nowiki>'''Tinker001'''

The '''bold purple''' part is the channel name
----
When watching a video:

<nowiki>https://www.youtube.com/watch?v=</nowiki>'''y8o7qkmiDso'''

The '''bold purple''' part is the video name
----
When watching a playlist:

<nowiki>https://www.youtube.com/watch?v=</nowiki>'''oizvS01ovHE'''&list='''PLLG8IgBGD4eUG2n6p3GGfkQ-KPceDuuWj'''&pp=gAQB

The '''bold purple''' part is the playlist name

Bonus fun bit: The '''bold green''' part is actually the name of the video being watched inside that playlist.

= Even more options? =
'''WIP'''

YT-DLP Scripting

2025-10-06T01:31:21Z

Tinker: /* Gettin' Fancy... One script with multiple personalities */

= YTchan =

A script for downloading entire channels...

(assumes your base for downloading YouTube videos is at '''/mnt/Download_Space/YTDL/''' and creates a subfolder using the channel name(s) )

For the potentially required cookie file, see [https://github.com/yt-dlp/yt-dlp/wiki/Extractors#exporting-youtube-cookies Exporting YouTube cookies]

<syntaxhighlight lang="Bash" line copy>
#!/bin/bash

echo -e "~~~~~~~ " $0 $@ "~~~~~~~ "\\n

if [ $# -eq 0 ] || [ $1 = "-u" ] || [ $1 = "-?" ]; then
echo -e USAGE: YTchan \[...option$s$...\] ChannelName ChannelName ...
echo
echo -e OPTIONS:
echo -e \\t -BOB = Grant Style Folder Structure
echo
echo -e \\t -s = simulate this run
echo -e \\t -c = use a cookie file exported from your browser to identify yourself.
echo -e \\t -r RATE = Limit bandwidth
echo -e \\t\\t Maximum download rate in bytes per second,
echo -e \\t\\t\\t e.g. 50K or 4.2M
echo -e \\t -p MIN MAX = Sleep interval $pause$ between downloads $in seconds$
echo -e \\t\\t $will result in a random pause between MIN \& MAX seconds long$
echo -e \\t -e DATE = Earliest video upload date to grab.
echo -e \\t\\t see https://github.com/yt-dlp/yt-dlp?tab=readme-ov-file#video-selection for format info
echo -e \\t\\t $Simplest is YYYYMMDD but fancier options exist$
echo -e \\t ChannelName = The name of the channel $as defined by YouTube$
echo -e \\t\\t You can specify multiple channels
echo
echo -e The following \"ERROR\"s will also appear in 00000000-ERRORS
echo -e $Unfortunately, without indications of which files caused them...$
echo
echo -e \"ERROR: \[youtube:tab\] \@ChannelName: This channel does not have a videos tab\"
echo -e \"ERROR: \[youtube:tab\] \@ChannelName: This channel does not have a shorts tab\"
echo -e \"ERROR: \[youtube:tab\] \@ChannelName: This channel does not have a streams tab\"
echo -e \\t Is simply indicating that the channel doesn\'t have that type of content.
echo -e \\t $All 3 means the channel doesn\'t actually have any content... :P $
echo
echo -e \"ERROR: unable to download video data: HTTP Error 403: Forbidden\"
echo -e \\t Means you may need to supply cookies.
echo -e \\t Often caused by things like age restrictions.
echo -e \\t May also be caused by YouTube noticing you\'ve been downloading a bunch...
echo

exit
fi

#TYPES="videos"
TYPES="videos shorts streams"
COOKIEfile=/mnt/Download_Space/YTDL/www.youtube.com_cookies.txt

SIMULATE=""
WITHcookies=""
LIMITrate=""
PAUSEbetween=""
TIMEspan=""
BOBstyle=""

while (( "$#" ));
do
if [ ${1} = "-c" ]; then
echo Using Cookies: $COOKIEfile
WITHcookies="--cookies "$COOKIEfile
shift
elif [ ${1} = "-s" ]; then
echo Simulating
SIMULATE="-s"
shift
elif [ ${1} = "-r" ]; then
shift
LIMITrate="-r $1"
shift
echo Rate Limiting $LIMITrate
elif [ ${1} = "-p" ]; then
shift
Pmin=$1
shift
Pmax=$1
PAUSEbetween="--sleep-interval $Pmin --max-sleep-interval $Pmax"
shift
echo Time Span $TIMEspan to current
elif [ ${1} = "-e" ]; then
shift
TIMEspan="--dateafter $1"
shift
echo Time Span $TIMEspan to current
elif [ ${1} = "-BOB" ]; then
echo Grant Style Folder Structure
BOBstyle="True"
shift
else
CHANNEL=$1
shift
echo
echo Working on channel: $CHANNEL
echo ========================================

URL=https://www.youtube.com/\@$CHANNEL

if wget --spider --quiet $URL > /dev/null 2>&1; then

DEST=/mnt/Download_Space/YTDL/$CHANNEL

ERRfile=$DEST/00000000-ERRORS
ARCHfile=$DEST/11111111-ARCHIVE

if [ -d $DEST ]; then
echo $DEST exists
else
echo no $DEST... Building it...
mkdir $DEST
if [ -z $BOBstyle ]; then
for ACK in $TYPES
do
mkdir $DEST/$ACK
done
fi
fi

OPTIONS="$SIMULATE $LIMITrate $TIMEspan $WITHcookies $PAUSEbetween"
OPTIONS+=" --no-overwrites --write-description -t mp4"

cd $DEST
echo -e $storing in $DEST$
echo

for ACK in $TYPES
do
echo ">>>>>>>> " $URL/$ACK
if [ -z $BOBstyle ]; then
cd $ACK
fi
/usr/local/bin/yt-dlp $OPTIONS --download-archive $ARCHfile -o "%(upload_date)s - %(title)s.%(ext)s" $URL/$ACK 2> >(/usr/bin/tee -a $ERRfile)

# Clear out .part files
rm -f *.part

if [ -z $BOBstyle ]; then
cd ..
fi
done

else
echo YouTube says this channel does not exist.
fi
fi
done
</syntaxhighlight>

= YTvid =

A script for downloading separate videos...

(assumes your base for downloading YouTube videos is at '''/mnt/Download_Space/YTDL/''' and creates a subfolder named '''_Individual_''')

For the potentially required cookie file, see [https://github.com/yt-dlp/yt-dlp/wiki/Extractors#exporting-youtube-cookies Exporting YouTube cookies]

<syntaxhighlight lang="Bash" line copy>
#!/bin/bash

echo -e "~~~~~~~ " $0 $@ "~~~~~~~ "\\n

if [ $# -eq 0 ] || [ $1 = "-u" ] || [ $1 = "-?" ]; then
echo -e USAGE: YTvid \[...option$s$...\] VideoHash VideoHash ...
echo
echo -e OPTIONS:
echo -e \\t -BOB = Grant Style Folder Structure
echo
echo -e \\t -s = simulate this run
echo -e \\t -c = use a cookie file exported from your browser to identify yourself.
echo -e \\t -r RATE = Limit bandwidth
echo -e \\t\\t Maximum download rate in bytes per second,
echo -e \\t\\t\\t e.g. 50K or 4.2M
echo -e \\t -p MIN MAX = Sleep interval $pause$ between downloads $in seconds$
echo -e \\t\\t $will result in a random pause between MIN \& MAX seconds long$
echo -e \\t -e DATE = Earliest video upload date to grab.
echo -e \\t\\t see https://github.com/yt-dlp/yt-dlp?tab=readme-ov-file#video-selection for format info
echo -e \\t\\t $Simplest is YYYYMMDD but fancier options exist$
echo -e \\t VideoHash = The name YouTube has given to the individual video
echo -e \\t\\t $This is gonna take some work to explain...$
echo -e \\t\\t You can specify multiple videos
echo
echo -e The following \"ERROR\"s will also appear in 00000000-ERRORS
echo -e $Unfortunately, without indications of which files caused them...$
echo
echo -e \"ERROR: unable to download video data: HTTP Error 403: Forbidden\"
echo -e \\t Means you may need to supply cookies.
echo -e \\t Often caused by things like age restrictions.
echo -e \\t May also be caused by YouTube noticing you\'ve been downloading a bunch...
echo
echo -e \"ERROR: \[youtube\] oizvS01ovH0: Video unavailable\"
echo -e \\t Most likely caused by a typo in the VideoHash
echo

exit
fi

COOKIEfile=/mnt/Download_Space/YTDL/www.youtube.com_cookies.txt

SIMULATE=""
WITHcookies=""
LIMITrate=""
PAUSEbetween=""
TIMEspan=""
BOBstyle=""

while (( "$#" ));
do
if [ ${1} = "-c" ]; then
echo Using Cookies: $COOKIEfile
WITHcookies="--cookies "$COOKIEfile
shift
elif [ ${1} = "-s" ]; then
echo Simulating
SIMULATE="-s"
shift
elif [ ${1} = "-r" ]; then
shift
LIMITrate="-r $1"
shift
echo Rate Limiting $LIMITrate
elif [ ${1} = "-p" ]; then
shift
Pmin=$1
shift
Pmax=$1
PAUSEbetween="--sleep-interval $Pmin --max-sleep-interval $Pmax"
shift
echo Time Span $TIMEspan to current
elif [ ${1} = "-e" ]; then
shift
TIMEspan="--dateafter $1"
shift
echo Time Span $TIMEspan to current
elif [ ${1} = "-BOB" ]; then
echo Grant Style Folder Structure
BOBstyle="True"
shift
else
vidID=$1
shift
echo
echo Working on video: $vidID
echo ========================================

URL=https://www.youtube.com/watch\?v=$vidID

if wget --spider --quiet $URL > /dev/null 2>&1; then
# Weirdly, YouTube says the URL is valid even if it doesn't exist when asking about videos...
# Then it gives an error while attempting the download.
# Not fatal, just annoying...

DEST=/mnt/Download_Space/YTDL/_Individual_/

if [ -d $DEST ]; then
echo $DEST exists
else
echo no $DEST... Building it...
mkdir $DEST
fi

ERRfile=$DEST/00000000-ERRORS
ARCHfile=$DEST/11111111-ARCHIVE

OPTIONS="$SIMULATE $LIMITrate $TIMEspan $WITHcookies $PAUSEbetween"
OPTIONS+=" --no-overwrites --write-description -t mp4"

cd $DEST
echo -e $storing in $DEST$
echo

/usr/local/bin/yt-dlp $OPTIONS --download-archive $ARCHfile -o "%(uploader_id)s - %(upload_date)s - %(title)s.%(ext)s" $URL/$ACK 2> >(/usr/bin/tee -a $ERRfile)

else
echo YouTube says this video does not exist.
fi
fi
done

# Clear out .part files
rm -f *.part
</syntaxhighlight>

= Gettin' Fancy... One script with multiple personalities =
Yup...

Looks even more complicated. In fact, it might even be considered kinda silly.

(assumes your base for downloading YouTube videos is at '''/mnt/Download_Space/YTDL/''' and contains subfolders named '''_Individual_''' and '''_Playlists_''')

Save this script as '''YTall'''

Then make symbolic links to it.
* <code>ln -s YTall YTc</code>
* <code>ln -s YTall YTv</code>
* <code>ln -s YTall YTl</code>
Call it as '''YTc''' for doing whole channels, '''YTv''' for individual videos, & '''YTl''' for playlists.

<syntaxhighlight lang="Bash" line copy>
#!/bin/bash

BASENAME=$(basename "$0")

echo -e "~~~~~~~ $BASENAME $@ ~~~~~~~ "\\n

case $BASENAME in
"YTv")
TYPE="VideoHash"
DESC="$TYPE = The name YouTube has given to the individual video"
DESC+="\n\\t\\t (This is gonna take some work to explain...)"
DESC+="\n\\t\\t You can specify multiple videos"
;;
"YTc")
TYPE="ChannelName"
DESC="$TYPE = The name of the channel $as defined by YouTube$"
DESC+="\n\\t\\t You can specify multiple channels"
;;
"YTl")
TYPE="ListHash"
DESC="$TYPE = The name YouTube has given to the playlist"
DESC+="\n\\t\\t (This is gonna take some work to explain...)"
DESC+="\n\\t\\t You can specify multiple lists"
;;
*)
TYPE="poop"
DESC="$TYPE = Some crap"
;;
esac

if [ $# -eq 0 ] || [ $1 = "-u" ] || [ $1 = "-?" ]; then
echo -e USAGE: $(basename "$0") \[...option$s$...\] $TYPE $TYPE ...
echo
echo -e OPTIONS:
echo -e \\t -BOB = Grant Style Folder Structure
echo
echo -e \\t -s = simulate this run
echo -e \\t -c = use a cookie file exported from your browser to identify yourself.
echo -e \\t -r RATE = Limit bandwidth
echo -e \\t\\t Maximum download rate in bytes per second,
echo -e \\t\\t\\t e.g. 50K or 4.2M
echo -e \\t -p MIN MAX = Sleep interval $pause$ between downloads $in seconds$
echo -e \\t\\t $will result in a random pause between MIN \& MAX seconds long$
echo -e \\t -e DATE = Earliest video upload date to grab.
echo -e \\t\\t see https://github.com/yt-dlp/yt-dlp?tab=readme-ov-file#video-selection for format info
echo -e \\t\\t $Simplest is YYYYMMDD but fancier options exist$
echo -e \\t $DESC
echo
echo -e The following \"ERROR\"s will also appear in 00000000-ERRORS
echo -e $Unfortunately, without indications of which files caused them...$
echo
echo -e \"ERROR: unable to download video data: HTTP Error 403: Forbidden\"
echo -e \\t Means you may need to supply cookies.
echo -e \\t Often caused by things like age restrictions.
echo -e \\t May also be caused by YouTube noticing you\'ve been downloading a bunch...
case $BASENAME in
"YTc")
echo
echo -e \"ERROR: \[youtube\] oizvS01ovH0: Video unavailable\"
echo -e \\t Most likely caused by a typo in the VideoHash
;;
"YTv")
echo
echo -e \"ERROR: \[youtube:tab\] \@ChannelName: This channel does not have a videos tab\"
echo -e \"ERROR: \[youtube:tab\] \@ChannelName: This channel does not have a shorts tab\"
echo -e \"ERROR: \[youtube:tab\] \@ChannelName: This channel does not have a streams tab\"
echo -e \\t Is simply indicating that the channel doesn\'t have that type of content.
echo -e \\t $All 3 means the channel doesn\'t actually have any content... :P $
;;
"YTl")
echo
echo -e \"mv: cannot stat \'*\': No such file or directory\"
echo -e \\t Don\'t panic. This one\'s just a bit of residue from the playlist folder cleanup
;;
esac
echo

exit
fi

BASEdest=/mnt/Download_Space/YTDL/
COOKIEfile=/mnt/Download_Space/YTDL/www.youtube.com_cookies.txt

SIMULATE=""
WITHcookies=""
LIMITrate=""
PAUSEbetween=""
TIMEspan=""
BOBstyle=""

while (( "$#" ));
do
case $1 in
"-c")
echo Using Cookies: $COOKIEfile
WITHcookies="--cookies "$COOKIEfile
shift
;;

"-s")
echo Simulating
SIMULATE="-s"
shift
;;

"-r")
LIMITrate="-r $2"
echo Rate Limiting $2
shift 2
;;

"-p")
PAUSEbetween="--sleep-interval $2 --max-sleep-interval $3"
echo Random pause between $2 \& $3 seconds
shift 3
;;

"-e")
TIMEspan="--dateafter $2"
echo Time Span $2 to current
shift 2
;;

"-BOB")
BOBstyle="True"
echo Using Grant Style Folder Structure
shift
;;

*)
case $BASENAME in
"YTv") # Video
echo Working on video: $1
echo ========================================
WeirdID=$1
URL=https://www.youtube.com/watch\?v=$1
TYPES=""
DEST="$BASEdest""_Individual_/"
FORMAT="%(uploader_id)s - %(upload_date)s - %(title)s.%(ext)s"
;;

"YTc") # Channel
echo Working on Channel: $1
echo ========================================
WeirdID=$1
URL=https://www.youtube.com/\@$1
TYPES="videos shorts streams"
# TYPES="videos shorts streams playlists podcasts courses"
DEST="$BASEdest""$1"
FORMAT="%(upload_date)s - %(title)s.%(ext)s"
;;

"YTl") # Playlist
echo Working on playlist: $1
echo ========================================
WeirdID=$1
URL=https://www.youtube.com/playlist\?list=$1
TYPES=""
DEST="$BASEdest""_Playlists_/"
FORMAT="%(playlist_index)s_%(playlist_count)s %(uploader_id)s - %(title)s.%(ext)s"
;;
esac

ERRfile=$DEST/00000000-ERRORS
ARCHfile=$DEST/11111111-ARCHIVE

OPTIONS="$SIMULATE $LIMITrate $TIMEspan $WITHcookies $PAUSEbetween"
OPTIONS+=" --no-overwrites --write-description -t mp4"

echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
echo $URL
echo $WeirdID
echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"

if wget --spider --quiet $URL > /dev/null 2>&1; then
# Weirdly, for individual videos, YouTube says the URL is valid even if it doesn't exist when asking about videos...
# Then it gives an error while attempting the download.
# Not fatal, just annoying...

echo -en "(storing in $DEST"

case $BASENAME in
"YTv")
echo ")"
echo VID $WeirdID >> $ARCHfile
cd $DEST
pwd
/usr/local/bin/yt-dlp $OPTIONS --download-archive $ARCHfile -o "$FORMAT" $URL 2> >(/usr/bin/tee -a $ERRfile)
rm -f *.part
;;
"YTl")
echo "$WeirdID)"
echo LIST $WeirdID >> $ARCHfile
cd $DEST
pwd
if [ -d $DEST/$WeirdID ]; then
echo $DEST exists
else
echo no $DEST$WeirdID... Building it...
mkdir $DEST$WeirdID
fi
cd $WeirdID
pwd
/usr/local/bin/yt-dlp $OPTIONS --download-archive $ARCHfile -o "$FORMAT" $URL 2> >(/usr/bin/tee -a $ERRfile)
rm -f *.part

NAME=`find . -maxdepth 1 -regextype posix-extended -regex './0+_.*' -type f `

Listname=`echo ${NAME::-12} | cut -d'@' -f 2`

if [ -d ../"$Listname" ]; then
echo "$Listname" already exists
else
mkdir ../"$Listname"
fi

mv "$NAME" ../"$Listname/000 $Listname.description"
mv * ../"$Listname"
cd ..
rmdir $WeirdID
;;
"YTc")
echo ")"
if [ -d $DEST ]; then
echo $DEST exists
else
echo no $DEST... Building it...
mkdir $DEST
if [ -z $BOBstyle ]; then
for ACK in $TYPES
do
mkdir $DEST/$ACK
done
fi
fi

cd $DEST
pwd

if [ -z $BOBstyle ]; then
echo -e "(using subfolders...)"
else
echo -e "(using Grant format...)"
fi
echo

for ACK in $TYPES
do
echo ">>>>>>>> " $URL/$ACK
if [ -z $BOBstyle ]; then
cd $ACK
echo CHAN $WeirdID $ACK >> $ARCHfile
fi

/usr/local/bin/yt-dlp $OPTIONS --download-archive $ARCHfile -o "$FORMAT" $URL/$ACK 2> >(/usr/bin/tee -a $ERRfile)

# Clear out .part files
rm -f *.part

if [ -z $BOBstyle ]; then
cd ..
fi
done
;;
esac
else
echo YouTube says this channel does not exist.
fi
shift
;;

esac
done
</syntaxhighlight>

= About the weirdness that is YouTube naming conventions =
When looking at a channel page:

<nowiki>https://www.youtube.com/@</nowiki>'''Tinker001'''

The '''bold purple''' part is the channel name
----
When watching a video:

<nowiki>https://www.youtube.com/watch?v=</nowiki>'''y8o7qkmiDso'''

The '''bold purple''' part is the video name
----
When watching a playlist:

<nowiki>https://www.youtube.com/watch?v=</nowiki>'''oizvS01ovHE'''&list='''PLLG8IgBGD4eUG2n6p3GGfkQ-KPceDuuWj'''&pp=gAQB

The '''bold purple''' part is the playlist name

Bonus fun bit: The '''bold green''' part is actually the name of the video being watched inside that playlist.

= Even more options? =
'''WIP'''