Merge branch 'main' into linux_internals

src/articles/interesting_sites.md

@@ -80,6 +80,8 @@ This is an unstructured list of web-sites I find interesting. I keep them here r
 
 * [sockets](https://www.cs.rpi.edu/~moorthy/Courses/os98/Pgms/socket.html)
 * [http made easy](https://www.jmarshall.com/easy/http/)
+* [Wireguard for private network](https://taggart-tech.com/wireguard/)
+* [Sysadmin tools](https://mwl.io/nonfiction/tools)
 
 # Database #
 
@@ -113,9 +115,16 @@ This is an unstructured list of web-sites I find interesting. I keep them here r
 
 * [Fuzzing GStreamer](https://github.blog/security/vulnerability-research/uncovering-gstreamer-secrets/)
 
+# Maths #
+
+* [The Matrix Cookbook](https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf)
+* [The Tensor Cookbook](https://tensorcookbook.com/)
+
 # Machine Learning #
 
 * [MLE](https://rish-01.github.io/blog/posts/ml_estimation/)
+* [Gradient Descent Overview](https://www.ruder.io/optimizing-gradient-descent/)
+* [Generative Models Overview](https://nrehiew.github.io/blog/2024/)
 
 # Linked Data #
 
@@ -125,6 +134,21 @@ This is an unstructured list of web-sites I find interesting. I keep them here r
 
 * [Flappy Bird C Clone](https://github.com/alxyng/flappybird)
 
+# Operating Systems #
+
+* [Unix History](http://crn.hopto.org/unix/)
+
+
+# Live and Minimalist Computing #
+
+* [Emacs like editor on steroids](https://albertzak.com/run-build-grow/)
+* [Direnv and .envrc files](https://offby1.website/posts/uv-direnv-and-simple-envrc-files.html)
+
+
+# Hardware #
+
+* [Making an analogue synth](https://www.peterzimon.com/hog/)
+
 # Gardening #
 
 * [Planning a vegetable garden for beginners](https://www.youtube.com/watch?v=9R-utqpmwmE)

src/articles/linux_tools.md

@@ -0,0 +1,58 @@
+# Logging and Monitoring #
+
+* rsyslog
+* [syslog](https://en.wikipedia.org/wiki/Syslog): RFC5424
+
+`/var/log/syslog`
+`/var/log`
+
+* eBPF
+
+* Zabbix
+* Influx
+* Grafana
+* Prometheus (+ Grafana + Loki as stack)
+* timescaleDB
+* AlertManager
+* Loki
+* Graphite
+* Spiceworks
+* Crowdsec
+* Netdata
+* NodeExtractor/NodeExporter
+* ELK - Elasticsearch, Kibana, Logstash
+
+
+https://grafana.com/blog/2016/01/05/logs-and-metrics-and-graphs-oh-my/
+
+
+Setting up Grafana: https://grafana.com/docs/grafana/latest/setup-grafana/installation/docker/
+
+Setting up Prometheus: https://github.com/prometheus/prometheus
+
+Some things to measure:
+
+- apt status (for security/critical updates that haven't been run yet)
+- reboot needed (presence of /var/run/reboot-required)
+- fail2ban jail status (how many are in each of our defined jails)
+- CPU usage
+- MySQL active, long-running processes, number of queries
+- iostat numbers
+- disk space
+- SSL cert expiration date
+- domain expiration date
+- reachability (ping, domain resolution, specific string in an HTTP request)
+- Application-specific checks (WordPress, Drupal, CRM, etc)
+- postfix queue size
+* apt/yum/fwupd/... pending updates
+* mailqueue length, root's mailbox size: this is an indicator for stuff going wrong silently
+* pending reboot after kernel update
+* certain kinds of log entries (block device read error, OOMkills, core dumps).
+* network checksum errors, dropped packets, martians
+* presence or non-presence of USB devices: desktops should have keyboard and mouse. servers usually shouldn't. usb storage is sometimes forbidden.
+
+## Further Reading ##
+
+* https://www.redhat.com/en/blog/log-aggregation-rsyslog
+
+#  Auto updates

src/articles/self_hosting.md

@@ -4,42 +4,46 @@ This article is intended to be an ever-evolving guide to setting up a self-hosti
 
 ## Location ##
 
-I want to host with my own hardware, mostly so I have control over costs but general control and independence reasons are appealing also.
+I want to host with my own hardware where feasible, mostly so I have control over costs but general control and independence reasons are appealing also.
 
-I don't have a static IP and don't really want the fuss of getting and maintaining one, nor making my home IP publically visible. For this reason I've opted to run a small VPS on an external hosting provider to act as a public gateway to locally hosted content via a reverse proxy. I also want to run a VPN on this VPS, so I can access services only visible to my home network while away from it. For the external provider I went with Hetzner for reasons of price, reputation and that they are based in Europe. I wasn't too happy with the extent of personal data they collect during registration but understand there are positives to it also, such as IP ranges not being (as) overrun with crap.
+I don't have a static IP and don't really want the fuss of getting one, nor making my home IP publically visible. For this reason I've opted to run a small VPS on an external hosting provider to act as a public gateway to locally hosted content. I also want to run a VPN on this VPS, so I can access services only visible to my home network while away from it. For the external provider I went with [Hetzner](https://www.hetzner.com/) for reasons of price, reputation and that they are based in Europe. I wasn't too happy with the extent of personal data they collect during registration but understand why they do it.
 
 ## Hardware ##
 
-I had an old Raspberry Pi laying around but felt it wouldn't have enough juice to handle the bunch of services I had planned for it so grabbed a Raspberry Pi 5 with 8 GB RAM. This seems to have been a wise choice when importing my media libraries at least. 
+I have a hodge-podge of old hardware laying around and where possible am trying to use that over buying anything new. That said, my Raspberry Pi was ancient and I don't think it was up to the task of running my planned services, so I grabbed a Raspberry Pi 5 with 8 GB RAM. I had a couple of old SSDs - one 256 GB and one 2 TB, so hooked them up via PCIe to USB connectors. The Pi is running on WiFi at the moment, although I could attach it to Ethernet over the house wiring if there is a performance issue.
 
-I had a couple of old SSDs too - one 256 GB and one 2 TB, so hooked them up via PCIe to USB connectors. 
+## Public Host ##
 
-The Pi is running on WiFi at the moment, although I could attach it to Ethernet over the house wiring if there is a performance issue.
+I'm running a Debian image on a small Hetzner VPS. This hosts public facing services, such as my personal website and Forgejo code repository.
 
-## Operating System ##
 
-For now, I've gone with the default Debian image for the Raspberry Pi - although it has plenty of bloat that I don't need. Hopefully in future I can get Guix running on there, which I'm currently using on my laptop.
+## Home Network ##
 
-## Core Services ##
+My home network just consists of a Raspberry Pi based server for now - with access from various laptops, mobile clients and a Smart Tv. I don't have any short term plans to add to that. It isn't exposed to the public internet but is planned to be accessible by VPN.
 
-I am planning to set up:
+### Operating System ###
 
-### DNS: Pi-Hole ###
+For now, I've gone with the default Debian image for the Raspberry Pi. I wasn't too careful setting it up so have ended up with a full blown desktop installation. Hopefully in future I can eventually get [Guix](https://guix.gnu.org/) running on there, which I'm currently happily using on my laptop.
 
-https://docs.pi-hole.net/ Which I want to use for internal DHCP/DNS management, helpful for the likes of NextCloud - however the ad blocking functionality is certainly appealing too.
+### Core Services ###
 
-Followed option 'Alternative 1' here:
+These are the core services that support access, monitoring and security on my system.
 
-```
+#### DNS ####
+
+I'm using [pi-hole](https://docs.pi-hole.net/) for ad blocking and am planning to also use it for  internal DHCP/DNS management on my home network.
+
+For starters I set it up to act as a DHCP server. Before starting I needed to configure the pi to get a static IP, this is through `/etc/network/interfaces`. 
+
+Then I followed option 'Alternative 1' on the pi-hole getting started docs:
+
+```sh
 git clone --depth 1 https://github.com/pi-hole/pi-hole.git Pi-hole
 cd "Pi-hole/automated install/"
 sudo bash basic-install.sh
 ```
 
-Before starting off you will need to force the machine to get a static IP, this is through `/etc/network/interfaces`. 
+Finally I forced my router's DHCP server to only assign one IP, a fixed one to the pi and then enabled DHCP serving on the Pi.
-
-Once set up for my router I needed the pi-hole to act as both DNS and DHCP servers. Did this by forcing the router DHCP server to only assign IP, a fixed one to the pi and then enabling DHCP serving on the Pi.
-
 
 ### VPN: Wireguard ###