I recognize this will vary depending on how much you self-host, so I’m curious about the range of experiences from the few self-hosted things to the many self-hosted things.

Also how might you compare it to other maintenance of your other online systems (e.g. personal computer/phone/etc.)?

  • henfredemars@infosec.pub
    link
    fedilink
    English
    arrow-up
    57
    ·
    edit-2
    5 months ago

    Huge amounts of daily maintenance because I lack self control and keep changing things that were previously working.

  • Max-P@lemmy.max-p.me
    link
    fedilink
    English
    arrow-up
    30
    ·
    edit-2
    5 months ago

    Very minimal. Mostly just run updates every now and then and fix what breaks which is relatively rare. The Docker stacks in particular are quite painless.

    Couple websites, Lemmy, Matrix, a whole email stack, DNS, IRC bouncer, NextCloud, WireGuard, Jitsi, a Minecraft server and I believe that’s about it?

    I’m a DevOps engineer at work, managing 2k+ VMs that I can more than keep up with. I’d say it varies more with experience and how it’s set up than how much you manage. When you use Ansible and Terraform and Kubernetes, the count of servers and services isn’t really important. One, five, ten, a thousand servers, it matters very little since you just run Ansible on them and 5 minutes later it’s all up and running. I don’t use that for my own servers out of laziness but still, I set most of that stuff 10 years ago and it’s still happily humming along just fine.

    • Footnote2669@lemmy.zip
      link
      fedilink
      English
      arrow-up
      4
      ·
      5 months ago

      +1 for docker and minimal maintenance. Only updates or new containers might break stuff. If you don’t touch it, it will be fine. Of course there might be some container specific problems. Depends what you want to run. And I’m not a devops engineer like Max 😅

    • MBV ⚜️@lemm.ee
      link
      fedilink
      English
      arrow-up
      1
      ·
      5 months ago

      Same same - just one update a week on Friday btw 2 yawns of the 4VMs and 10-15 services i have + quarterly backup. Does not involve much + the odd ad-hoc re-linking the reverse proxy when containers switch ips on the docker network when the VM restarts/resets

  • 0110010001100010@lemmy.world
    link
    fedilink
    English
    arrow-up
    10
    ·
    5 months ago

    Typically, very little. I have ~40 containers in my Docker stack and by in large it just works. I upgrade stuff here and there as needed. I am getting ready to do a hardware refresh but again with Docker that’s pretty painless.

    Most of the time spent in my lab is trying out new things. I’ll find a new something that looks cool and go down the rabbit hole with it for a while. Then back to the status quo.

  • CarbonatedPastaSauce@lemmy.world
    link
    fedilink
    English
    arrow-up
    10
    ·
    5 months ago

    It’s bursty; I tend to do a lot of work on stuff when I do a hardware upgrade, but otherwise it’s set it and forget it for the most part. The only servers I pay any significant attention to in terms of frequent maintenance and security checks are the MTAs in the DMZ for my email. Nothing else is exposed to the internet for inbound traffic except a game server VM that’s segregated (credential-wise and network-wise) from everything else, so if it does get compromised it would be a very minimal danger to the rest of my network. Everything either has automated updates, or for servers I want more control over I manually update them when the mood strikes me or a big vulnerability that affects my software hits the news.

    TL;DR If you averaged it over a year, I maybe spend 30-60 minutes a week on self hosting maintenance tasks for 4 physical servers and about 20 VM’s.

  • dlundh@lemmy.world
    link
    fedilink
    English
    arrow-up
    7
    ·
    5 months ago

    A lot less since I started using docker instead of running separate vms for everything. Less systems to update is bliss.

  • fartsparkles@sh.itjust.works
    link
    fedilink
    English
    arrow-up
    7
    ·
    edit-2
    5 months ago

    I run everything in containers that rebuild every time / auto update but I do so because I have everything backed up and I can easily roll back if something doesn’t work. I mount storage separately so I can simply remount an older backup to the same place in the file system.

    Took a good day to set it all up but now I can add anything else I need with a simple compose script. The main host is also built from an image that auto updates every night.

    So basically I spend zero time anymore bar when XZ happened since I use a rolling release (yes, I’m a mad man but I enjoy the odd breakage to see what’s going on in the Linux world).

    The real trick is partitioning the OS up so that boot image and system are on an SSD and everything else on a massive RAID array with snapshots written to the array and all scripts/configs in git. It means I can rebuild the entire setup in a matter of minutes.

  • mikyopii@programming.dev
    link
    fedilink
    English
    arrow-up
    7
    ·
    5 months ago

    For some reason my DNS tends to break the most. I have to reinstall my Pi-hole semi-regularly.

    NixOS plus Docker is my preferred setup for hosting applications. Sometime it is a pain to get running but once it does it tends to run. If a container doesn’t work, restart it. If the OS doesn’t work, roll it back.

  • Opisek@lemmy.world
    link
    fedilink
    English
    arrow-up
    4
    ·
    5 months ago

    As others said, the initial setup may consume some time, but once it’s running, it just works. I dockerize almost everything and have automatic backups set up.

  • CronyAkatsuki@lemmy.cronyakatsuki.xyz
    link
    fedilink
    English
    arrow-up
    4
    ·
    edit-2
    5 months ago

    Minimal, I have to force myself to check the servers for updates atleast once a week.

    Main problem for me is I automated podman and docker updates with their respective autoupdate mechanisms and use ntfy for push notifications so I know if a service stops working and I had an update recently on it that it’s an update issue.

    Also have uptime monitor wih uptime kuma to monitor state of my services to catch them not working before I do, also ntfy for push notifications.

    Also have grafana+prometheus seted up on my biggest server for monitoring and alerting with alertmanager+mail to get notifications on even more errors.

    So in general I only have to worry about occasional once every few months error and updates of the host system (debian).

  • DeltaTangoLima@reddrefuge.com
    link
    fedilink
    English
    arrow-up
    4
    ·
    edit-2
    5 months ago

    Not heaps, although I should probably do more than I do. Generally speaking, on Saturday mornings:

    • Between 2am-4am, Watchtower on all my docker hosts pulls updated images for my containers, and notifies me via Slack then, over coffee when I get up:
      • For containers I don’t care about, Watchtower auto-updates them as well, at which point I simply check the service is running and purge the old images
      • For mission-critical containers (Pi-hole, Home Assistant, etc), I manually update the containers and verify functionality, before purging old images
    • I then check for updates on my OPNsense firewall, and do a controlled update if required (needs me to jump onto a specific wireless SSID to be able to do so)
    • Finally, my two internet-facing hosts (Nginx reverse proxy and Wireguard VPN server) auto-update their OS and packages using unattended-upgrades, so I test inbound functionality on those

    What I still want to do is develop some Ansible playbooks to deploy unattended-upgrades across my fleet (~40ish Debian/docker LXCs). I fear I have some tech debt growing on those hosts, but have fallen into the convenient trap of knowing my internet-facing gear is the always up to date, and I can be lazy about the rest.

  • N-E-N@lemmy.ca
    link
    fedilink
    English
    arrow-up
    4
    ·
    5 months ago

    As a complete noob trying to make A TrueNAS server, none and then suddenly lots when idk how to fix something that broke

  • thirdBreakfast@lemmy.world
    link
    fedilink
    English
    arrow-up
    3
    ·
    edit-2
    5 months ago

    I run two local physical servers, one production and one dev (and a third prod2 kept in case of a prod1 failure), and two remote production/backup servers all running Proxmox, and two VPSs. Most apps are dockerised inside LXC containers (on Proxmox) or just docker on Ubuntu (VPSs). Each of the three locations runs a Synology NAS in addition to the server.

    Backups run automatically, and I manually run apt updates on everything each weekend with a single ansible playbook. Every host runs a little golang program that exposes the memory and disk use percent as a JSON endpoint, and I use two instances of Uptime Kuma (one local, and one on fly.io) to monitor all of those with keywords.

    So -

    • weekly: 10 minutes to run the update playbook, and I usually ssh into the VPS’s, have a look at the Fail2Ban stats and reboot them if needed. I also look at each of the Proxmox GUIs to check the backs have been working as expected.
    • Monthly: stop the local prod machine and switch to the prod2 machine (from backups) for a few days. Probably 30 minutes each way, most of it waiting for backups.
    • From time to time (if I hear of a security update), but generally every three months: Look through my container versions and see if I want to update them. They’re on docker compose so the steps are just backup the LXC, docker down, pull, up - probs 5 minutes per container.
    • Yearly: consider if I need to do operating systems - eg to Proxmox 8, or a new Debian or Ubuntu LTS
    • Yearly: visit the remotes and have a proper check/clean up/updates
  • Showroom7561@lemmy.ca
    link
    fedilink
    English
    arrow-up
    3
    ·
    5 months ago

    Synology user running some docker contains.

    Very, very little maintenance. If there’s an update for something on docker, a simple click in the container manager, and it’s done. Yes, I can automate, but prefer to manually do these as many of the docker apps I use are in high development and I like to know what’s changing with each version.

    Synology packages update easily, and the system updates happen only once in a while. A click and reboot.

    I’ve tried to minimize things as much as possible, and to make things easier for me. One day, someone in my family will need to take over, and I don’t want to over-complicate things for them, lest they lose all our family photos, documents, etc.

    I probably spend more time keeping the fans on my actual NAS clean of dust, than I do maintaining the software end of things. LOL