I was burned out. Not just from work but from the lab as well. I nearly tore down the whole lab because it just wasn't getting used. I had/have a plan to do more short lived and cloud stuff, but that really wasn't gaining traction either. Then someone at work started messing around with Rancher and it inspired me a bit to find out what was actually wrong with my Rancher cluster.
I thought etcd had lost quorum after a couple power outages and that my poking and prodding had just left it in a permanently bad state. But it turns out it was able to recover once I fixed the nodes' dns resolvers and vcenter access. Once vcenter access was resolved it started processing a bunch of stuff and eventually sorted itself out so that the pods were mostly running. I had some other things that caused some pod issues like the mysql server being down. The final issue I had was the nginx ingress not being able to find the backends. I tried completely redeploying the nginx ingress pod but that didn't make a difference. I also upgraded rancher but again no change. Finally, I just re-created the load balancer entries using the service instead of workload and that either fixed it though it seems like a kind of jiggle the handle fix and I'm not completely sure what caused the original issue of nginx not being able to find the keys in the first place. Though in all likelihood nginx had just gotten into a bad state when the rest of the cluster did as well.
So now my rancher cluster is happily running emby, firefly III, and Bookstack. Next, I will probably move the mysql databases over to it as well so I can try out scheduled jobs in k8s.
One of the big lab things I've always wanted is to be able to deploy the whole lab in a single command. I'm maybe halfway there and lately with not really being as concerned about the lab hardware I've started working more toward being able to quickly deploy the things I want to use without having to rely on heavy things like Katello. Ironically, while looking at Rancher I found their quickstart and it's exactly the kind of thing I'm looking to build.
On a completely random note Rancher is starting to remind me a lot of VMware, they have that sense of maturity and polish that is lacking in a lot of other products. But Rancher's gui is so easy to use I often find myself using it rather than doing things by code.