The thing that happened

I have noticed really bad inbound latency on the VPS I host this site on. Way slower than what skhron says it should be on their site so clearly something was up. I noticed when pinging example.com the first ping took a while but after that it was fine compared to pinging its IP address which was pretty much instant so clearly the DNS server was shit. While doing this I had the VPS dashboard open and saw DNS server settings which I assumed (assuming things often lead to bad outcomes) would change something on the DHCP server. After that I noticed the network on the VPS go completely down. Only way I could get in was with VNC and from the server I couldn't ping anything not even IP addresses. After changing back to the default DNS the issue continued. Turned out there is no DHCP the VPS server just configures everything statically with some scripts and it messed up. I use freebsd which wasn't well tested by whoever made that system so it nuked itself.

Even after turning on manual mode and setting up the networking by hand it was still fucked. The only thing I backed up was my git repos. All the configs (some of them a pain in the ass to make) were stuck on there without any way to get them off except painfully hand copy with the 80 column VNC display. Linux live environments don't support zfs and the drivers to make that happen require rebooting which live environments reset on boot. Uploading freebsd live images to the VPS would be painful due to the stuppa way it handles them.

Because I didn't want to go through any of those things I contacted tech support (I hate contacting those types of places). Most places have an under powered LLM running the show, people who have good writing/talking skills but have computer skills limited to using word processors and web browsers, over worked intern who has million other things on their mind... Skhron turned out to be one of the rare gems that has tech support that not only knows what they are doing but also has the time/will to do so. The tech support person actually took the time to recreate the issue on another freebsd instance while most tech support people by now would have sent a copy pasted corporate answer that translates to "go fuck yourself". After a few hours they sent a manual route config which fixed it and they also said to configure DNS from /etc/resolv.conf instead.

What was learned

Less so when using more popular operating systems on a VPS but do be careful with dashboard settings that might be changing things in the operating system itself instead of on the network and VM. If you can preform a task from a config file that is often the safer choice because you don't know what the fuck the VPS dashboard is doing behind the scenes. Also anything you don't want to loose make backups of because even if you do everything right things will still fuck up! If this issue wasn't fixed a fresh install would have been the only other option. If up time is important for you then automatic backups and full backups are a must have. Regardless keep an offline backup of important things on your own hard drive.