Minor Updates

This is mostly a post to try to keep up with the blogging habit: I’m currently evaluating Linux distributions on my laptop and putting together a post of random stuff makes for a good test of the Docker install on here, and keeps me remembering to write things here. So, here’s some random stuff.

The Mastodon server has been up for a month and a half now, which has been nice. I’ve mostly decided to keep with the individual instance rather than move to a different one (though I did stake a claim on the mastodon.mit.edu server - I’m @jkachel@mastodon.mit.edu there). I’ve been following the story of the hachyderm.io instance by dint of following Kris Nóva over there and that’s been interesting - that instance would probably be where I move to if I get tired of running my own; they’re doing a pretty great job with it. I’ve mostly given up on Twitter. My apartment has heating so I don’t really need the ever-larger dumpster fire that it’s become.

I’ve pulled the Twitter widget from the sidebar - sometime soon I’ll put up either a RSS embed or an ActivityPub thing to grab my Mastodon timeline. (Did you know that most, if not all, Mastodon timelines can be subscribed to via RSS? It’s true. Just append .rss to the end and you get an RSS feed. Like so.)

Some Linux distro tryouts are commencing at the moment. I’m sort of planning a new system build now that all the new AMD stuff is out, and the plan is for it to be a red team build and also a Linux build. So, I’m fiddling a bit with Linux distributions on my Lenovo Yoga laptop. I’ve tried straight Debian 11 and Linux Mint 21 so far. I liked Linux Mint anyway but I may try Fedora or Alma or Rocky too. Debian 11 gave me some flashbacks nostalgia for old times installing Linux - out of the box, the wireless didn’t work, it dropped me into a screen to choose from any of dozens of ancient Ethernet drivers for my modern Gigabit Realtek card for some reason (i.e., unnecessarily), and it wouldn’t take my password. Reminded me a bit of installing Debian 1.3.1 off of some like $2 CheapBytes CDs back around, oh, 1998 or so. I don’t think I ever got that installed properly. (In my defense, I was a Slackware user at that point and more familiar with the “compile everything” sort of situation that existed at the time. Package management was a new thing back then - Red Hat and Debian were leading the way on it with managers that actually cared about dependencies - and man oh man were things broken for a long time.)

I have succumbed to YouTuber sponsorship spots and now I own a bunch of Vessi shoes. (Not that I do YouTube, but I do watch a lot of it.) They’re pretty comfy. I had some but got more because of a bundle deal with some gloves and a toque (they’re Canadian, so..), but also because I’m a bit afraid of wearing out the pair I wear all the time.

I’ll probably be working on testing out my 10Gb Ethernet switch some more over the holiday break and will write up a thing about that. I don’t have any SFP+ cables yet but I should at least be able to get to a console and use the lone GigE port on it. I’ve gotten a NetApp 1U 16-port switch with redundant power supplies; it’s gotten a full Noctua fan swap so might be interesting to do a bit of writing about. So far I’m only about $200ish in on 10GbE rollout - the switch was < $100 and the 4 dual-port NICs I bought were like $25/ea. Cabling will be another couple hundred (from FS.com, probably) but these are entirely reasonable prices.

That’s it for now - time to see if this publishes properly.

Stupid Computer Drag Racing

Two mini PCs, facing off against each other in a race that’s somewhat network dependent. What fun!

I got a couple of those weird mini NUC-style PCs. They’re very cosmetically similar PCs, of the “AK2” variety, that you can get on the Amazon for between $70 and $175 depending on what deals are going on and spec. They were bought for other things, but I figured why not see what the difference is between a couple of generations of Celeron?

Similar things on each: both have 2x HDMI ports, a smattering of USB 2 and 3 ports, RTL8111-family GbE network, onboard single-port SATA, AC wireless (one with an Intel card, one with a Realtek). The differences are memory, CPU, and storage (outside the SATA).

AK2: J3455 - Celeron J3455 Apollo Lake (4c4t), 6GB RAM, 64GB eMMC, no NVMe slot (there is an open slot but it’s Mini-PCI for some reason)
AK2 “Pro”: N5095 - Celeron N5095 Jasper Lake (4c4t), 12GB RAM, 256GB NVMe SSD (this came with an SSD that threw a ton of errors during Ubuntu installation.. swapped it for a known-good 256GB drive but not sure if that was just weirdness or that the pack-in drive is flaky)

To do the drag race, I set both of these up with Ubuntu Server 22.04 LTS with full updates, pyenv, and Docker Engine; and connected them to my network via Ethernet. The Ethernet connection is somewhat bottlenecked as I’m using the two Ethernet ports on the TP-Link Deco P9 mesh pod in the room where they are, and that’s generally using the slower HomePNA powerline backhaul to the rest of the netwrok. But, they ranged from 7-10MB/s when both were hitting the network simultaneously to about 15MB/s when one got full shouting rights over the cable, and they were run so they were both basically sharing space all the time.

The workload I chose was setting up an Open edX devstack instance on each from scratch. Open edX is a pretty big thing - a full “large and slow” setup ends up with 14 Docker containers - and there’s a smattering of compiling stuff and decompression and database ops and all that, so it seemed like a good fit. (Plus, I’m really familiar with it. The day job mostly entails writing software that interfaces with Open edX in some manner, so I’ve run it on much faster systems than these two.) However, it’s worth noting that some of these steps are very network bound, and those steps are noted as such. I did include the preliminary Python setup steps here too, so that’s a lot more compiling.

Here’s the results. The times listed are the Real time from time(1).

Round 2: now with identical TeamGroup AX2 SATA SSDs (512GB) connected to onboard storage and fresh install of Ubuntu Server 22.04. Some of the network speeds went up here; the machines got kinda out of sync and so they had the network to themselves for a bit.

The most telling of these is the first and last result - pyenv install 3.11.0 and make dev.provision are places where you can really tell what the difference a couple of generations of Intel architecture enhancement make. As a reminder, these two chips are about 5 years apart (Skylake to Ice Lake; 6th gen Core to 11th gen). Interestingly, the performance difference is about the same as the cost difference. The J3455 system was about $75 and the N5095 system was about $150.

Neither of these systems are particularly performant (and they’re probably gonna lose those 512GB SSDs) but they make good point of need systems for lower-end tasks. They’re pretty small - roughly 5in square and about 3in high. The J3455 is going to be a Home Assistant box because it’ll outperform the Raspberry Pi 3 that’s currently doing that task and it’ll fit nearly anywhere.

A couple weird hardware things I’ve noticed:

• They both have a USB-C under the lid. You can get power out of it, but it doesn’t seem to do anything. I plugged a drive into it and nothing.
• The J3455 has a micro SD card reader on the board (that evidently works). The N5095 one doesn’t.
• The J3455 has a mini PCI slot on it. I was thinking maybe I could put a M.2 2242 drive but nope! I suppose you could use it for a WwAN modem or something, though.. do they still make those in mini-PCI? I have a CDMA one floating around, I could try it to see if it works in the slot..
• If you get one and take it apart, be careful about the WiFi antennas. I disconnected one taking apart the J3455 unit and in the process of trying to wedge the connector underneath the plastic thing they glued down to the top of the WiFi module (to keep the antennas connected..) I really broke the other one. Surprisingly it still connects to my local network, but that may be a function of it being basically next to one of the mesh pods.
• I also learned that Realtek USB WiFi NICs are less than great for use in Linux.

Most of this was from some videos by Goodmonkey on YouTube. He had some better luck with the AK2/GK2 pricing than I did. (But I might also look at deploying these TP-Link Omada WiFi dingles..)

Mastodon Week 2

After another week of Mastodon instance running, I’ve learned a few more things. So, here they are.

Sizing: Sizing was a problem! Turned out this was due to some choices I’d made (which I’ll discuss later). By the end of things, I went from a giant Sidekiq job backlog to none at all, and from a 2 vCPU/4GB droplet to a 4 vCPU/8GB droplet. This is actually too big now but it’s going to stay that way until I get some actual monitoring going on the machine. DigitalOcean provides some but I’d like some better stuff.

Relays: I added like 7 relays to the server to help grab stuff for my federated timeline. Relays are sort of a garden hose of posts - if you’re connected to a relay, your posts get aggregated and resent by the relay to the other connected instances. For me, this ended up creating two problems.

1. I had too many of them and that spawned a ton of jobs that my smaller droplet couldn’t keep up with. If you want to add a relay to your instance, maybe only do one or two. (And you’ll want to be very judicious about what relays you add.)
2. As for the relays I’d added from this list that I’d mentioned in the previous post: yeah, don’t do that. If you add a relay, you need to go through the published connections list to see who’s connected to the relay first or you’re going to get a bunch of hate speech and shitposting.

To be honest, once you get started following folks and rolling through hashtags and stuff like that, your instance will find things on its own. If there ends up being some focused relay systems in the future for certain communities, then I can see that being a thing to join up with; otherwise it really does seem to take care of itself.

Upgrades: As I’d noted, the DO one-click installer rolled forward with version 3.1.3 of the Mastodon server software. That’s pretty old and I upgraded it to 4.0.0rc3. (Aside: it’s nice that I can do that! One of the benefits of running your own stack.) It was a pretty easy process. The Mastodon docs mostly cover it but this was the gist of what I ended up doing (note that this is for the DigitalOcean one-click installer environment, yours will be different if it’s not that):

1. Stopped all Mastodon services.
2. Install newer Ruby and Node. rbenv is installed already, so you just use that to install 3.0.4 or newer. I added the NodeSource packages (see nodejs.org) to the apt sources list and installed Node 16 using that.
3. Follow the instructions in the tagged release.

I took a snapshot of my instance before starting. That took longer than the upgrade process. If you’re moving from an old version of Mastodon to a newer one and you’re out a few versions, you should go back and re-trace the release notes to see if anything special needs to be done - in my case, going from 3.1.3 to 4, the instructions for 4.0.0rc3 included all the extra steps. I would have had to do those steps even if I didn’t go to 4.0.0rc3 as they were recommended in verion 3.2 or something. (There’s probably a reason why they’re not, but I kinda think the steps attached to the 4.0.0rc3 release should probably be the steps to do going forward.)

Hashtag pinning and following: This is in 4.0 and it’s great. I heart it.

Anyway, that’s it for now. The machine is humming along nicely and as an added benefit I check Twitter far less often now. Don’t get me wrong, they’re still complimentary services (to me), but I’m enjoying being on the pachyderm site. Next thing on the list is adding some more instrumentation. You get some with DigitalOcean and I have it linked into Pulseway but it’d be nice to get some Prometheus/Grafana and/or Zabbix or something running too.