flak rss random

go module bloat

For some of the software I write, I try to make occasional releases, and for the go software I vendor the dependencies so it’s all there. I was just reminded that I hadn’t made a release of something in five years, so I reran my release script, and ended up with a tarball that was 10x bigger than the previous one. Some terrible choices have been made.

 317140 May  6  2019 humungus-0.9.6.tgz
3094567 Apr 30 22:25 humungus-0.9.7.tgz

There’s been five years of development, but that’s really not that much. I most certainly have not 10xed the functionality. Where did this all come from?

I poked around the vendor directory for a bit for some large files. One of them is hsluv-snapshot-rev4.json, a 1.5MB json file. I wouldn’t recommend clicking on that link, unless your browser likes displaying single lines of text 1.5MB bytes long.

I have no idea what this file does. The only color I need is for a little green menu in the terminal admin interface. I checked, and go-colorful was a dependency in previous releases, but it didn’t have this file at the time. Apparently I still don’t need it. I just deleted it after running go vendor and everything still works. That reduced the size of the final release tarball by 500K.

2MB (post compression) still to go, or thereabouts. I was in a hurry, so I didn’t actually trim anything more down. The majority of the code is the golang/sys/unix repository. I still have to find out what’s dragging this in, and then delete or rewrite it.

Ah, there it is. tcell nonblock_bsd.go imports sys/unix. 9.5MB of vendored source code to set a flag on a file descriptor.

I have tried to be mindful of the number of dependencies I’m adding. I keep an eye on go.mod to make sure it doesn’t explode. But I’ve been lax in running du on the vendor directory. Even limited sets of dependencies are subject to extreme bloat.

Posted 01 May 2024 18:46 by tedu Updated: 01 May 2024 22:38
Tagged: go programming

memory leak proof every C program

Memory leaks have plagued C programs for as long as the language has existed. Many solutions have been proposed, even going so far as to suggest we should rewrite C programs in other languages. But there’s a better way.

Presented here is a simple solution that will eliminate the memory leaks from every C program. Link this into your program, and memory leaks are a thing of the past.

#include <dlfcn.h>
#include <stdio.h>

struct leaksaver {
        struct leaksaver *next;
        void *pointer;
} *bigbucket;

void *
malloc(size_t len)
        static void *(*nextmalloc)(size_t);
        nextmalloc = dlsym(RTLD_NEXT, "malloc");
        void *ptr = nextmalloc(len);
        if (ptr) {
                struct leaksaver *saver = nextmalloc(sizeof(*saver));
                saver->pointer = ptr;
                saver->next = bigbucket;
                bigbucket = saver;
        return ptr;

Every allocated pointer is saved in the big bucket, where it remains accessible. Even if no other references to the pointer exist in the program, the pointer has not leaked.

It is now entirely optional to call free. If you don’t call free, memory usage will increase over time, but technically, it’s not a leak. As an optimization, you may choose to call free to reduce memory, but again, strictly optional.

Problem sovled!

Posted 19 Jan 2024 16:55 by tedu Updated: 19 Jan 2024 16:55
Tagged: c programming rants

write your own terminal

What’s next after you write your own text editor and mail client? How about a terminal? In fact, as a practice exercise or to learn some new skills, I’d say a terminal emulator makes for a much better target. It’s composed of many parts, but at an approachable level, making it easy to make tangible progress. In this way, I think it makes for a good introductory project. At the same time, there’s a very long tail of features that can be added to keep things interesting.


Posted 10 Nov 2023 07:44 by tedu Updated: 11 Nov 2023 18:42
Tagged: programming software

two hackers one keyboard two ways

There’s an amazing and famous scene of two hackers sharing a keyboard on NCIS, but it gets a lot of derision on the internet. What people don’t realize is that their mockery only reveals their own lack of skill. As everyone knows, a true hacker has mastered the art of dual wielding, and a keyboard that can be dual wielded by one might also be used by two hackers in tandem.


Posted 31 Oct 2023 04:13 by tedu Updated: 31 Oct 2023 04:13
Tagged: programming software x11

porting linux pledge to go

I like using pledge and unveil in my web apps. Especially unveil offers a nice degree of protection against common web app problems, like the dreaded double dot traversal. For go, I use a simple wrapper which gets pasted into each project.


Posted 25 Oct 2023 17:15 by tedu Updated: 27 Oct 2023 18:13
Tagged: go openbsd programming

banging errors in go

One of the many problems with programming in go is there’s functions, and the functions are written by people, and the people make mistakes, and the functions return errors, and now you have to check for the errors. This is all very tedious and tiresome. We can’t fix the people who cherish their imperfections as a sign of humanity, but we can change go to pretend the errors aren’t there.


Posted 19 Oct 2023 18:09 by tedu Updated: 20 Oct 2023 02:04
Tagged: go programming


I was planning on working on a redesign of a photo site, and wanted to use JPEG-XL as the preferred image format for storage. The only implementation I know of is the libjxl reference implementation written in C++. Alas, JPEG successors have not had a great security track record recently, and I would much prefer not to run this code on my server.


Posted 03 Oct 2023 20:57 by tedu Updated: 03 Oct 2023 20:57
Tagged: go programming wasm

an aborted experiment with server swift

I wanted to write a fun little experimental ActivityPub server. I have a solid idea how to handle this in go, or at least I think I do, so that’d be pretty boring. Instead, let’s try a new technology.


Posted 28 Sep 2023 06:23 by tedu Updated: 28 Sep 2023 06:23
Tagged: programming

www which wasm works

We’ve gotten libjxl built for wasm. It was a struggle, but we got it done, and we’re ready to run it. WASM is a straightforward standard designed for ease of implementation, so this should be a walk in the park.


Posted 22 Sep 2023 17:38 by tedu Updated: 22 Sep 2023 17:38
Tagged: go programming

on building jpeg-xl for wasm

I was planning on working on a redesign of a photo site, and wanted to use JPEG-XL as the preferred image format for storage. The only implementation I know of is the libjxl reference implementation written in C++. Alas, it hasn’t been a good week for JPEG successors, and I would much prefer not to run this code on my server.


Posted 14 Sep 2023 08:08 by tedu Updated: 14 Sep 2023 08:08
Tagged: programming