For some of the software I write, I try to make occasional releases, and for the go software I vendor the dependencies so it’s all there. I was just reminded that I hadn’t made a release of something in five years, so I reran my release script, and ended up with a tarball that was 10x bigger than the previous one. Some terrible choices have been made.

 317140 May  6  2019 humungus-0.9.6.tgz
3094567 Apr 30 22:25 humungus-0.9.7.tgz

There’s been five years of development, but that’s really not that much. I most certainly have not 10xed the functionality. Where did this all come from?

I poked around the vendor directory for a bit for some large files. One of them is hsluv-snapshot-rev4.json, a 1.5MB json file. I wouldn’t recommend clicking on that link, unless your browser likes displaying single lines of text 1.5MB bytes long.

I have no idea what this file does. The only color I need is for a little green menu in the terminal admin interface. I checked, and go-colorful was a dependency in previous releases, but it didn’t have this file at the time. Apparently I still don’t need it. I just deleted it after running go vendor and everything still works. That reduced the size of the final release tarball by 500K.

2MB (post compression) still to go, or thereabouts. I was in a hurry, so I didn’t actually trim anything more down. The majority of the code is the golang/sys/unix repository. I still have to find out what’s dragging this in, and then delete or rewrite it.

Ah, there it is. tcell nonblock_bsd.go imports sys/unix. 9.5MB of vendored source code to set a flag on a file descriptor.

I have tried to be mindful of the number of dependencies I’m adding. I keep an eye on go.mod to make sure it doesn’t explode. But I’ve been lax in running du on the vendor directory. Even limited sets of dependencies are subject to extreme bloat.

Memory leaks have plagued C programs for as long as the language has existed. Many solutions have been proposed, even going so far as to suggest we should rewrite C programs in other languages. But there’s a better way.

Presented here is a simple solution that will eliminate the memory leaks from every C program. Link this into your program, and memory leaks are a thing of the past.

#include <dlfcn.h>
#include <stdio.h>

struct leaksaver {
        struct leaksaver *next;
        void *pointer;
} *bigbucket;

void *
malloc(size_t len)
{
        static void *(*nextmalloc)(size_t);
        nextmalloc = dlsym(RTLD_NEXT, "malloc");
        void *ptr = nextmalloc(len);
        if (ptr) {
                struct leaksaver *saver = nextmalloc(sizeof(*saver));
                saver->pointer = ptr;
                saver->next = bigbucket;
                bigbucket = saver;
        }
        return ptr;
}

It is now entirely optional to call free. If you don’t call free, memory usage will increase over time, but technically, it’s not a leak. As an optimization, you may choose to call free to reduce memory, but again, strictly optional.

Problem sovled!

What’s next after you write your own text editor and mail client? How about a terminal? In fact, as a practice exercise or to learn some new skills, I’d say a terminal emulator makes for a much better target. It’s composed of many parts, but at an approachable level, making it easy to make tangible progress. In this way, I think it makes for a good introductory project. At the same time, there’s a very long tail of features that can be added to keep things interesting.

more...

There’s an amazing and famous scene of two hackers sharing a keyboard on NCIS, but it gets a lot of derision on the internet. What people don’t realize is that their mockery only reveals their own lack of skill. As everyone knows, a true hacker has mastered the art of dual wielding, and a keyboard that can be dual wielded by one might also be used by two hackers in tandem.

more...

I like using pledge and unveil in my web apps. Especially unveil offers a nice degree of protection against common web app problems, like the dreaded double dot traversal. For go, I use a simple wrapper which gets pasted into each project.

more...

One of the many problems with programming in go is there’s functions, and the functions are written by people, and the people make mistakes, and the functions return errors, and now you have to check for the errors. This is all very tedious and tiresome. We can’t fix the people who cherish their imperfections as a sign of humanity, but we can change go to pretend the errors aren’t there.

more...

I was planning on working on a redesign of a photo site, and wanted to use JPEG-XL as the preferred image format for storage. The only implementation I know of is the libjxl reference implementation written in C++. Alas, JPEG successors have not had a great security track record recently, and I would much prefer not to run this code on my server.

more...

I wanted to write a fun little experimental ActivityPub server. I have a solid idea how to handle this in go, or at least I think I do, so that’d be pretty boring. Instead, let’s try a new technology.

more...

We’ve gotten libjxl built for wasm. It was a struggle, but we got it done, and we’re ready to run it. WASM is a straightforward standard designed for ease of implementation, so this should be a walk in the park.

more...

I was planning on working on a redesign of a photo site, and wanted to use JPEG-XL as the preferred image format for storage. The only implementation I know of is the libjxl reference implementation written in C++. Alas, it hasn’t been a good week for JPEG successors, and I would much prefer not to run this code on my server.

more...