For some of the software I write, I try to make occasional releases, and for the go software I vendor the dependencies so it’s all there. I was just reminded that I hadn’t made a release of something in five years, so I reran my release script, and ended up with a tarball that was 10x bigger than the previous one. Some terrible choices have been made.

 317140 May  6  2019 humungus-0.9.6.tgz
3094567 Apr 30 22:25 humungus-0.9.7.tgz

There’s been five years of development, but that’s really not that much. I most certainly have not 10xed the functionality. Where did this all come from?

I poked around the vendor directory for a bit for some large files. One of them is hsluv-snapshot-rev4.json, a 1.5MB json file. I wouldn’t recommend clicking on that link, unless your browser likes displaying single lines of text 1.5MB bytes long.

I have no idea what this file does. The only color I need is for a little green menu in the terminal admin interface. I checked, and go-colorful was a dependency in previous releases, but it didn’t have this file at the time. Apparently I still don’t need it. I just deleted it after running go vendor and everything still works. That reduced the size of the final release tarball by 500K.

2MB (post compression) still to go, or thereabouts. I was in a hurry, so I didn’t actually trim anything more down. The majority of the code is the golang/sys/unix repository. I still have to find out what’s dragging this in, and then delete or rewrite it.

Ah, there it is. tcell nonblock_bsd.go imports sys/unix. 9.5MB of vendored source code to set a flag on a file descriptor.

I have tried to be mindful of the number of dependencies I’m adding. I keep an eye on go.mod to make sure it doesn’t explode. But I’ve been lax in running du on the vendor directory. Even limited sets of dependencies are subject to extreme bloat.

Can you use go run for scripting? I wrote a small program to find out.

package main

import (
    "fmt"
    "os"
    "os/exec"
    "time"
)

func main() {
    start := time.Now()
    gogo := fmt.Sprintf(`package main
    import "fmt"
    func main() {
        fmt.Printf("the time was %s\n")
    }`, start)
    os.WriteFile("gogo.go", []byte(gogo), 0666)
    cmd := exec.Command("go", "run", "gogo.go")
    cmd.Stdout = os.Stdout
    cmd.Run()
    end := time.Now()
    fmt.Printf("that took %s\n", end.Sub(start))
}

$ go run gorun.go
the time was 2024-01-05 12:04:01.042488 -0500 EST m=+0.000077959
that took 329.81325ms

On my somewhat older m3 running openbsd, it takes about 330ms. On a less old chromebook, it takes about 100ms. Not bad. On an M1 macbook, it varies wildly between 150ms and 300ms.

So I think not fast enough for interactive use. Probably fast enough for a continuously changing cronjob, though I wonder why you wouldn’t change the inputs. But I guess if you’re a big believer in configuration through compilation, and also have constantly changing requirements, it’s fast enough to get by.

For comparison, yaegi is still 10x or more quicker at putting something on the screen.

go build

I almost exclusively run go build. Actually, I run make and that runs go build, because that’s how my fingers work. So it wasn’t until recently that I noticed that go run is as fast as it is. I was used to a very observable one second per build.

Turns out go build will run hg stat for every build, which go run skips, and that was the slowest part of the process. Easily remedied by switching to chg.

Go 1.21 added experimental support for fixing the loop capture bug. For reasons it’s never really bothered me, but the other loop bug does bite me some times. This is the one where the loop values are values.

I want references but go doesn’t have references.

values

The problem is easy to see when you’re looking for it, and hard to see when you’re glossing over the code.

func resetScores(players []Player) {
    for _, player := range players {
        player.score = 0
    }
}

Before and after “the” loop fix, this will dutifully set the player’s score to zero. But which player? An anonymous player that only exists in the loop. The array will remain unchanged. Ugh.

hand fix

Fixing this requires slightly rewriting the loop.

func resetScores(players []Player) {
    for i := range players {
        players[i].score = 0
    }
}

This looks kinda weird I think, and it’s not my first instinct to write it this way. I really just want a way to say hi, I’m iterating over the elements of this array, give me a reference to them.

mechanical fix

So let’s add references to go.

func resetScores(players []Player) {
    for _, &player := range players {
        player.score = 0
    }
}

This can be translated by powerful state of the art tools into the go you’d have to write otherwise.

func resetScores(players []Player) {
    for  _i, _ := range players {
        player := &players[_i]
        player.score = 0
    }
}

oh well

What’s annoying is the above broken form works if the array contains pointers. A lot of my code uses pointers, so I get used to writing the first form, and then it breaks when I do something different. Or I decide to optimize the code and reduce allocations by switching from []*Player to []Player without a careful audit of the impacted loops. Alas, I probably wouldn’t think to use the reference form even if available until it’s too late.

The common mistakes wiki lists the capture bug twice, but doesn’t mention this bug.

I like using pledge and unveil in my web apps. Especially unveil offers a nice degree of protection against common web app problems, like the dreaded double dot traversal. For go, I use a simple wrapper which gets pasted into each project.

more...

One of the many problems with programming in go is there’s functions, and the functions are written by people, and the people make mistakes, and the functions return errors, and now you have to check for the errors. This is all very tedious and tiresome. We can’t fix the people who cherish their imperfections as a sign of humanity, but we can change go to pretend the errors aren’t there.

more...

I was planning on working on a redesign of a photo site, and wanted to use JPEG-XL as the preferred image format for storage. The only implementation I know of is the libjxl reference implementation written in C++. Alas, JPEG successors have not had a great security track record recently, and I would much prefer not to run this code on my server.

more...

We’ve gotten libjxl built for wasm. It was a struggle, but we got it done, and we’re ready to run it. WASM is a straightforward standard designed for ease of implementation, so this should be a walk in the park.

more...

Some people, when presented with a data structure, think let’s encode this to json. Now they have two problems. Encoding and decoding. In response to this dilemma, various libraries were created, such as rust serde or go encoding/json, to facilitate drama and debate about which approach is best.

more...

One of the traps in go is the reuse of loop variables, confounding novices and even catching the unwary expert. It’s so bad they may even change the language to fix it.

more...

For the first version of where’s all the code I did a simple tree walk to find all the code, one file at a time. But I wrote it in go, a supposedly concurrency friendly language, so it seemed the obvious thing should be to walk the tree in parallel. This went great, until it didn’t, but then it ended up okay.

more...