In a garbage collected language, one needs to be cautious about leaks caused by errant references, but caution can turn to paranoia. In go, I like to make little fifo queues by appending to the tail of a slice. But does this grow forever?

s = append(s[1:], n)

One theory is that this will eventually consume all memory, as somehow the slice grows and grows. The garbage collector is unable to reduce its size because the head of the slice is still out there somewhere.

Another theory is no, the slice gets reallocated when it needs to expand, but only the referenced section gets copied.

We can test this!

package main

import (
    "fmt"
    "math/rand"
)

func main() {
    var s []int

    grow := false
    for i := 0; i < 10000000000; i++ {
        n := rand.Int()
        if grow || len(s) < 100 {
            s = append(s, n)
        } else {
            s = append(s[1:], n)
        }
    }
    fmt.Printf("%v\n", s)
}

This runs basically forever, but uses very little memory. Changing grow to true, it reliably explodes.

I knew this, and was already relying on it, but also in most cases a slow leak may take weeks to develop and go unnoticed. Thirty seconds to check.

A follow up to what is the go proxy even doing? now with more answers. Russ Cox (rsc) and I traded a few emails trying to work out what was going on. They’re going to make a few changes, and I also learned a few things it would have been helpful to know.

more...

The go module proxy caches requests from users, so everyone has a consistent and reliable experience regardless of upstream host. But what if the go proxy is contributing to the instability of the upstream host? There have been other complaints about the go proxy, that’s just the way it works, but I collected a few minutes of logs to examine that may be interesting.

more...

Perl has a regex option /o, which pretends to optimize your code, but actually introduces bugs. Go has been missing out. Until now.

We need regexp of course, but also text/template for interpolation. Add in runtime to get the pc, and we have all the elements for a cache of only once regex.

func Compile(s string, vals any) (*regexp.Regexp, error) {
        cacheMtx.Lock()
        defer cacheMtx.Unlock()
        pc, _, _, _ := runtime.Caller(1)
        k := key{s: s, pc: pc}
        re := cache[k]
        if re != nil {
                return re, nil
        }
        t := template.New("regex")
        t, err := t.Parse(s)
        if err != nil {
                return nil, err
        }
        var sb strings.Builder
        err = t.Execute(&sb, vals)
        if err != nil {
                return nil, err
        }
        re, err = regexp.Compile(sb.String())
        if err != nil {
                return nil, err
        }
        cache[k] = re
        return re, nil
}

type key struct {
        s  string
        pc uintptr
}

var cacheMtx sync.Mutex
var cache = make(map[key]*regexp.Regexp)

And a test program.

func main() {
        fmt.Printf("starting\n")
        inputs := []string{"Apple", "bananas", "42"}
        for i := 0; i < 3; i++ {
                re, err := regexpo.Compile("{{ call .Letters }}", map[string]any{
                        "Letters": func() string {
                                time.Sleep(1 * time.Second)
                                fmt.Printf("substitute\n")
                                return "[a-z]"
                        },
                })
                if err != nil {
                        fmt.Printf("failure: %s\n", err)
                        return
                }
                ok := re.MatchString(inputs[i])
                fmt.Printf("match: %v\n", ok)
        }
        fmt.Printf("done.\n")
}

The substitution is only done once.

I had the idea to write a little shell in go. Called gosh, of course. There’s a few people playing with the same theme, but nothing exactly the same.

more...

I used to use the go log package, then I switched to the slog package, and it’s been a bumpy ride.

more...

Trying to go too fast may be slow.

more...

It’s been one year since our previous adventure, so it’s time for another round of guess why that dependency shows up in the tarball. This time we’re looking at honk, an ActivityPub server that’s supposed to be idiosyncratic with minimal dependencies, so you can keep all your attention focused where it’s needed.

more...

Poking through layers of indirection in go trying to recover some efficiency.

more...

For some of the software I write, I try to make occasional releases, and for the go software I vendor the dependencies so it’s all there. I was just reminded that I hadn’t made a release of something in five years, so I reran my release script, and ended up with a tarball that was 10x bigger than the previous one. Some terrible choices have been made.

 317140 May  6  2019 humungus-0.9.6.tgz
3094567 Apr 30 22:25 humungus-0.9.7.tgz

There’s been five years of development, but that’s really not that much. I most certainly have not 10xed the functionality. Where did this all come from?

I poked around the vendor directory for a bit for some large files. One of them is hsluv-snapshot-rev4.json, a 1.5MB json file. I wouldn’t recommend clicking on that link, unless your browser likes displaying single lines of text 1.5MB bytes long.

I have no idea what this file does. The only color I need is for a little green menu in the terminal admin interface. I checked, and go-colorful was a dependency in previous releases, but it didn’t have this file at the time. Apparently I still don’t need it. I just deleted it after running go vendor and everything still works. That reduced the size of the final release tarball by 500K.

2MB (post compression) still to go, or thereabouts. I was in a hurry, so I didn’t actually trim anything more down. The majority of the code is the golang/sys/unix repository. I still have to find out what’s dragging this in, and then delete or rewrite it.

Ah, there it is. tcell nonblock_bsd.go imports sys/unix. 9.5MB of vendored source code to set a flag on a file descriptor.

I have tried to be mindful of the number of dependencies I’m adding. I keep an eye on go.mod to make sure it doesn’t explode. But I’ve been lax in running du on the vendor directory. Even limited sets of dependencies are subject to extreme bloat.