I had some go code I wanted to quickly iterate on. Go compiles pretty quickly, but not instantly. Like 2 seconds. In some places, I can use gopher-lua, which gets me pretty close to 0 second iteration delay, but there’s a big up front development cost. It’s useful for scripting an existing program and adding custom behavior, but less useful for experimenting to see what happens when I do X. What I need is an actual interpreter for go, not an interpreter in go.

My first thought of course is that luajit provides a solid interpreter base with great performance. Can we translate go to lua bytecodes somehow? This would be a fun, if rather ambitious, hacking project. Turns out this has been done. Behold gijit. Exactly what I wanted! Alas, trying to build from source failed checking out some dependency of another dependency because the right tag couldn’t be found in the git and figuring out exactly why was indecipherable madness. So no dice. But it’s a thing and it sounds cool.

Next I looked at gomacro. This is a (mostly) simpler interpreter with very few dependencies. Got it building right away. After that, I could try using it.

It’s ok for the interpreter to have some differences from the go toolchain, but our eventual goal is a code base that can be interpreted for development, then compiled for deployment. And then later we can switch back to interpreting for further development. If accommodating the interpreter causes us to diverge too much, it won’t be of much use. I don’t want to develop in a REPL, then copy code into a file, and be unable to return to it. So here’s how that goes.

hello

Very simple test. hello.go.

import "fmt"
fmt.Println("hello")

> time ./gomacro /tmp/hello.go  
hello
    0m00.09s real     0m00.04s user     0m00.02s system

That’s fast enough, produces the expected out. Very promising start.

imports

Now let’s use bcrypt.

import "fmt"
import "golang.org/x/crypto/bcrypt"

password := []byte("password")
hash, _ := bcrypt.GenerateFromPassword(password, 4)
err := bcrypt.CompareHashAndPassword(hash, password)

fmt.Println("Error:", err)

> time ./gomacro /tmp/hello.go
// warning: created file "/home/tedu/go/src/github.com/cosmos72/gomacro/imports/thirdparty/golang_org_x_crypto_bcrypt.go", recompile gomacro to use it

Oops. This isn’t a big deal though. Run go build as instructed, wait a few seconds, then try again.

time ./gomacro /tmp/hello.go
Error: <nil>
    0m00.17s real     0m00.04s user     0m00.10s system

Success. Now I can do some basic experimentation, changing the cost factor from 4 to 8 to 16, measuring the impact without waiting for recompiles.

This is a documented limitation, but the tool is nice enough to guide you along in case you forget. In the beginning, it takes a few rebuilds as you try out new packages, but they accumulate over time until you have a gomacro interpreter capable of importing just about anything. As long as you have some finite number of imports, this is only a tiny speed bump.

recursion

Let’s look at another sample. We’ll start with a full go program.

package main
import "fmt"
import "os"
import "strconv"
func a(x int) {
        if x == 0 {
                fmt.Println("A wins")
        } else {
                b(x - 1)
        }
}
func b(x int) {
        if x == 0 {
                fmt.Println("B wins")
        } else {
                a(x - 1)
        }
}
func main() {
        x, _ := strconv.Atoi(os.Args[1])
        a(x)

        fmt.Println("All done here")
}

Build and run.

> time go build mutrec.go 
    0m00.53s real     0m00.25s user     0m00.24s system
> time ./mutrec 4                                                                          
A wins
All done here
    0m00.01s real     0m00.00s user     0m00.01s system
> time ./mutrec 5 
B wins
All done here
    0m00.01s real     0m00.00s user     0m00.01s system

Now it’s gomacros turn:

> time ./gomacro /tmp/mutrec.go  
/tmp/mutrec.go:9:3: undefined identifier: b
/tmp/mutrec.go:16:3: undefined identifier: a
/tmp/mutrec.go:21:2: undefined identifier: a
    0m00.12s real     0m00.05s user     0m00.05s system

We need to prototype our functions, C style.

func b(x int)

func a(x int) {
        if x == 0 {
                fmt.Println("A wins")
        } else {
                b(x - 1)
        }
}

This elicits a warning, but runs.

> time ./gomacro /tmp/mutrec.go 5 
// warning: redefined identifier: b
A wins
All done here
    0m00.11s real     0m00.05s user     0m00.04s system

But now it won’t build in go.

> time go build mutrec.go  
# command-line-arguments
./mutrec.go:6:6: missing function body
./mutrec.go:15:6: b redeclared in this block
        previous declaration at ./mutrec.go:6:10
    0m00.33s real     0m00.12s user     0m00.24s system

So we need to create a decls.go file. Or some other workaround. And we need a bit of build machinery to hide this from the regular go build. (A file named decls.notgo seems to work well.)

I don’t necessarily write a lot of mutually recursive code, but I write tons of code in which config.go calls functions in parse.go, and a simple gomacro *.go loads the files in the wrong order.

This is not an insurmountable challenge, but it requires a bit of work to arrange things in the correct order.

main

The above example cheats a bit by skipping ahead. gomacro doesn’t call main() when interpreting a file. If we want to keep our original sources unmodified, we need to call main from another file at the end.

> time ./gomacro /tmp/decls.notgo  /tmp/mutrec.go /tmp/runit.notgo 4  
// warning: redefined identifier: b
A wins
All done here
    0m00.12s real     0m00.05s user     0m00.04s system

Another issue is that init() functions are not run.

func init() {
        fmt.Println("Greetz from init")
}

> go build mutrec.go  
> ./mutrec 4          
Greetz from init
A wins
All done here

Regular go calls init automatically. gomacro does not call it.

> time ./gomacro /tmp/decls.notgo  /tmp/mutrec.go /tmp/runit.notgo 4 
// warning: redefined identifier: b
A wins
All done here
    0m00.11s real     0m00.05s user     0m00.02s system

Unfortunately, there can be more than one init function, so we can’t just call it from runit.notgo.

func init1() bool {
        fmt.Println("Greetz from init")
        return true
}
var did1 = init1()

The workaround here is to give every function a name, and call it to initialize a variable. gomacro will execute this code.

It’s a bit ugly, but it mostly works. The good news is that init functions in imported packages will be called, since they are compiled go code.

os.Args

Another challenge is os.Args. The example above shows the problem. B is supposed to win with an argument of 5, not A. But os.Args will be mutrec.go (or decls.go).

For a short example, this can be hacked around.

        x, _ := strconv.Atoi(os.Args[len(os.Args)-1])

> time ./gomacro /tmp/decls.notgo  /tmp/mutrec.go /tmp/runit.notgo 5 
Greetz from init
// warning: redefined identifier: b
B wins
All done here
    0m00.11s real     0m00.06s user     0m00.04s system

OK. Right answer at last. But...

> time ./gomacro /tmp/decls.notgo  /tmp/mutrec.go /tmp/runit.notgo /tmp/mr.xml
Greetz from init
// warning: redefined identifier: b
A wins
All done here
/tmp/mr.xml:1:1: expected statement, found '<' (and 6 more errors)
/tmp/mr.xml:8:2: expected statement, found '<'
/tmp/mr.xml:17:2: expected statement, found '<'
/tmp/mr.xml:19:2: expected statement, found '<' (and 1 more errors)
/tmp/mr.xml:23:3: expected statement, found '<'
/tmp/mr.xml:25:3: expected statement, found '<'
/tmp/mr.xml:26:3: expected statement, found '<'
/tmp/mr.xml:31:3: expected statement, found '<' (and 10 more errors)
/tmp/mr.xml:44:3: expected statement, found '<'

If a file doesn’t exist, gomacro silently ignores it. We’ve just been playing with numbers, so we didn’t notice. But give it a filename that does exist and it will interpret as go code. If your program uses command line arguments to find files, this can be a big problem.

I don’t have a good workaround. This requires editing the code to read arguments from a file known name. Or just fixing the arguments in the code. gomacro is fast enough you can easily edit the source to point at different test inputs, but that’s clumsy. What if you want to test three inputs in sequence?

Alternatively, a small patch to main.go allows -- as a separator.

diff --git a/main.go b/main.go
index 21b079e..0d7387a 100644
--- a/main.go
+++ b/main.go
@@ -24,7 +24,19 @@ import (
 
 func main() {
        args := os.Args[1:]
+       last := 0
 
+       for i, arg := range args {
+               if arg == "--" {
+                       last = i
+                       break
+               }
+       }
+       if last > 0 {
+               args[last] = os.Args[1]
+               os.Args = args[last:]
+               args = args[:last]
+       }
        cmd := cmd.New()
 
        err := cmd.Main(args)

template

gomacro includes support for the holy grail of go features, generics, using a new keyword template. This conflicts with any existing use, such as an import of html/template. You have to give the import a new name.

import tmpl "html/template"

And then use that throughout. It’s not bad, but your code won’t look like normal go code.

Related note, glomming all your code together will produce duplicate imports, which generates warnings but isn’t otherwise harmful. Unless there are conflicts. For example, crypto/rand and math/rand in separate files will end up in the same translation unit here. gomacro complains about this as well, but it can be worked around by giving them distinct names.

conclusion

My goals are a little different from gomacro’s goals, so it’s not surprising there’s some friction. I wouldn’t necessarily consider the caveats above as actual failures. gomacro never promised to be a dropin execution engine. Even so, with a little effort to adapt existing code, it works quite well.