I'd like to replace retool with go modules tools.go "tools as dependencies". However I'm struggling to understand how this works when my developers and CI env all use different operating systems.
I want to make sure each environment uses the exact same version of tools.
For a concrete example, my app requires the protoc compiler to generate go code via github.com/golang/protobuf/protoc-gen-go. I have 3 OS, all needing to execute protoc with the protoc-gen-go plugin/generator:
I currently use retool to make sure ALL environments are locked in on the same version of tools (protoc-gen-go in this ex):
retool do build/bin/protoc -Ibuild/protoc/include -I. rpc/platform/platform.proto --go_out=.
My new go modules / "tools as dependencies" setup
tools.go:
// +build tools
package tools
import (
_ "github.com/golang/protobuf/protoc-gen-go"
)
Set the path that go install will use:
export GOBIN=$PWD/bin
Install:
go install github.com/golang/protobuf/protoc-gen-go
If Ryan runs the go install .. a bin/protoc-gen-go MacOS executable is created.
Questions:
protoc-gen-go tool version (or git hash) NOT listed in go.mod?protoc know to use the protoc-gen-go executable generator in my ./bin dir?I was able to accomplish vendored tools build for protoc (and plugins like Twirp) following the Go Modules tools guidelines, plus a little Makefile-Fu for the protoc binary.
A complete working example can be found in the Aspiration Labs pyggpot repo. Following are the essential details. Worth noting: getting the import path right for some of the tools was very fiddly, but ultimately, successful.
For protoc itself, I vendor the binary release in the Makefile and set it up into a tools/bin dir:
TOOLS_DIR := ./tools
TOOLS_BIN := $(TOOLS_DIR)/bin
# protoc
PROTOC_VERSION := 3.7.1
PROTOC_PLATFORM := osx-x86_64
PROTOC_RELEASES_PATH := https://github.com/protocolbuffers/protobuf/releases/download
PROTOC_ZIP := protoc-$(PROTOC_VERSION)-$(PROTOC_PLATFORM).zip
PROTOC_DOWNLOAD := $(PROTOC_RELEASES_PATH)/v$(PROTOC_VERSION)/$(PROTOC_ZIP)
PROTOC := $(TOOLS_BIN)/protoc
# protoc
$(PROTOC): $(TOOLS_DIR)/$(PROTOC_ZIP)
unzip -o -d "$(TOOLS_DIR)" $< && touch $@ # avoid Prerequisite is newer than target `tools/bin/protoc'.
$(TOOLS_DIR)/$(PROTOC_ZIP):
curl --location $(PROTOC_DOWNLOAD) --output $@
The PROTOC_PLATFORM string can be automated with something like OS detecting makefile. The version of that we use is at https://github.com/aspiration-labs/pyggpot/blob/master/build/makefiles/osvars.mk.
On to building the go tools. Create a tools.go something like
// +build tools
package tools
import (
// protocol buffer compiler plugins
_ "github.com/golang/protobuf/protoc-gen-go"
_ "github.com/twitchtv/twirp/protoc-gen-twirp"
_ "github.com/twitchtv/twirp/protoc-gen-twirp_python"
_ "github.com/thechriswalker/protoc-gen-twirp_js"
)
Note: the // +build tools tag will keep go build from over-building tools imports in your final build.
Finally, some make code to build your go tools:
# go installed tools.go
GO_TOOLS := github.com/golang/protobuf/protoc-gen-go \
github.com/twitchtv/twirp/protoc-gen-twirp \
github.com/twitchtv/twirp/protoc-gen-twirp_python \
github.com/thechriswalker/protoc-gen-twirp_js \
# tools
GO_TOOLS_BIN := $(addprefix $(TOOLS_BIN), $(notdir $(GO_TOOLS)))
GO_TOOLS_VENDOR := $(addprefix vendor/, $(GO_TOOLS))
setup_tools: $(GO_TOOLS_BIN)
$(GO_TOOLS_BIN): $(GO_TOOLS_VENDOR)
GOBIN="$(PWD)/$(TOOLS_BIN)" go install -mod=vendor $(GO_TOOLS)
And finally, a make setup target to run go mod vendor and process the targets above.
setup: setup_vendor $(TOOLS_DIR) $(PROTOC) setup_tools
# vendor
setup_vendor:
go mod vendor
$(TOOLS_DIR):
mkdir -v -p $@
Go modules work with the imports of your .go files. If they find an import, they will automatically download the latest version that satisfies your requirements. You have to read https://github.com/golang/go/wiki/Modules and understand how modules work since Go 1.11 and later.
At this point, why is protoc-gen-go tool version (or git hash) NOT listed in go.mod?
This is because protoc-gen-go is just an external tool as far Go modules are concerned. You don't import golang/protobuf/tree/master/protoc-gen-go but the code it generates.
When Joe clones the app repo, how does he get and compile the same version of protoc-gen-go that Ryan used?
Use:
GIT_TAG="v1.2.0" # change as needed
go get -d -u github.com/golang/protobuf/protoc-gen-go
git -C "$(go env GOPATH)"/src/github.com/golang/protobuf checkout $GIT_TAG
go install github.com/golang/protobuf/protoc-gen-go
to install a specific version on each machine of the users. Probably write a build script that automates the process.
How does protoc know to use the protoc-gen-go executable generator in my ./bin dir?
From the github docs: The compiler plugin, protoc-gen-go, will be installed in $GOPATH/bin unless $GOBIN is set. It must be in your $PATH for the protocol compiler, protoc, to find it.