I'm trying to debug the following build error in our CI where "A depends on B which can't build because it depends on C." I'm building my data service which doesn't directly depend on kafkaAvailMonitor.go which makes this error hard to trace. In other words:
data (what I'm building) depends on (?) which depends on kafkaAvailMonitor.go
It may seem trivial to fix for a developer they just do "go get whatever" but I can't do that as part of the release process - I have to find the person that added the dependency and ask them to fix it.
I'm aware that there are tools to visualize the dependency tree and other more sophisticated build systems, but this seems like a pretty basic issue: is there any way I can view the full trace to see what's causing the build issue? In response to one of the comments - every programming languages I've used to date e.g. Java/Python all provide this information out of the box and it is very helpful for debugging code that I didn't write especially since I can't just look in a single place like a POM or requirements.txt to see who added the dependency.
Also in response to a comment - if the following isn't a stack trace what is it? An list of errors logged to stdout by a binary? Aren't those errors first put on the stack by the go build binary that's being executed? Seems like a silly distinction to me to say "that's not a stack trace" especially when it has nothing to do with my question, but I removed the word "stack" from the question if that makes people happy.
go build -a -v
../../../msgq/kafkaAvailMonitor.go:8:2: cannot find package
"github.com/Shopify/sarama/tz/breaker" in any of:
/usr/lib/go-1.6/src/github.com/Shopify/sarama/tz/breaker (from $GOROOT)
/home/jenkins/go/src/github.com/Shopify/sarama/tz/breaker (from $GOPATH)
/home/jenkins/vendor-library/src/github.com/Shopify/sarama/tz/breaker
/home/jenkins/go/src/github.com/Shopify/sarama/tz/breaker
/home/jenkins/vendor-library/src/github.com/Shopify/sarama/tz/breaker
if the following isn't a stack trace what is it?
It is the list of path where Go is looking for your missing package.
I have no idea who is importing kafkaAvailMonitor.go
It is not "imported", just part of your sources and compiled.
Except it cannot compile, because it needs github.com/Shopify/sarama/tz/breaker, which is not in GOROOT or GOPATH.
Still, check what go list would return on your direct package, to see if kafkaAvailMonitor is mentioned.
go listcan show both the packages that your package directly depends, or its complete set of transitive dependencies.
% go list -f '{{ .Imports }}' github.com/davecheney/profile
[io/ioutil log os os/signal path/filepath runtime runtime/pprof]
% go list -f '{{ .Deps }}' github.com/davecheney/profile
[bufio bytes errors fmt io io/ioutil log math os os/signal path/filepath reflect run
You can then script go list in order to list all dependencies.
See this bash script for instance, by Noel Cower (nilium)
#!/usr/bin/env bash
# Usage: lsdep [PACKAGE...]
#
# Example (list github.com/foo/bar and package dir deps [the . argument])
# $ lsdep github.com/foo/bar .
#
# By default, this will list dependencies (imports), test imports, and test
# dependencies (imports made by test imports). You can recurse further by
# setting TESTIMPORTS to an integer greater than one, or to skip test
# dependencies, set TESTIMPORTS to 0 or a negative integer.
: "${TESTIMPORTS:=1}"
lsdep_impl__ () {
local txtestimps='{{range $v := .TestImports}}{{print . "
"}}{{end}}'
local txdeps='{{range $v := .Deps}}{{print . "
"}}{{end}}'
{
go list -f "${txtestimps}${txdeps}" "$@"
if [[ -n "${TESTIMPORTS}" ]] && [[ "${TESTIMPORTS:-1}" -gt 0 ]]
then
go list -f "${txtestimps}" "$@" |
sort | uniq |
comm -23 - <(go list std | sort) |
TESTIMPORTS=$((TESTIMPORTS - 1)) xargs bash -c 'lsdep_impl__ "$@"' "$0"
fi
} |
sort | uniq |
comm -23 - <(go list std | sort)
}
export -f lsdep_impl__
lsdep_impl__ "$@"
The above answer still doesn't show me a dependency tree so I've taken the time to write a script to do what I need - hopefully that helps other people.
The issue with the above solution (the others proposed like go list) is that it only tells me the top level. They don't "traverse the tree." This is the output I get - which doesn't help any more than what go build gives me.
.../npd/auth/
.../mon/mlog
.../auth/service
This is what I'm trying to get - I know that auth is broken (top) and that breaker is broken (bottom) from go build but I have no idea what's in between - my script below gives me this output.
.../npd/auth/
.../npd/auth/service
.../npd/auth/resource
.../npd/auth/storage
.../npd/middleware
.../npd/metrics/persist
.../npd/kafka
.../vendor-library/src/github.com/Shopify/sarama
.../vendor-library/src/github.com/Shopify/sarama/vz/breaker
My script
import subprocess
import os
folder_locations=['.../go/src','.../vendor-library/src']
def getImports(_cwd):
#When the commands were combined they overflowed the bugger and I couldn't find a workaround
cmd1 = ["go", "list", "-f", " {{.ImportPath}}","./..."]
cmd2 = ["go", "list", "-f", " {{.Imports}}","./..."]
process = subprocess.Popen(' '.join(cmd1), cwd=_cwd,shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
out1, err = process.communicate()
process = subprocess.Popen(' '.join(cmd2), cwd=_cwd,shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
out2, err = process.communicate()
out2clean=str(out2).replace("b'",'').replace('[','').replace(']','').replace("'",'')
return str(out1).split('\
'),out2clean.split('\
')
def getFullPath(rel_path):
for i in folder_locations:
if os.path.exists(i+'/'+rel_path):
return i+'/'+rel_path
return None
def getNextImports(start,depth):
depth=depth+1
indent = '\t'*(depth+1)
for i,val in enumerate(start.keys()):
if depth==1:
print (val)
out1,out2=getImports(val)
noDeps=True
for j in out2[i].split(' '):
noDeps=False
_cwd2=getFullPath(j)
new_tree = {_cwd2:[]}
not_exists = (not _cwd2 in alltmp)
if not_exists:
print(indent+_cwd2)
start[val].append(new_tree)
getNextImports(new_tree,depth)
alltmp.append(_cwd2)
if noDeps:
print(indent+'No deps')
_cwd = '/Users/.../npd/auth'
alltmp=[]
start_root={_cwd:[]}
getNextImports(start_root,0)