I'm working on a web application written in Golang that needs to call a Python program/module to do some heavy work. Since that is very memory/CPU intensive, it may be on a separate machine. Since Golang and Python can't talk directly, there are 3 ways to achieve this:

1. Just execute the python program as an OS process from Go (if on same machine) (or RPC?)
2. Wrap Python process in a service and expose it for it to be called from Go (may be a simple CRUD like service - A Bottle/flask restful service)
3. Have a simple pub-sub system in place to achieve this (Redis or some MQ system) - Adding Redis based caching is on the radar so maybe a good reason to go this way. Not sure.

The main thing is that the python process that takes really long to finish must "inform" the web application that it has finished. The data could either be in a file/DB or 'returned' by the process.

What could be the simplest way to achieve this in a pub/sub like environment?

UPDATE REST seems like one way but would incur the cost of implementing server side push which may or may not be easily doable with existing micro web frameworks. The pub/sub would add an additional layer of complexity for maintainability and a learning curve nevertheless. I'm not sure if an RPC like invocation could be achieved across machines. What would be a good choice in this regard?

Use ZeroMQ

Python has nice pyzmq

GO bindngs exists too.

ZeroMQ code is typically short and effective.

You can even use local socket on Unix.

There are even other messaging patterns apart from pub/sub, you can use req/resp or push/pull.

Not knowing much about your applications, I can offer sample of integrating lock manager using zeromq in Python

Using ZeroRPC

There is experimental Golang lib being able communicating with Python and Node.js code via ZeroMQ as shown (for Python and Node.js) zerorpc.dotclould.com

Writing functions in Python being callable over ZeroRPC is extremely simple - in fact you do not have to write a line using zeromq (there is command line tool, which allows integrating functions with properly shaped signatures).

For your specific pattern, simply spawning the process from Go and reading the stdout is the most efficient, there's no point adding an over head.

It highly highly depends on what your python script does, if it's one specific task then simply spawning the process and checking the exit code is more than enough, if you have to keep the script in the background at all time and communicate with it then Redis or ZeroMQ are good, and very mature on both Go and Python.

If it's on a different server then ZeroMQ/RPC or just a plain http server in python should be fine, the overhead should be minimal.