I have a Golang program working with Twirp and I want to create GraphQl server, but as far as I know, twirp build on top of the Rest API

It is definitely possible.

You Just have to make a wrapper over the Twirp based RPC API.

This is a similar case to wrapping a GraphQL API over a Rest API.

You should also read this article, where the wrapping of a graphql api over a rest one is shown.