将十六进制值的字符串转换为[] byte表示形式

I have seen multiple questions to convert String to byte array, but what I want is to cast to byte[]. So that for instance, if I have a := "68656c6c6f20776f726c64", if I do []byte(a), it will convert each number into its hex value, but what I want is that it is directly interpreted as hex directly, so that I have []byte b = [68, 65, 6c, etc.]

Is there any other way different to iterating the string and every 2 characters appending them to the []byte?

A string is a byte array. Casting one to the other gets what you're seeing; the character a has an integer value (97, or 61 in hexadecimal), and you're getting the byte array that makes up the string of characters. If you want the string "ff" to become the integer 255 (ff in hexadecimal), that's not casting, that's parsing (decoding) the string based on a particular logic (i.e. that the string is made up of ASCII representations of hexadecimal digits). To do that you want to use an appropriate decoder, i.e. the standard library's hex.DecodeString as mh-cbon suggested:

src := []byte("68656c6c6f20776f726c64")

dst := make([]byte, hex.DecodedLen(len(src)))
n, err := hex.Decode(dst, src)
if err != nil {
    log.Fatal(err)
}

fmt.Printf("%s
", dst[:n])