Golang-嵌套地图不支持内部级别的索引，而外部级别很好

Almost a go-newbie, and for the first time I have to made a question about it, about a problem with interfaces, types and maps.

So, my starting point is a database query that retrieves an object like this one:

+-------------+---------------------+----------+------------+
| category_id | category_name       | group_id | group_name |
+-------------+---------------------+----------+------------+
|           1 | Category1           |        1 | Group1     |
|           1 | Category1           |        2 | Group2     |
|           1 | Category1           |        3 | Group3     |
|           1 | Category2           |        4 | Group4     |
|           2 | Category2           |        5 | Group5     |
+-------------+---------------------+----------+------------+

and my final goal is having a json object with the groups under the same category under that category, like this one:

{
  "id": 1,
  "name": "category1",
  "groups": [
    {
      "id": 1,
      "name": "Group1"
    },
    {
      "id": 2,
      "name": "Group2"
    },
    {
      "id": 3,
      "name": "Group3"
    }
  ]
},
{
  "id": 2,
  "name": "Category2",
  "groups": [
    {
      "id": 4,
      "name": "Group4"
    },
    {
      "id": 5,
      "name": "Group5"
    }
  ]
}

I don't want to use multiple queries, cause this is just a part of the final query, I used just 2 field to be more clear with this example. In my current situation I just have 5 levels...

So I created a struct that should be used on all levels of my object, that implements an interface:

type NestedMapObjs interface {
  getOrderedKeys() []int
}

and the type that implements this interface, that should be a map of int in order to append elements to the correct map:

type BuilderMapObjs map[int]NestedMapObj

when NestedMapObject is:

type NestedMapObj struct {
    ID        int
    Name      *string
    NestedObj NestedMapObjs
}

so, on my method that builds the map object that I want, I have no problem to add the first level of my object (Category) but, I found some problems on the second level, the group one. In particular, this is my function that adds a new row:

func (m BuilderMapObjs) addNewRow(scanned audienceBuilderScannedObject) error {
  if _, ok := m[scanned.CategoryID]; !ok {

    var innerObjs BuilderMapObjs
    innerObjs = make(BuilderMapObjs, 0)

    m[scanned.CategoryID] = NestedMapObj{
      ID:        scanned.CategoryID,
      Name:      &scanned.CategoryName,
      NestedObj: innerObjs,
    }
  }

  if _, ok := m[scanned.CategoryID].NestedObj[scanned.GroupID]; !ok {
    m[scanned.CategoryID].NestedObj[scanned.GroupID] = NestedMapObj{
      ID:   scanned.GroupID,
      Name: &scanned.GroupName,
    }
  }

  return nil
}

(I know, I can refactor and make this code more readable, but this is not the point now...)

The problem is when I try to get the inner object by its key, and when I try to add it. This line:

m[scanned.CategoryID].NestedObj[scanned.GroupID]

produce this error: invalid operation: m[scanned.CategoryID].NestedObj[scanned.GroupID] (type NestedMapObjs does not support indexing)

Actually, I just found that with a better implementation, implementing two more methods in the interface (getIndex and addToIndex) I fixed the problem, but I'd like to understand this problem.

Why I have an error on the inner object and not on the outer one?

Thanks for reading until this point!