Accessing struct fields inside a map value (without copying)

Assuming the following

type User struct {
    name string
}

users := make(map[int]User)

users[5] = User{"Steve"}

Why isn't it possible to access the struct instance now stored in the map?

users[5].name = "Mark"

Can anyone shed some light into how to access the map-stored struct, or the logic behind why it's not possible?

Notes

I know that you can achieve this by making a copy of the struct, changing the copy, and copying back into the map -- but that's a costly copy operation.

I also know this can be done by storing struct pointers in my map, but I don't want to do that either.