It would make even more sense to store it as a List, since that actually guarentees order by contract. The map structure is only required to delete the listener, which can be easily accomplished by simply parsing the list.
If that proves to be too slow, then a secondary map can be used to store the location, but I doubt that would be necessary.