Contacts can be assigned to specific Users in one of two places:

1) When initially adding the new Contact in the pop up window

2) On the contact's personal profile page

Some users may be assigned contact's who are in tagged categories to which they do not have access. They will still be able to see these contacts on their assignment list without viewing the entire tagged category.