Private contacts
For contacts, you will see this padlock icon next to name in your contact list.
Private work items
For work items, you will see this padlock icon next to name of the item in the header.
Colleagues without access will not be able to click on those contacts or work items
Further reading:
Overview of Privacy
Video Overviews of Privacy