Of course it’s possible that someone comes in uninvited with the help of someone who has a key, but I’m sure that a stranger wandering around, looking at people’s screens would attract some attention.
Actually, it usually won’t. An interesting phenomenon has been demonstrated many times in supposedly secure buildings: Once somebody gets in, everybody assumes they belong there, and they have the run of the place. Journalists have used this to get stories in hospitals, airports, construction sites. There was even a case in the Australian military not that long ago of a civilian sitting in on a classified meeting after walking through a door that hadn’t closed fully.