I think I made something similar to this in my SM64 recreation project.
What I did was give Mario's materials a next pass. That pass was set to nevermind the depth buffer, and draw the scene below it, but darker. To prevent Mario from being forever dark though, another pass used that also renders the screen texture. Because the sillouette isn't drawn to the screen buffer, it gets drawn over by this third pass. But whenever the third pass does not get rendered, the sillouette appears. Like when Mario's behind an object. I'm pretty sure this is just a hack though, it's all done in the transparent pipeline, but by playing with render priorities it plays well with other transparent objects, by having the sillouette draw before them.
Edit: I just realized that you meant just an outline and not a sillouette. Oopsie! :P
