Herring Houdini seats a Donner party of five. The Donner Party ate each other, out of starvation, while traveling to California in 1847. It goes along with Esmé saying, "I am so hungry, I could chew off your arm," earlier in the episode.
While leaving the powder room Violet looks over the small table by the bath tub and see a tea set but the sugar bowl is missing. That's a reference to the backstory of the series: someone (possibly Lemony Snicket and/or Beatrice) stole a sugar bowl from Esme and she seeks vengeance for that.
The portraits in the Squalors' dining rooms are of the show's executive producer/director, Barry Sonnenfeld. In the "informal" dining room, the portrait has him in cowboy attire. Sonnenfeld is a huge cowboy fanatic, wearing hats and boots often, and even has a saddle that he sits in while directing.
As the Baudelaires are walking up the candle lit staircase to the penthouse, Snicket narrates about the meaning of the phrase "in the dark". The entirety of this speech is taken, word for word, from the 13th book, The End.
Esmé Gigi Genevive Squalor is named after the short story "For Esmé - With Love and Squalor," by J. D. Salinger. Gigi may be a reference to Gigi by Sidonie-Gabrielle Colette.