Oliver left the film business in the late 1930's and retired to Thousand Palms in the California desert in 1940. There, he built Old Fort Oliver, published the Desert Rat Scrapbook and became a something of a hermit. He collected desert lore and established a well-founded reputation as a joker and an outback character. On a website dedicated to him, he has been described as a 'mirage salesman' and 'press agent to a ghost'.