Where did the gold go?

Thinking of the Nativity again, this question is troubling me. Presumably the gift of gold, offered to a king, would have consisted of a substantial amount. It would have been the equivalent of winning the lottery today. So where did it go? Joseph seems to have continued to work as normal as a carpenter, and there is no indication of Jesus' family being particularly wealthy. There is no mention of what they did with it.
It would make a good story, I think. Especially if told from the point of view of Joseph, because he is stangely unrepresented in the Bible. Not much more than a walk on part, really.
Strange.