Delish reports that if you look all the way back to the 4th century, many Christians marked the end of the Christmas season on the Twelfth Night (or 12 nights after Christmas)-- this night is also known as the Eve of the Epiphany. The Epiphany marks the Three Kings (or Wise Men) visiting baby Jesus, and is either celebrated on January 5th or January 6th (depending on if you count Christmas as day one or not.) Tradition dictates that this night (whichever date you believe it is) is the best night to take down your Christmas decorations, and says waiting too long after this date will bring bad luck.
(Yahoo)