Harry Potter spent the first ten years of his life with his late mother's sister, Aunt Petunia. As seen in the first Harry Potter movie, Petunia lived with her husband, Vernon Dursley, and their son, Dudley at Four, Privet Drive. Harry experienced mistreatment at the hands of the Dursleys, and he was an unwelcome addition to their family. He was given the cupboard under the stairs for a bedroom, and constantly grounded for petty reasons. Before Hagrid's visit, Harry was kept in the dark about his magical legacy. His aunt and uncle were unkind and Dudley pestered him. Vernon's sister, Marge had suggested Harry should have been dropped off at an orphanage, but he maintained he took his nephew in purely out of the goodness of his heart.
As depicted in the Harry Potter movies, Petunia and Vernon held Harry in contempt and yet allowed him houseroom. While the Dudleys were Muggles, Harry was a wizard living among them. He didn't know the circumstances of his parents' deaths; he was told they died in a car crash and kept on believing so until his eleventh birthday. Petunia and Vernon aimed to squash magic out of Harry, keeping him unaware of Hogwarts and the fact that his parents were martyred in the First Wizarding War. Eventually, there came a time when Harry Potter learned who he was, about Lily's sacrifice, and why he had to return "home" once a year to stay with his aunt when many Wizarding families would have gladly raised him.
Lily Potter's Sacrificial Protection
At the end of Harry Potter and the Sorcerer's Stone, Albus Dumbledore visits Harry in the Hospital Wing and lets him know Professor Quirinus Quirrell doesn't have the Stone and that it has been destroyed. The wizard of eleven proceeds to ask his professor certain things, starting with, "Voldemort said that he only killed my mother because she tried to stop him from killing me. But why would he kill me in the first place?" While Dumbledore gatekeeps this, he answers the question about why Quirrell couldn't touch Harry:
Your mother died to save you. If there is one thing Voldemort cannot understand, it is love. He didn't realize that love as powerful as your mother's for you leaves its own mark. Not a scar, no visible sign… to have been loved so deeply, even though the person who loved us is gone, will give us some protection forever. It is in your very own skin. Quirinus Quirrell, full of hatred, greed, and ambition, sharing his soul with Voldemort, could not touch you for this reason. It was agony to touch a person marked by something so good.
As explained in Harry Potter, Lily had consciously, willingly, and out of deep and pure love sacrificed herself to save the life of her son. In doing so, she formed an ancient magical protection, i.e., the Sacrificial Protection which some regard as "the ultimate protection." It's precisely why Voldemort's Killing Curse backfired, and Harry became "The Boy Who Lived."
In the fifth Harry Potter book - The Order of the Phoenix, Dumbledore goes into detail about the effects of Sacrificial Protection. In chapter 37 of Harry Potter and the Order of the Phoenix "The Lost Prophecy," Dumbledore explains why he made certain decisions about Harry's life. He starts by explaining that fifteen years ago when he saw Harry's forehead scar, he guessed that it could be the sign of a connection forged between the infant and Lord Voldemort. Dumbledore also touches on why he asked Snape to teach Harry Occlumency when he could have done it himself. "I was sure that if he realized that our relationship was — or had ever been — closer than that of headmaster and pupil, he would seize his chance to use you as a means to spy on me," he explains. Dumbledore finally opens up to Harry about why he chose to leave him on his aunt's doorstep.
Dumbledore Condemned Harry To Petunia's And Bond of Blood Charm
In Harry Potter and the Order of Phoenix's "The Lost Prophecy," Dumbledore admits he knew leaving Harry at Petunia's was equivalent to condemning him to a miserable childhood. Dumbledore explains that after the attack at Godric's Hollow, he knew Voldemort wasn't gone forever and would return to kill Harry. This harsh truth was compounded by Dumbledore's realization that all his protective spells and charms would fail if Voldemort were to return to full power. "But I knew too where Voldemort was weak. And so I made my decision. You would be protected by an ancient magic of which he knows, which he despises, and which he has always, therefore underestimated — to his cost. I am speaking, of course, of the fact that your mother died to save you." he says.
This raises the question of Lily Potter's sacrifice and how it gave Harry a lingering protection. Furthermore, Dumbledore explains that this protection flows in Harry's veins, and it was the reason why he chose to deliver the infant to Lily's only remaining relative in place of one of the many Wizarding families. By agreeing to take Harry in, Petunia sealed the charm that Dumbledore placed upon the infant. "Your mother’s sacrifice made the bond of blood the strongest shield I could give you," Dumbledore reveals in Harry Potter and the Order of the Phoenix.
Dumbledore explains the Dursley household was a miserable place, but a haven for Harry, nonetheless. Voldemort couldn't so much as touch him at Petunia's because Lily's blood dwelled there. "He [Voldemort] shed her blood, but it lives on in you and her sister," the most powerful wizard reveals. Petunia's blood become Harry's refuge from that day until he reached the age of majority in the Wizarding World, i.e., seventeen. It's precisely why Harry needed to return to Four, Privet Drive at least once a year. "...but as long as you still call it home, there he cannot hurt you," Dumbledore explains in the fifth Harry Potter book. In other words, Albus Dumbledore placed the charm on Harry in 1981 after Lily refused to step aside and sacrificed herself, thereby paving the way for Sacrificial Protection and allowing the Bond of Blood charm to be cast. By allowing Harry houseroom, Lily's blood relative, Petunia sealed the charm. Simply put, Harry had to return to Aunt Petunia's house because his life depended on it.