When Anna first met Hans, she became infatuated with his good looks and charm. As the two became more acquainted with each other and their lonely pasts, they soon shared a romantic duet which ended with Hans proposing to Anna, which she gladly accepted despite having only known him for a day. While Anna was away, she proclaimed that Hans would be in charge during her and Elsa's absence. While it seemed during Anna's absence that he truly cared and loved her, when Anna returned to Arendelle to receive a true love's kiss from him in the hopes of healing Elsa's curse inflicted on her, Hans revealed that he never loved her as well as how he only planned to marry her to become king of Arendelle and kill Elsa to ensure this. He left Anna to die as he went to kill Elsa, and any feelings Anna had for him disappeared. She managed to interfere with his plans, and saved Elsa by using her own body as a shield to block his killing blow and later expressing her renouncement of Hans before abruptly punching him off a ship in a rare-for-Disney but highly justified act of non-defensive violence. Truly, the broken relationship with Anna can be cited as an error in Hans' part, effectively creating a force that would prevent his usurpation into king from seeing the light of day and plunge him into humiliation and defeat.
It is evident that due to her years of loneliness and desperation for love, Anna's "love" for Hans was nothing more than a small crush based on his outward appearance and her belief that she may not have another chance to find someone else, while her feelings for Kristoff were more genuine and based on a mutual attraction, even though she had known him no longer. Even Hans has stated how she was so desperate for love that she was willing to marry him almost at first sight.
It is also apparent that there's no longer any affection between the two sides, as Anna currently hates Hans for his treacherous actions, and Hans appears to have never had any strong feelings of any kind toward Anna at all even with his current designation as an enemy of her family and kingdom, having considered her nothing more than a convenient means to an end during their brief prospective relationship, for which any affection he may have had at any point was eclipsed by his ruthless political ambitions. However, at the very least Hans did not intend to outright kill Anna had his initial plans for using her gone smoothly, provided she remained oblivious to the circumstances planned for the eventual death of Elsa, regarding whom the same cannot be said. Even in Once Upon a Time, where the show's version of Hans is portrayed as considerably more irredeemably cruel than his canonical self can be confirmed to be and does try to outright murder Anna during events transpiring after those equivalent to the Frozen film, he has a line indicating that her situation would not have come to this had she never left to help Elsa during her self-imposed exile, setting off everything that proceeded to happen later.
Inadvertently, Elsa and her nature as a winter elemental have been a major obstacle to Hans' pursuit of becoming Arendelle's sole monarch. Originally, it was Hans' plan to marry the soon-to-be queen regnant of Arendelle so he could rule the kingdom with her by his side (or more likely Elsa as queen, but with him as the prince consort, it would give him considerable political power), but gave up on the idea when he got word of her dislike of being close to people and her stand-offish behavior. He did not learn of her abilities until Elsa and Anna got into an argument about being married to Hans after only knowing him for a day, which causes her powers to manifest, much to everyone's shock. From then on, inadvertently thanks to Elsa and her winter magic, he had to constantly change his plans to become the king of Arendelle, from making Elsa seem like a monster in the eyes of the public to killing her to end her unintentional winter curse and become a hero to the people and taking advantage of the winter curse to claim the throne without marrying Anna. This also required getting into Elsa's good graces by stopping her from killing the Duke of Weselton's bodyguards after they attempted to murder her in obvious cold blood. He briefly succeeded in ingratiating himself to her, taking advantage of her emotional instability by acting as her apparent confidant while she was in the Arendelle dungeon, before dropping this facade when she escaped, finally ending with him promising to kill her to end the curse of her powers (but never actually revealing his machinations to her the way he'd done with Anna before her).
According to Hans, three of his older siblings pretended he didn't exist for two years. Hans brushed off Anna's shock at such emotional abuse by saying "It's what brothers do," which implies his worldview and understanding of how family relationships work may have been severely warped as a result of his upbringing.
Though there was some initial speculation over whether or not Hans' description of his home situation was reliable, it has been confirmed twice by official material. The book A Frozen Heart, though not fully canon, portrays his childhood as truly horrible and abusive, and his father as a ruthless man who admires power and despises weakness, urging the princes to pick mercilessly on the weak (Hans). A canon confirmation has also been received from Jennifer Lee, who stated in a Reddit AMA interview that she sees Hans as "a tragic figure because he's a consequence of being raised without love". This removes all speculation about how his family has treated him, growing up.
At the end of the film, Hans was sent back to his country to receive punishment from his brothers for his actions in Arendelle, indicating that they would hold him responsible for his crimes if nothing else. In the short film Frozen Fever he is seen shoveling horse manure at the stables, indicating his punishment is forced labor.
Throughout the film, Hans and the Duke of Weselton were often at odds. They first interact after Elsa's icy curse has been put upon Arendelle, causing the Duke to panic and declare Elsa a monster, with Hans trying to calm him down, pointing out he only slipped on ice. Later on, after Hans acts as regent of Arendelle under Anna's orders, the Duke openly disapproves of the Prince using Arendelle's resources as emergency supplies for the people, and tries to spread treasonous rumors about both the Queen and the Princess "conspiring to kill us all", causing Hans to snap and threaten to charge the diminutive dignitary with treason unless he stops questioning Princess Anna. Hans is seen as his angriest when asserting authority over the Duke, and his expressions show he's very annoyed by him. He is unaware the Duke has ordered his men to assassinate Queen Elsa on sight as he accepts them as volunteers for the rescue expedition.
However, despite their disagreements, the Duke shows kindness and even concern towards Hans when he staggers into the room, lying that Princess Anna has already died from the curse. Despite previously trying to turn the populace against the sisters, he expresses shock and sadness at the news of Anna's death and is also the first of the dignitaries to indirectly declare his support for Hans, stating once again that Queen Elsa is a monster and they are all in grave danger, prompting the Italian dignitary to state that Hans is now in charge.
The Dignitaries of the other kingdoms (possibly also representing the nobility of Arendelle since none is ever seen) support and rely on Hans and accept his leadership without question. It's notable that when Hans, worried, is planning to launch a second rescue mission in order to find Anna, still "missing" in the wilderness, the dignitaries try to convince Hans to abandon Anna to survive alone, and tell him that should anything happen to the Princess, Hans will rule Arendelle ("should anything happen to the Princess, you are all Arendelle has left"). Hans' expression at this is shocked, suggesting that it may be the first time he realized he could rule Arendelle on his own, even without marrying either of the royal sisters.
Also when Hans falsely informs them Anna has died, the dignitaries declare him the new ruler of Arendelle, stating "Arendelle looks to you". The Arendelle military, as well, seems to fully accept Hans' leadership and obeys his order of sentencing Queen Elsa to death without any opposition, going with him to execute her at the dungeon.
However, despite their apparent strong, wholehearted support of Hans, the dignitaries and the military are quick to change loyalties the moment they see Hans get blown away on the ice. In the end, they are shown to be on good terms with Queen Elsa, who is possibly unaware of how eager they all were to support her execution earlier.