The playboy of all playboys, Hugh Hefner, and “Girl Next Door” Holly Madison are splitting up. The Playboy mogul said he is "down in the dumps" about the split.

Madison, 28, stars in the reality show along Kendra Wilkinson and Bridget Marquardt. The three are Hefner’s girlfriends who live with him in the Playboy Mansion. "The Girls Next Door" was launched on E! in 2001 and is in its fifth season.

"If Holly says it's over, I guess it's over," Hefner told E!Online. The 82-year-old mogul added that a few days ago, Madison was still in the Playboy Mansion and they were “still sharing the same bed."

Although he was surprised by the fact that Madison talked about their split with the media, Hefner said he was expecting a break up. The main reason why Madison wants to split is that he told her that they will never get married or have children.

However, despite the fact that Hefner still has two girlfriends left (Kendra and Bridget) he said that Madison “is the love of my life, and I expected to spend the rest of my life with her." Madison announced that she and Hefner were splitting up in a video posted on TMZ.com Tuesday.

It al started eight months ago when Hefner learned that he could no longer become the father of a child due to his low sperm count.

"She'd like to be married and have children, but it's not in the cards here for me," Hefner said of Madison.

He may be broken hearted, but Hefner surely doesn’t show it. The playboy was seen last week with 19-year-old twins in tow. The Playmate twins, Karissa and Kristina Shannon, are also living in the Playboy Mansion but aren’t his girlfriends.