The villa, built in the 1930s, was used as a coastal getaway before the outbreak of the Korean war in 1950 by North Korean leader Kim Jong Un's grandfather and father who both spent holidays there with the families of Soviet commanders.
It's now in South Korea as a result of the war, a tourist attraction of Goseong County. Local officials proposed that Seoul invite a visiting North Korean Olympics delegation to pay a visit to the villa this month as a gesture of peace.
The delegation of athletes, officials, musicians and cheer-leaders are attending South Korea's first winter Games in Pyeongchang, about 90 km away from the villa, in what Seoul hopes will be a breakthrough in improving relations.
"We were ready for them, but my understanding is that they don't plan to come," said Lee Kang-hoon, a businessman who heads a local association to promote Goseong as a destination for tourists and investors.
"We had hoped that if they travelled to South Korea by road, there would have been an opportunity to drop by briefly and help create a mood of reconciliation between the two Koreas," Lee said.
After the war ended in 1953, redrawn frontiers left Goseong and the villa about 10 km inside the border.
The building's links to Kim Il Sung, communist founding father of the North Korean state and Kim Jong Un's grandfather, makes it a potent political landmark -- perhaps too potent.
Seoul and Pyongyang both claim sovereignty over the entire Korean peninsula, and the Kim family are afforded godlike reverence by the North Korean state, so a visit could be construed as granting a kind of political pilgrimage.
Allowing the visit would have been a "PR disaster" for South Korean President Moon Jae-in, said Christopher Green, senior advisor on Korean affairs for the International Crisis Group, a research and advocacy group for avoiding conflict.
"The Moon administration is already treading a fine line between engagement and what a large and vocal segment of the population regards as appeasement," said Green.
The villa was built in 1938 by a German architect who used it as a chapel and retreat for foreign missionaries. Under North Korean control, it became a holiday home for officials of the ruling Worker's Party and their Soviet comrades.
Soviets were stationed in North Korea in the years leading up the outbreak of war.
Now, the partially restored villa hosts an exhibition dedicated to hopes of a reunified Korea.
A photo in the exhibition shows the North Korean leader's father, Kim Jong Il, as a bare-footed six-year-old wearing shorts and sitting on the villa's steps next to a Soviet Army commander's son. Beside them, Kim Kyong Hui, the once-powerful aunt of current leader Kim Jong Un, stares at the camera.
Even if Seoul had allowed the invitation, it may have prompted an awkward response from the North Korean side.
"A place where Kim Il Sung lived is in principle an important commemorative site for North Korea," Green said.
"But the residence has not been cared for in anything like the reverential manner of Kim Il Sung's putative birthplace in Pyongyang, for example. It is kept in the manner of a provincial seaside tourist attraction".