The change of heart followed a visit to the man's home from South Australian Attorney-General Michael Atkinson today.

The RSL had called the Nazi Swastika sickening, saying the flag was a symbol of a regime that took the world to war in 1939 costing millions of lives, including those of 40,000 Australian servicemen and women.

"Australians will be rightly offended by this and most veterans are grossly offended by it," said RSL national president Bill Crews.

Mr Atkinson said there was little the government could do about the flag and while it was in bad taste, the issue was one of freedom of expression.

"Unless one can argue that it incites racial hatred, then I suspect it's a matter of free speech, free expression," he told ABC Radio.

But he decided to visit the man's home this morning and said he explained to him the suffering of people in Europe under the Nazi regime.

Mr Atkinson said the man told him he only put the flag up for a party a few weeks ago and had simply neglected to take it down.

He was also angry at the media for invading his property but accepted that the flag would be offensive to some people.

"He agreed that it would come down and it has," Mr Atkinson said.