Even if you say that slavery was only 50% of the argument for secession itself, the underlying attitude with regard to blacks held by the majority of those in favor of secession still remains. While racism obviously isn't limited to the South, it remained a large part of Southern society WAY past the point that the issue of slavery was settled. And when you look back at the fight for civil rights, the South didn't end legal racist practices voluntarily. Wide-spread, LEGAL Southern racist practices are also hardly ancient history as they were still in place only a generation ago. So, it's kind of hard for a lot of people to not look at a symbol, the Confederate flag, of a slave owning region that STILL had the same attitudes with respect to blacks over a century after slavery ended. It's even more difficult when you take into account that the KKK (which was founded by ex Confederate veterans) has continued to use the same symbol that has ALWAYS been intertwined with a widespread racist mind set since shortly after the Civil War. Now, does everyone that likes to express Southern pride through the Confederate flag have such a mindset, of course not. However, given that from DAY ONE, the Flag of the Confederacy was taken up by a region where racism wasn't the fringe, but the majority AND has also been a symbol (at least one at least) for damn near every group that openly touts hate and racism against minorities, I don't honestly see how anyone can be surprised at how it's looked at by many.
For the majority of this i've been playing devils advocate, but i'm having difficulty fighting this particular line of logic.
Waving the confederate flag doesn't make you a racist, but you are wearing a racists uniform [to paraphrase a comedian, on a different topic]
--- of course I have to wonder if atleast some of the prevelant racist attitudes of the south are due to the fact that slavery wasn't allowed to end naturally for them.