I wouldn't classify myself as a "fan" of either, and I don't have anything against these two either, but for some reason I always took Barrett more seriously than Swagger. (i.e. Swagger used to be a goofball before his new gimmick and Barrett was always a serious face type guy.)
With Swagger I think that it's almost certainly because of physical cues. The goofy smile, the chest beating and the lisp pretty much screamed don't take me seriously.
Add in a silly gimmick like the All American American and the fact that he's a decent wrestler but not a stand out one and he was dead in the water.
That's why I don't see his current gimmick having much of a shelf life. Bad first impressions, especially in pro wrestling, are extremely difficult to rebound from.