I’m Anything But Average, Thank You

Is it just me or does anyone else consider the phrase “Average Joe” to be a pejorative comment on how we’re viewed by certain candidates?