I have three kids. Sometimes, I like to sit down and have a relaxing cup of coffee. Sometimes my kids make sure that does not happen. :) I have been in coffee shops and such, and if my little one happens to get fussy, well...that's his way of telling me that's all he can take. As infants, toddlers, and beyond, fussing or acting up, although it may not seem loud to you (who is used to it), is your child's way of telling you that they don't want to be a part of your fun.
I know you think the store clerk was being rude, but I bet your son would like to thank him if he could for getting him out of Starbucks when he clearly didn't want to be there.
I suggest you give the coffee date a shot at another time. Eventually your child may grow to like it, but right now it sounds like he's just not ready.