This has actually happened to me, but with regular carts, not just the special ones. There are a couple of baggers/cart collectors that are challenged, and though they mean well, they will sometimes stand behind me while I unload my groceries and ask if they can have my cart. I always tell them that I will put the cart away after I am done, that I will be a minute putting in my groceries and my kids. Sometimes they leave me alone, sometimes they still hover. I just go about my business. If they are still around when I am done, I pass the cart over to them.

You shouldn't have rushed. I would have told the girl "Fine, but I am going to be a minute", and then just taken your sweet time putting the groceries in at your lesiure. She can't make you hurry up. You allowed your self to feel rushed.
Had she continued to badger you for the cart, then you could have told her to go look for another one.