Tea Bags, both used and unused, can be put in to your composter. These can help add good structure to your compost.
Depending on the brand, the bag itself may or may not completely break down, even after a few years. Avoid adding nylon-based bags but other natural fibre-based bags should be fine. As long as you get sufficient heat into your composter you will maximise your chance of it decomposing. Your best bet is to give it a try, and if you’re not happy with the result, maybe tear open the bags before putting in the composter.
If you are fussy about your compost, you might want to remove the tiny staples (if there are any) from your tea bags.
You should avoid pouring a cup of tea, or tea from the pot, straight into the composter, only because of the risk of introducing too much moisture.