Good question; never thought about it before. Though, I would think at some point, the "oak" would be milder the longer you use them. If they are kept clean/sanitized, I do not think there would be any harm.
As long as you are re-using them on the same style brew as I would think also that they would soak up from the previous batch and add back to the next batch; albeit very minimal.
Enjoy the little things, for one day you may look back and realize they were the big things.
Kegged: RIS, West Coast Blaster, Porter