In general only writes cause wear but the card might be doing some in the background - most cards do internal housekeeping and will detect and remap weak sectors without you even knowing, so it's very hard to tell.
I think that unless you are doing a lot of writes to that card, and that include not only user apps but also things like swap files, there is no point in worrying about that. If you were to not use the card at all then flash cells (especially the modern, dense / multi-level ones) will slowly loose charge and glitch the data after a decade or two. Some modern SSDs can only last about 6 months when unpowered. Granted, that is not card damage since it can be just re-written but it is data loss.