Is there any difference between declining a challenge and letting it expire?

Answer 1

Not really. You don't lose xp this way or the other.

Answer 2

No, but declining a challenge is the polite way to manage a challenge you either can't do or don't want to.

If there is a language you don't know, you can go to your settings, "Manage Weapons" and remove it.