Rival nations are countries that the inhabitants of your nation consider their direct competitors. They tend to be of similar size, and for smaller nations they are often neighbors. Many missions and events use the rival concept, see for example Claims on Our Rivals!, but more importantly, it has a big influence on AI behavior. The computer might suddenly break a long-standing alliance when it starts to see you as a rival.
A nation can have up to four rivals. The concept was introduced in In Nomine.