I get the meanings behind what the colors mean. But what about the number?

Im assuming youre talking about how many stars there are, which shows the difficulty. The more stars there are, the harder it is to defeat. Enemies up to Umbra Forest range from 1-3 stars except for certain enemies, which are 4-5 stars. Starting from Astryls Vale, the enemies are most often 2 stars except for certain ones. Bosses are 4 or 5 stars.

Number of stars is "difficulty on field", but more specifically it tells you how much base mana a pet has.

For common species:
1 = 60
2 = 80
3 = 100
4 = 120
5 = 120 (boss, not catchable in any case)

For fabled, it's +20 on what regulars have, so they'd cap at 3 = 120

Example of a 1 star pet
2 star
3 star
I don't own any 4 star pets to use as an example (I've only ever seen a single one)