Well you brought up baseball, cricket, and football (which I assume you mean American Football?). Looking at those sports they are hardly universal sports though right? How many countries play baseball (less than 15?)? How many plays cricket (definitely less than 15). How many plays American Football? (2 that I know of). It's not only participation by numbers that should be the main consideration, but participation by 'countries' should also be considered. I mean India has like 1/3 of the world's population or something like that, and just because they're the only country playing cricket (or where cricket is popular) doesn't automatically make it an Olympics sport.
Same thing with rugby, they took out the 15 man game way back when, then brought it back in the form of the Sevens rugby because it has become more popular amongst many countries (the US team has actually gotten pretty good at it now for instance, as well as Kenya and Japan, etc).
Skateboarding (to my knowledge) is played by people of all countries all around the world, from Africa to America to Asia. Kids of all ages and race pick up a skateboard every single day. It is a very Universal thing. Think about this - I believe more kids pick up a skateboard and skate on it than they do a shot-put or a javelin. And yet those are Olympic sports.
There is participation and Universality. To me there is no reason it should not be added to the Olympics.