So im not sure if this is a bug or not but i was trying to change my wardrobe outfit and every time i didn't like something i would just hit remove. But instead of it removing it from the picture it removed it from my inventory completely which is what i did not want.

Ah, I can see how this would be confusing.

Items that you hit "remove" on should've been sent back to your inventory. You can use them there to move them back into the wardrobe.

To simply take an item off your avatar, just click the item's box again. Anywhere in the box aside from the remove button. (Note that outfit items cannot be removed, only changed to another outfit.)