So I have the quest that's supposed to teach me how to capture pets. It says that, while I'm exploring, I should use the "Capture" ability to attempt to catch what I'm capturing, but... I don't seem to have any sort of ability? Am I supposed to buy it somewhere or something?

Oh wait... it's an event! I just now got it. Whoops. Sorry!

Actually, there is a an item you need to buy to catch pets, they are called traps. Yes there are a few events in which you can't or don't catch pets, namely mini events(like the 4th of July event that ended on the 10th), battles with named pets in the dungeons, and occasionally during the wave type events...however other than that you need traps to catch pets.

Green traps can catch common(white star), and Uncommon(green star)
Blue traps can catch common, uncommon, and rare(blue star)
Purple traps can catch common, uncommon, rare, and epic(purple star)
Master traps can catch everything, and currently is the only way to catch Legendary(yellow/orange star), and Mythical(red star) pets.

Hope I helped you out some.

Where do you buy the traps

Yes, that did help out a ton, thank you! I figured out about the traps (but only the first ones), though... is there something special I'm supposed to do? I've bought like ten traps so far, got the other pet's health down as low as I could without killing it, and yet the traps have never worked. Since they're expensive (at least right now), that's kind of problematic, and I'm honestly debating just giving up since it doesn't seem worth it if they never work.

Frankly I found low level traps to be pretty worthless when it comes to catching pets. It's a whole lot cheaper to use the advanced search and buy pets with specific traits you like.

You buy them from the markets in the different cities/towns. All three of the current story zones have a market.

There's always a fail chance for all traps except for the master trap which has a 100% catch rate. I do know that early on that yes the traps are a bit on the pricey side, my best suggestion in regards to that is to save up and stay as well stocked as you can on the blue traps, and ONLY catch(or try to) pets that you know you will like to have for a long span of time.