Question about wire gauges: I assume the diameter of a given gauge (e.g.: 18 or 2/0) corresponds to the diameter of the copper filament itself, NOT including any of the insulation?
Correct. Make sure you're using 100% copper wire; some of the cheap wire out there (e.g.., most jumper cables) are copper-clad-aluminum (CCA) and have much higher resistance.
Figure out how much cable you're using, both ways (add up the length of the two wires). Decide how much of the rated pull you want the winch to deliver; ideally, you'll have a chart for the winch that shows the ampere draw for that pulling force. That, and the voltage drop you're willing to tolerate, with determine the maximum voltage drop in the cable, according to Ohm's Law: Resistance = (voltage drop)/(current). IMO 1 volt is a good voltage drop. Divide the resistance by the total length of cable gives you the maximum resistance per foot; look up the needed wire size on a table for copper wire.
Based on that. IMO 1/0 is a good size to use in many cases; you can always go bigger if you want to spend the money and want to put up with dealing with the larger size.