I have used both (crossbows are legal for bowfishing for rough fish in Illinois by anyone with a fishing license). A compound bow designed for bowfishing works best (my favorite by far is the Oneida Osprey, a real gem of a bowfishing bow). Bowfishing bows typically have low letoff (less than 50%) and relatively long limb length, designed to be shot accurately with fingers vs. a release. Check out the bowfishing outfitters Sully's Bowfishing Stuff, Bowfishing Extreme, and Backwater Bowfishing for supplies.
The best all-around reel is the AMS Retriever. For smaller fish the standard Retriever with 130 to 200 lb Fast Flite line is best. For large fish that are powerful swimmers the slotted Retriever with a float and 400 lb test line is best.
There are a variety of tips available, which one you use depends on the type of fish you are going after and the shooting conditions. This is best learned by experience.
Crossbows have a short arrow and more powerful draw weight causing more undesirable passthrus (where the arrow blasts completely through the fish). In addition, crossbows cannot be short draw snap shot. In most states it is illegal to shoot fish with a crossbow.
Bowfishing is a challenging sport. The best way to learn how to be successful at bowfishing is to join a local club and get some coaching from the experienced members.