Your ability to control a pet is calculated from your Taming and Animal Lore skill using the following formula, as stated by Oaks of the UO Dev team:

The actual formula for combined skill is: Effective Taming=(Animal Taming * 4 Animal Lore) / 5
Rounding does not occur until division, so you only lose your decimal place at the very end.

For example:
Say you have 88.9 Taming and 88.9 Lore. These numbers are actually represented as integers internally,
so we have 889 and 889.

(889 * 4 / 5) (889 / 5)=711 177=888 (88.8)
(889 * 4 889) / 5=4445 / 5=889 (88.9)
If Pet Difficulty - Effective Taming > 5 then the tamer cannot control the pet.

The game actually uses two linear formulas, one has a coefficient of 14, the other has a coefficient of 6. Any value less than 20 is automatically 'rounded up' to 20.

If Effective Taming is less than or equal to the minimum taming requirement of the creature,
then Pet Control %=70 + 14 * (Effective Taming - Pet Difficulty).
If Pet Control % is less than 20, then Pet Control % becomes 20.

If Effective Taming is greater than the minimum taming requirement of the creature,
then Pet Control %=70 + 6 * (Effective Taming - Pet Difficulty).
If Pet Control % is greater than 99 then Pet Control % becomes 99.

Side note: the expression "Effective Taming" in the above text is very misleading. It should actually be something like "Effective Control Factor", since it is the combination of Taming and Animal Lore skill used to determine your control ability. I just left it unchanged since I wanted to pass on the actual information as given by Oaks.

Credit for the above information goes to Nadya Modavia of Catskills, who was the first person to figure out the correct control formula from the example given by Oaks on the OSI Tamers & Rangers forum. Oaks personally confirmed the accuracy of Nadya's calculation.