Or you could just go the other direction and cache the known results ahead of time. You're only looking at 13 numbers in all, so it is not a big memory hog to just store those known values inside of the method and be done with it.
static uint Factorial(uint x)
{
if (x > 12)
throw new ArgumentException("Cannot calculate a factorial for numbers larger than 12");
return _factorials[x];
}
static readonly uint[] _factorials = new uint[13] { 1, 1, 2, 6, 24, 120, 720, 5040, 40320, 362880, 3628800, 39916800, 479001600 };
Edit 10/11/2011: Corrected for goof on the last number (thanks Graham Toal).