I should try to spell our that algorithm with lots of double variables instead of immutable vector classes to cut down on object creation on the heap and garbage collection.
And do the hit tests in float on multiple CPUs, then calculate the exact hit location again in double.