factor: Fix return type of do_factor_by_matrix().