Populate the moment matrices with the categorical entries