Destroy the categorical variables when the covariance matrix is destroyed