Optical systems are a popular choice for quality inspection because they are not only contactless and precise but also comparably fast. Particularly in cases where a 100% quality inspection is required low measurement and evaluation time is of essential importance. With high production rates of several parts per second, manual inspection is not feasible anymore and the evaluation needs to be automatically carried out by algorithms. In this article we propose a fast method for anomaly detection in image data based on principal component analysis and filtering. The method shows competitive performance on a data set of challenging surface inspection.