Completely agree with what you said! And that’s why I said it works better when you have a lot of correlated features. If you drop noisy features from a group of decently correlated features, you won’t be losing out on much information due to presence of other correlated features. You’re just eliminating information which is present in noisy form, but available elsewhere in better form.

The thing you mentioned about interaction between two features is also the reason why I say you shouldn’t drop too many important features.

Ultimately, your choice of features should be based on the one which gives best results on your validation set. If you have your validation strategy right, you’ll see better performance on test for better performance on validation data through such feature selection. In real life, the best validation strategy is to have an out-of-time validation set because your real test set is basically future data which is out-of-time.

Hope that helps.