Do you ever delete tests?

Yes. There are basically two fundamental reasons for deleting tests:

1. The test is syntactically poor: it doesn’t compile, it’s commented out, it has been refactored into nonsense, or it has been used for training by a developer who just started learning about unit testing.
2. The test is redundant or outgrown. Sometimes tests are written from one angle, but then something changes and suddenly there’s a better approach, method, scope, or API boundary which makes the old test look poor and duplicated.

Book References
Read more about this topic in Developer Testing: Building Quality into Software:

• Chapter 17: Working with Test Code, pages 241-243