How do you remove colors from the swatches palette?

Is there a way to remove colors from the palette? I'm referring to the Fonts section and I'm experimenting with different colors for comments and strings etc.

There is a color I added that I wish to remove but I don't see a way to remove it.