For a contour map, the only time you should set the colormap to use logarithmic scaling is when the Level method for the contours layer is set to Logarithmic. If you use the Simple contour level method, and a logarithmically-scaled colormap, then most of the colors will be squished at one end of the color scale. Select the contour map, click the Levels tab in the Properties window, and change the Level method to Logarithmic.

Updated December 2, 2021