Grid edges can sometimes have a jagged or rough appearance.
The jagged appearance along an edge is caused by the low resolution of the grid file. When the NoData boundary goes through a grid cell, the entire cell is assigned the NoData value. The larger the grid cells, the more jagged an edge will look. The smaller the grid cells, the more smooth an edge will look.
To reduce the jagged appearance of the edge of a grid file, you can reduce the size of the grid cells by increasing the grid resolution (number of grid nodes). You can do this following one of these methods:
Method 1: Regrid the data. Click Grids | New Grid | Grid Data and regrid the original data with more grid nodes, and then use the higher resolution grid file for assigning the NoData value. The default is 100 grid lines in the longest direction. In the Grid Data dialog, increase the # of Nodes by a factor of 2, 5, or 10 for increasingly higher resolutions and smoother edges.
Method 2: Smooth the grid. Smoothing the grid file using Grids | Edit | Spline Smooth to insert more nodes in the grid file prior to assigning the NoData value.
Method 3: Resample the grid. Inserting more grid nodes by clicking Grids | Resize | Mosaic, selecting the grid file and clicking Open. In the Grid Mosaic dialog, increase the # of Nodes values and click OK.
Once the grid has a higher resolution, update the map with the higher resolution grid file. If you have a vector data file of the NoData boundary, you may also want to consider adding a base layer on top of the contour map and setting the line width for the base layer to a slightly larger width to mask the jagged edge.
Updated October 26, 2018