Helpful Tips For Integrating Rasters
By Andrew M. Wright, May 2018
Adding a Coordinate System/Spatial Reference to a Raster
This cannot be stressed enough. Every GeoTIFF must contain the coordinate system! This ensures that the raster will be projected properly and accurately, and if sharing data with collaborators, yours must be able to play nicely with their own. It is possible for a GeoTIFF to have no coordinate system (that is to say, no Spatial Reference), but to be able to display properly. This is because the raster can know at which Extent to project in space. In other words, at which northings and eastings its top, left, right, and bottom sides should be set. However, when viewing the raster in a coordinate system other than that in which it was intended to be viewed may result in failure, distortion, or inaccuracy in placement. This is particularly important at Zincirli since we use a custom, local coordinate system! Whenever using third-party software like Agisoft PhotoScan, be sure to choose the proper coordinate system (you may have to upload the PRJ file) and choose “Write World File” when exporting a GeoTIFF. To see if a GeoTIFF has Spatial Reference, import it into ArcMap. It will warn you if there is no coordinate system by saying that the data source has “unknown spatial reference.” Alternatively, find the file in ArcCatalog, right click it, choose Properties, and find Spatial Reference information there. If there is no coordinate system, but you know in which coordinate system the raster was made, and the extent is correct*, follow these steps:
1) Open ArcCatalog.
2) Navigate to the file in question.
3) Right click the file and choose Properties.
4) Under Spatial Reference, click Edit, and choose the proper coordinate system.
5) Click OK.
If there is no extent, or the extent seems wrong in any way, it is easiest to start from scratch, and re-georeference the raster in ArcMap.
*You can tell if the extent is correct in ArcMap by adding a raster from the same geographic area with spatial reference. Then, add the problem raster. If the problem raster lines up, the extent is correct, and you just need to add the coordinate system.
Adding Pyramid Files to a Raster
Some rasters may be missing pyramid files. Typically, these are sidecar files with the extension OVR (“overview”). A pyramid file contains various scaled versions of the full-size raster, and data that allows the raster tiles to be rendered quickly and sharply. If there are no pyramid files, and the GeoTIFF is displayed in OCHRE or ArcMAP, when you are not zoomed to the full extent of the raster, the image will be blurry, and when you do zoom to the raster resolution, the raster will render slowly. Some third-party programs are capable of creating internal pyramids within GeoTIFFs*, so the sidecar file may not always be present. To find out if a GeoTIFF has pyramids, navigate to the file in ArcCatalog, right click it, and choose Properties. Under Raster Information, you will see details on Pyramids. If there are no pyramids, the quickest and easiest way to build them is by opening the raster in ArcMap. ArcMap loves pyramids and will usually prompt to create them whenever you import rasters. If you do not see this prompt, either A) the raster already has pyramids, B) you disabled the prompt, allowing for the automatic creation of pyramids, or C) you disabled the prompt, disallowing for the automatic creation of pyramids. Check your settings! Pyramids will, as stated before, be generated as OVR sidecar files in the same directory as the raster. ArcMap does not create internal pyramids except for certain compressed files that you will probably never encounter.
*For example, when exporting an orthomosaic, Agisoft PhotoScan gives the option to “Generate TIFF Overviews”. This does not create a sidecar OVR file, but embeds the pyramids in the TIFF, resulting in a larger TIF file size.
Fixing Old Rasters That Don’t Display Properly in OCHRE
(If using ArcMAP this requires minimum version of 10.5.1.)
Sometimes in OCHRE, when you try to view a raster, it may not display at all, or it will display only a small, enlarged section of the raster. Usually it will only display properly when you are zoomed in to the full raster resolution. The exact reason for the display problem is unknown, as it seems to affect old GeoTIFFs at random. To fix this problem, follow these steps:
1) Download the problem GeoTIFF and its sidecar files from the server.
2) Open ArcMap.
3) Import the GeoTIFF as a layer in ArcMap.
4) If the raster is missing Spatial Reference, go to View > Data Frame Properties, and choose the proper coordinate system.
5) Right click the raster in the Table of Contents panel and choose Data > Export Data.
6) Leave all options as they are*, except if you must add the coordinate system to the raster. In that case, under Spatial Reference, select “Data Frame (Current).”
7) For Location, choose a new folder. The problem GeoTIFF must not be in this folder!
8) For Name, use the name of the original problem GeoTIFF. This way, OCHRE will be able to find the updated files on the server.
9) For Compression Type, LZW is standard for TIFs.
10) Make sure the original GeoTIFF and its sidecar files are backed up somewhere.
11) Delete the problematic GeoTIFF and all its sidecar files from the server.
12) Upload the new GeoTIFF and its sidecar files in the same directory on the server.
*As explained below, NoData may automatically be set to 256, a change from older versions of ArcMap.
Removing Band_4 in an RGB Raster:
The 4th band in an RGB raster is probably an alpha channel used for transparency. If there’s no need for transparency, it may be removed. (The alpha band also increases the file size of the GeoTIFF).
1) In ArcMap, select the tool “Composite Bands.” This is found under Data Management Tools > Raster > Raster Processing.
2) Simply drag the first three bands (Red, Green, Blue) into the Composite Bands list. If the original raster is already added as a layer in ArcMap, you can drag the desired bands from here. Alternatively, navigate to the file in ArcCatalog, expand it in the left-hand panel, and drag the bands from here.
3) Choose a location and name for your new raster under Output Raster.
4) Adjust the Environments if necessary (the defaults will usually be fine, especially if the raster was previously resampled).
5) Click OK.
Adding an alpha channel seems to be pretty standard in the new version of ArcMap. Even when creating rasters in grayscale (for example, when georectifying colorless top plans), it automatically sets the NoData Value to 256. That way, any borders around the image would be transparent (when existing rasters have borders, see the instructions below on how to get rid of them). Adding an alpha channel elevates the raster from 8_BIT_UNSIGNED to 16_BIT_UNSIGNED. The file size ends up larger, but the ArcGIS API in should have no problem with these files.
Removing White or Black Outlines Around Rasters:
When orthophotos are made containing gaps in the image, or when top plans for squares not oriented north-south are georectified, the background may be filled in either white or black in the shape of a square to the maximum extent of the GeoTIFF. This can be ugly, especially if you want to overlay a smaller raster on top of a larger basemap. Here is the first trick to get rid of the black or white border without resampling the raster, so that it appears as transparent:
1) Navigate to the file in ArcCatalog.
2) Right click it and choose Properties.
3) Under Raster Information is NoData Value. Click Edit.
4) If the border is black, change all values here to 0. If white, change all values to 255.
5) Click OK. All pixels that are either white or black will now be displayed as transparent. (The downside is that if any pure black or white pixels occur anywhere else in the raster besides the border, they will also be displayed transparent.)
If you still see a border, or if only some of the pixels turned transparent and others didn’t, then the remaining pixels may be understood as the “background.” You can create a copy of the raster with some new settings in order to have the background displayed as transparent.
1) In ArcMap, select the tool “Copy Raster.” This is found under Data Management Tools > Raster > Raster Dataset.
2) Choose the original raster that has the black or white border as Input Raster.
3) Choose a location and name for your new raster under Output Raster Dataset.
4) For Ignore Background Value, put either 0 (black) or 255 (white) depending on what color your border is.
5) Repeat step 4, except for NoData Value.
6) For Pixel Type, 8_BIT_UNSIGNED is standard.
7) Adjust the Environments if necessary (the defaults will usually be fine, especially if the raster was previously resampled).
8) Click OK.