When a crawler like GoogleBot tries to crawl a URL it will first test the HTTP header returned by the server
Example:
HTTP / 1.1 404 Not FoundDate: Tue, 03 Jan 2017 14:17:38 GMTServer: ApacheX-FRAME-OPTIONS: DENYX-XSS-Protection: 1; mode = blockVary: Accept-EncodingContent-Encoding: gzipContent-Length: 5019Content-Type: text / html; charset = UTFIf the response code starting by 4 (example 404 or 410) or by 5 (503 or 500), it means that the page can not be explored. Therefore, Googlebot will go explore some other contents without viewing the HTML content returned for your page.
However, the user will be confronted to a HTML content mentioning that a problem occurred = an extremely disappointing moment in the user journey.
Create a custom 404 page (also think of creating a custom page by type of error: 403,500 etc.) in order to keep the users and redirect them to new pages: visuals, site maps, promotions ... For an Apache server, for example, you can go to your .htaccess file located at the root of the server and paste the following lines while customizing the "ERRORS" folders: