How to make action to be called by only the mvc application, Not by direct url request - asp.net-mvc-4

Question

In my mvc application I handle 404 and 505 errors by checking response code in Application_EndRequest() method in global.asax.cs and returning custom 404/error page like below
protected void Application_EndRequest()
{
if (Context.Response.StatusCode == 404)
{
Response.Clear();
var rd = new RouteData();
rd.Values["controller"] = "Errors";
rd.Values["action"] = "PageNotFound";
IController c = new ErrorsController();
c.Execute(new RequestContext(new HttpContextWrapper(Context), rd));
}
else if (Context.Response.StatusCode == 500)
{
Response.Clear();
var rd = new RouteData();
rd.Values["controller"] = "Errors";
rd.Values["action"] = "ServerError";
IController c = new ErrorsController();
c.Execute(new RequestContext(new HttpContextWrapper(Context), rd));
}
}
}
And I have Errors controller like below
public ActionResult PageNotFound()
{
Response.StatusCode = (int)HttpStatusCode.NotFound;
return View();
}
public ActionResult ServerError()
{
Response.StatusCode = (int)HttpStatusCode.InternalServerError;
return View("Error");
}
Now when I type the url in addressbar like http://mydomain/errors/servererror, It displays my current custom error page
So I want to prevent user by accessing this page, so how can I accomplish that?
P.S.
So far I have tried to solve this by configuring route named "Errors/ServerError" and redirecting it to somewhere else, but I think there may be another good way out there..