I am wondering how can I issue a HTTP Status code, that will cause crawlers to try later for that content, as the system currently is running on high load.
Does a 304 status and no content display resolves this issue?
You should issue a 503 (Service Unavailable) with the Retry-Later option set. This seems to be an accurate solution.
See: http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html and http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
Well, if you answer with a 304 to an unconditional request, a client will likely just sort your answer as "generic error" (like 500). Why don't you send the status code intended for this kind of situation, 503 Service Unavailable?.
A 304 HTTP Status code means Not Modified.
If your content has not been modified since the last time the crawler came to its page (that's assuming the crawler send a HTTP header to indicate when it came), I suppose it's OK to send a 304 HTTP status -- in fact, it would be the right thing to do.
But you should probably not send a Not Modified status code if :
Instead, if your webserver is overloaded, the right HTTP status code would be 503 Service Unavailable (quoting) :
The server is currently unavailable (because it is overloaded or down for maintenance).
Generally, this is a temporary state.