EE & Varnish produces Error 503 intermittently

Could you post your varnish config file here for us to take a look at?

I am not strictly sure if this is it (we don’t handle this ourselves), but might this be the Varnish config: https://jpst.it/GcCg ?

I suggest trying a simpler config file and then slowly adding one line at a time to match your current config file to see where the issue is. Here’s a great article from EllisLab that is very helpful:

Making Sites Fly with Varnish

Unfortunately, EE isn’t the only thing running on the server, so I am not sure we can take that approach without causing other issues. But I will pass it on to our sysadmin to see what she thinks. Thank you. 😊

Linda,

You should be able to as you can apply varnish settings based on IP or hostname.

Best, Pedro

Thanks for posting your VCL. Is the website having the problems listed in this conditional?

if (req.http.Host == "westeros.org" || req.http.Host == "www.westeros.org" || req.http.Host == "testsite.westeros.org" || req.http.Host ~ "hippokrene.com" || req.http.Host ~ "hippoiathanatoi.com" ) {
    set req.backend = apache;
}

Yes, it is westeros.org/www.westeros.org and hippoiathanatoi.com that’s having the issue (hippokrene.com doesn’t use EE).

Hmm, it’s hard to say then. I wouldn’t think ExpressionEngine would be causing this, it seems like Varnish may think the backend is unhealthy and is just timing out. You already have first_byte_timeout on your apache backend which is good, that was going to be my first suggestion, which is why I asked if your sites were in that conditional.

Outside of that, I’d maybe try increasing the connect_timeout property on the backend from its default of 3.5 seconds. I wouldn’t normally think you’d need to increase that, but worth a shot. Also maybe try looking at the Varnish logs to see if you can see how its deciding to show a 503.

Okay, our friendly server admin set up a version entirely out of Varnish and we still get the error:

http://westeros-test.noderunner.net:8081/

If someone has visited it recently, it’ll pull from EE’s cache… but if not, you’ll get a content encoding error. Unfortunately, I’m not sure how to diagnose the problem further, but this does fit what the server admin speculated:

“t looks to me in Varnish’s logs like what happens is ExpressionEngine sends back a corrupted page; Varnish can’t parse it as a valid HTTP body and dies.

If my theory is right, then Chrome or Safari will also refuse to render the page appropriately when you trip the issue; i.e., the Varnish Guru issue isn’t Varnish, it’s just Varnish tossing up a webpage when ExpressionEngine throws back random garbage data instead of text/html content.”

Interesting. Do you have gzip compression enabled for this site? I’d try disabling that, if so, or any other output compression you may have going on.

Thank you, looking at the gzip led us down the right path. It turned out to be an issue with Magpie pulling in an RSS stream that had changed and was throwing errors.

Ah interesting! Thanks for letting us know, glad you got it sorted!