Being close to launch of a Magento 2 project for a client, we started Magento configuration fine tuning. To our big surprise, one of the categories just wouldn’t load, throwing a 503 error. As we do, the error arose when we started testing the project on production ready server.

Looked like Varnish was having issues with categories that contained larger number of products and we needed to make it ready ASAP.

After a while we had a solution worth sharing with the Magento community.

The issue

As you know, when using Varnish as the caching application in Magento 2, each page is tagged by cache tags of the content displayed on it. These tags are then used for cache invalidation and purges, once content is updated.

For example, if you open a home page on default installation, it will be tagged with

store, cms_page and cms_page_1 cache tags.

Categories are tagged with cache tags of all products they have assigned. This means that, if you have a large catalog, with a single category containing around 2000 products, you’re likely to have about 20kB worth of cache tags when first loading that category.

Cache tags are sent to Varnish as ‘X-Magento-Tags’ header on initial, non-cached request to Varnish, so it can tag the URL with displayed content, and be able to clear all pages where a object appears on request.

Once’X-Magento-Tags’ header gets longer than 8kB, default ‘http_resp_hdr_len’ Varnish limit is hit and Varnish returns a 503 error to customers (backend fetch failed).

Even after implementing the workaround from Magento Documentation, the issue was still there, due to another limit on Apache/nginx HTTP server might be hit, as they both have 8kB max header value by default.

AM I affected by this issue?

Your store is affected if you are:

• using Magento 2 (any version)
• having a large number of products on store, with at least one anchor category containing at least 2000 products
• using Varnish as Magento 2 caching application, with default parameters and Magento-generated VCL file

The solution

It’s not uncommon for a store to have more than 2000 products assigned to a single category, but it’s unfortunate that Magento 2 caching basically doesn’t work for medium-sized merchants because of a limit that was looked over.

You’re in luck, because, as part of our efforts to make Magento better for everyone, we created a pull request and a Magento 2 module, both of which address these issue and provide out-of-the-box solution.

Feel free to contribute, use and spread the word.