Making Statamic faster: everything we changed in a week

A client site kept falling over. The cause was Statamic's content cache, the Stache: rebuilding it from cold took 103 seconds and 2 GB of memory. Chasing that turned into a week of measuring the cold build, page rendering, the control panel and finally the YAML parser itself.

It produced eight pull requests to Statamic from us and two from Ryan Mitchell, a new free addon called Flatpack that cuts a 62-second build to 19, and a YAML parser, built with Ryan, that reads a large site's content in 1.6 seconds where Symfony's takes 22.

This is all of it: what we changed, what we measured, and what we tried that didn't work.

The outage that started it

The client site runs Statamic 5.73 with full static caching across 11 sites. Two things combined into outages.

• A cold Stache was very expensive. php please stache:warm on a clean machine took 103 seconds and 2.0 GB. Any deploy or cache clear meant the first requests paid for that.
• A cache stampede. Saving a navigation or a global flushed every static page on all 11 sites, while a listener forgot all 44 cached menu fragments at the same moment. Every visitor then rebuilt menus and pages at once.

The stampede was ours to fix, and the fix was ordering: rebuild the fragments in a job first, then flush the static pages, and never write a stale fragment into a static file. The 103 seconds was Statamic's, so that's where the measuring started.

Where the 103 seconds went

It wasn't one problem. It was four.

Measured end to end, as the median of three cold builds:

ext-yaml was tempting. It parsed everything in 2.6 seconds against Symfony's 21.5. But it isn't installed on Laravel Cloud, and it disagreed with Symfony on two files. That set the rule for everything after: it has to run on a stock Forge or Laravel Cloud server, with no extra extensions.

A benchmark, and an automated search

To stop arguing from one site, we built a repeatable benchmark: Statamic 6.40 on the stock skeleton, flat files, and generated content, with collections of 100 to 319 entries, an 85 KB page-builder blueprint and navigations from 84 to 4,680 nodes. It has scenarios for cold builds, renders, navs and saves. Every scenario's output is compared with a recorded snapshot, so a change that alters the output fails.

We then turned it into a hill-climbing problem: an automated loop where an AI model proposes a change to Statamic's source, the benchmark scores it, and the best candidate seeds the next round. It was useful, mostly for what it taught us about the benchmark.

• The first runs scored 1.76 and 2.14 times stock. Most of the first was a fast YAML parser. Most of the second was pausing PHP's garbage collector, which added 123 MB to one scenario and ran out of memory at 256 MB. The scorer now rejects anything that raises peak memory by more than 20%, plus a little slack.
• A later run on rendering scored 1.70. Then we found the benchmark was running many requests in one PHP process, so static caches carried from one request to the next, which never happens under PHP-FPM. With each request in its own process, 1.70 became about 1.16.

We ported the surviving ideas by hand into separate branches and re-measured them one at a time on the fixed benchmark. Of five, only two were worth opening: working out fieldtype config defaults once (11% faster renders) and fast paths for augmentation (5%).

Rethinking the Stache, and Flatpack

On the client site, the page-builder field is 96.4% of all entry YAML. Most requests, such as listings, navs, URLs and queries, never read it. So the idea was to parse the small fields up front and leave any top-level field over 2 KB unparsed until something asks for it.

We tried other things on the way. SQLite as the place the Stache is stored gave no gain, and requests were about 10% slower. A SQLite-backed entries addon took the cold first page from 55 seconds to 1.1, but a rebuild still took 46 to 64 seconds, and it needed seven fixes before it round-tripped files exactly.

The version that worked became Flatpack: a cache store for the Stache that writes plain PHP files, which OPcache serves from memory, with large fields compiled separately and loaded on demand. Your content stays in Markdown and YAML files. It's free (MIT), works on Statamic 5 and 6, and installs with composer require 1994/flatpack-statamic. The source is on GitHub.

An addon that replaces how Statamic stores its cache has to give exactly the same answers as Statamic. So Flatpack has a parity suite. It installs a fresh Statamic at each supported version, runs it plain and with Flatpack, and compares every entry, query, structure and term after reads, saves, a rebuild and files changed on disk. It runs daily, including against Statamic's unreleased branch. Flatpack also checks at boot that the Statamic internals it relies on still exist, and switches itself off if one has gone.

Building the parity suite found two bugs in Statamic's file watcher: files deleted on disk were never removed from the Stache, and editing a tree file moved a page but left its URL stale. Both became pull requests.

Rendering and the control panel

Rendering

We profiled 101 real pages of the client site, with a median of 1.6 seconds each on a warm Stache.

• Antlers re-parses every template on every request, about 30% of render time. A persistent cache of parsed templates took the median from 1,610 to about 1,240 ms, 23% less time, with byte-identical HTML on all 101 pages. On the benchmark, which has three small templates, it made no difference at all, so it's held back until there's a fair way to show it.
• Reading an entry from the Stache snapshots its whole state for dirty-checking (about 11%), and blueprint fields are resolved twice (about 8%).
• Blueprints are parsed from YAML on every request. An 85 KB page-builder blueprint costs 18 ms each time with Symfony.

The control panel

We timed all 43 control panel pages as a logged-in user, one request per PHP process.

Before opening those, we compared every response body byte for byte with stock, fingerprinted 14 screens in a real browser before and after, and searched public addons for anything the listing change could break. We found one addon, with one star.

One correction worth owning: the tree change was first written up as taking 10% less time on a 319-page collection. Re-measured with stock and the branch requested alternately, it was 3%. We corrected the pull request.

The YAML parser

Ryan Mitchell wrote a fast parser for the YAML Statamic itself writes. His commit replaces one line in Statamic\Yaml\Yaml::parse(): where it called Symfony's parser, it now calls a new FastParser.

Symfony handles a nested block by collecting its lines and parsing them again with a new parser, so every line is scanned once per level of nesting. Bard and page-builder content is deeply nested, which makes that expensive. Ryan's parser walks the lines once, and gives the whole document back to Symfony for anything it doesn't understand, so errors are always Symfony's errors. On the benchmark it halved a cold stache:warm, from 2.20 to 1.10 seconds.

Flatpack already shipped its own fast parser, for the build and for blueprints. So the two overlap, and the obvious question was which one is faster. It depended on the content:

* 4,112 entries, 190 blueprints and fieldsets, and 794 other YAML files: navs, trees, globals, taxonomy terms and forms.

Neither gave a wrong result. The difference was how often each gave up. On the client site, Flatpack's parser handed 1,776 of 4,112 entries (43%) back to Symfony. Ryan's handed back 22.

That came down to one design decision. When Flatpack's parser met anything outside its subset, it gave up on the whole file. Ryan's hands just the awkward value to Symfony's inline parser and carries on. Real content is full of awkward values: double-quoted strings with escapes, which is how the dumper writes some Bard text, keys like 1_simple_content: or 'New Zealand':, and empty maps written as { }. The generated benchmark had none of them.

So we built a hybrid: Flatpack's faster loop, a recursive descent with no stack of frame objects, with every one of Ryan's rules for when and how to give up.

The fuzzer

A parser that replaces Symfony's has one job besides being fast: return exactly what Symfony returns. So the test is differential. Generate a document, parse it both ways, compare.

for each case:
    doc = random data written by Symfony's own dumper (several styles and indents)
          or a real document with one to three random mutations
    fast = fastParser(doc)          // null means "I give up, ask Symfony"
    if fast is null: continue       // giving up is always allowed
    expected = symfony(doc)
    fail if symfony threw           // the fast path must not accept what Symfony rejects
    fail if serialize(fast) != serialize(expected)   // types and key order must match

The mutations are what find bugs: delete a line, change an indent, insert a #, a tab, a quote or ---, swap two lines, truncate. The comparison uses serialize(), so an integer is not a string.

It found bugs in all three fast parsers. The worst was in Flatpack 1.0's own parser: - - one, a list inside a list item, was read as the text - one. That's wrong data, silently, in a released parser that had passed a parity suite on real content. It found a key called null slipping through our first hybrid because of an isset() check, and 12 differences in 800,000 cases in Ryan's.

It also found a quirk in Symfony itself. In symfony/yaml 8.1, when lines have been stripped from the top of a document, a | block that is last in a nested mapping can lose its final newline:

---
a:
  b: |
    one
    two
c: 1
# b is "one\ntwo", not "one\ntwo\n". Remove the --- and the newline comes back.

Statamic front matter always starts with ---, so this happens with real entries. The fast parser gives those documents to Symfony. The goal is to match Symfony, quirks included.

Versions disagree with each other, too. Statamic allows symfony/yaml ^7.0.3 || ^8.0, and the same fuzz against 7.0.3 produced 390 differences that 8.1 didn't have. So the rule became: wherever versions disagree, give up. The final run covered seven Symfony versions, and Flatpack's copy was also run on 6.2 and 6.4 for Statamic 5. 8.4 million cases, no differences.

After a few rounds of cutting fallbacks (tabs in pasted scripts, |- blocks near the end of a file, and 105 copies of the two-byte document { }), we measured the floor. Building the same arrays in C with unserialize takes 7.5 ms on the benchmark, and only splitting lines and cutting out keys and values takes 23. The hybrid takes 86 ms, or 42 with PHP's JIT on. What's left is PHP executing roughly 75 opcodes per line, so that was the point to stop. By then only 107 of the 5,096 files went back to Symfony, and 105 of those were empty { } files.

On the benchmark, a cold stache:warm went from 2.20 seconds on stock to 0.95. Ryan also caught two things in review. My nesting guard counted columns of indentation, where his counted real depth, and his was right. And the parser's static cache of known keys would have grown without limit under Octane, so it now stops at 2,048 keys.

The hybrid is now part of Ryan's pull request to Statamic, and a Flatpack pull request swaps it in for Flatpack's own parser, with the fuzzer added to the parity suite.

Every change and where it stands

What we tried that didn't work

• Pausing the garbage collector. Fast, and 123 MB heavier. Out of memory at 256 MB.
• SQLite as the Stache's storage. No gain, and requests were slightly slower.
• ext-yaml. Five to eight times faster than Symfony, but not available where the sites are hosted.
• A cheaper computed field on the client site. Slower, and it changed 30 URLs.
• Three of the five ported render changes. They scored well in a benchmark that shared one process between requests, and did nothing once that was fixed.
• A regex tokeniser for the YAML parser. The prototype was fast because it skipped most of each file.

Lessons

1. Measure the real site first. The 103 seconds was four separate causes, two of them already fixed upstream in versions the site wasn't on.
2. Benchmark the way production runs. One request per process. Sharing a process between requests invented a 1.7× win that was really 1.16×.
3. Score memory as well as time. Otherwise the cheapest optimisation is to stop freeing memory.
4. Generated content lies in a particular direction. It's tidy. Real content has escaped quotes, odd keys and pasted scripts. Flatpack's parser looked 20% faster than Ryan's on generated content, and was 3.3 times slower on real content.
5. Compare output, not just speed. Byte-identical HTML, identical URLs, identical JSON, a parity suite, a fuzzer. Those checks found more bugs than the profiler did.
6. For small differences, alternate. Run A and B in turn on separate copies. Running all of A, then all of B, measured the machine warming up.
7. Profilers have side effects. One held a lock and made good fixes look terrible.
8. One change per pull request, each measured alone. Together, the render changes gave 18 to 23%. Alone, most gave nothing. Only the ones that stood up alone were opened.
9. Skip work instead of speeding it up. The biggest wins were not parsing 96% of the YAML until it's needed, not sending a megabyte to show "5 sets", and not parsing a file twice.
10. Give up small. In the YAML parser, handing one awkward value to Symfony instead of the whole file took fallbacks from 38% of YAML files to 2%, almost all of them empty { } files.
11. If you replace a parser, fuzz it against the original, on every supported version.
12. Find the floor before polishing. Knowing the limit made it easy to stop.
13. Correct your numbers in public.
14. Static state outlives the request. It's harmless under PHP-FPM, and needs a limit under Octane.