Posts

Go's race detector has a mutex blind spot

2025-07-28T00:00:00+00:00

I recently read Ralf Jung's blog post "There is no memory safety without thread safety</a>" which mentions that Go is not a memory safe language in the presence of data races.

"But Go comes with a built in data race detector 1</a>" some might say.

This reminded me of a quirk in Go's dynamic data race detection that causes it to miss data races in executed code that could easily be spotted by a human 2</a>.

Here is the code the data race detector struggles with:

package</span> main</span></span>
</span>
import</span> (</span></span>
        "</span>fmt</span>"</span></span>
        "</span>sync</span>"</span></span>
)</span></span>
</span>
var</span> counter</span> int</span></span>
var</span> mutex</span> sync</span>.</span>Mutex</span></span>
</span>
func</span> increment</span>(</span>wg</span> *</span>sync</span>.</span>WaitGroup</span>,</span> id</span> int</span>)</span> {</span></span>
        defer</span> wg</span>.</span>Done</span>(</span>)</span></span>
        mutex</span>.</span>Lock</span>(</span>)</span></span>
        counter</span>++</span></span>
        fmt</span>.</span>Printf</span>(</span>"</span>Counter: </span>%d</span>\n</span>"</span>,</span> counter</span>)</span></span>
        mutex</span>.</span>Unlock</span>(</span>)</span></span>
        if</span> id</span> ==</span> 1</span> {</span></span>
            counter</span>++</span>;</span></span>
        }</span></span>
}</span></span>
</span>
func</span> main</span>(</span>)</span> {</span></span>
        var</span> wg</span> sync</span>.</span>WaitGroup</span></span>
        wg</span>.</span>Add</span>(</span>2</span>)</span></span>
</span>
        go</span> increment</span>(</span>&</span>wg</span>,</span> 0</span>)</span></span>
        go</span> increment</span>(</span>&</span>wg</span>,</span> 1</span>)</span></span>
</span>
        wg</span>.</span>Wait</span>(</span>)</span></span>
        fmt</span>.</span>Printf</span>(</span>"</span>Final: </span>%d</span>\n</span>"</span>,</span> counter</span>)</span></span>
}</span></span></code></pre>
Threads 0 and 1 increment a shared counter, guarded by a lock. Thread 1 then performs an additional increment on the counter, not guarded by the lock.</p>
If thread 1 acquires the lock first, it is possible for the unguarded write to happen at the same time as the guarded write on thread 0.</p>
The race detector confirms this:</p>
% go run -race race.go</span></span>
Counter: 1</span></span>
==================</span></span>
WARNING: DATA RACE</span></span>
Read at 0x000105026dd0 by goroutine 6:</span></span>
  main.increment()</span></span>
      /Users/brad/race.go:14 +0x80</span></span>
  main.main.gowrap1()</span></span>
      /Users/brad/race.go:26 +0x38</span></span>
</span>
Previous write at 0x000105026dd0 by goroutine 7:</span></span>
  main.increment()</span></span>
      /Users/brad/race.go:18 +0x158</span></span>
  main.main.gowrap2()</span></span>
      /Users/brad/race.go:27 +0x38</span></span>
</span>
Goroutine 6 (running) created at:</span></span>
  main.main()</span></span>
      /Users/brad/race.go:26 +0xac</span></span>
</span>
Goroutine 7 (finished) created at:</span></span>
  main.main()</span></span>
      /Users/brad/race.go:27 +0x110</span></span>
==================</span></span>
Counter: 3</span></span>
Final: 3</span></span>
Found 1 data race(s)</span></span></code></pre>
... but only sometimes ...</p>
% go run -race race.go</span></span>
Counter: 1</span></span>
Counter: 2</span></span>
Final: 3</span></span></code></pre>
Go's data race detector is pretty neat. It detects unsynchronized concurrent accesses to shared memory, regardless of the actual timing during execution. For example, even if you add a sleep to ensure the threads never actually run simultaneously, Go will still report a race because they access the same variable without synchronization.</p>
func</span> increment</span>(</span>wg</span> *</span>sync</span>.</span>WaitGroup</span>,</span> id</span> int</span>)</span> {</span></span>
        defer</span> wg</span>.</span>Done</span>(</span>)</span></span>
        if</span> id</span> ==</span> 1</span> {</span></span>
           time</span>.</span>Sleep</span>(</span>10</span> *</span> time</span>.</span>Second</span>)</span></span>
        }</span></span>
        counter</span>++</span></span>
        fmt</span>.</span>Printf</span>(</span>"</span>Counter: </span>%d</span>\n</span>"</span>,</span> counter</span>)</span></span>
}</span></span></code></pre>
This is useful for detecting cases that rely on specific timing. For example, the tool would be much less useful if the race below could only be detected when bar()</code> returns quickly.</p>
func</span> foo</span>(</span>wg</span> *</span>sync</span>.</span>WaitGroup</span>,</span> id</span> int</span>)</span> {</span></span>
        defer</span> wg</span>.</span>Done</span>(</span>)</span></span>
        if</span> id</span> ==</span> 1</span> {</span></span>
           bar</span>(</span>)</span> //</span> usually slow, sometimes fast</span></span>
        }</span></span>
        counter</span>++</span></span>
}</span></span></code></pre>
However, in the original mutex case above, the unguarded write is always executed by thread 1, and yet Go is missing the race unless it actually occurs at runtime.</p>
It has been a while, so things may have changed or I may be misremembering, but I believe it is due to how locks are modeled in the data race detector's analysis.</p>
The data race detector creates a graph of "happens-before" relationships3</a></sup>. Operation A "happens-before" operation B if we can guarantee A completes before B starts. The detector uses these relationships to determine if two memory accesses could occur simultaneously. For example,</p>

a thread spawn must happen before the thread starts executing</li>
a thread must complete before it is joined</li>
</ul>
Therefore, all instructions before a thread is spawned happen before all instructions run in the thread, which must happen before all instructions that happen after the thread join.</p>
The data race detector checks for a happens before relationship when two threads access the same location in memory. If neither access is reachable from the other, a data race is reported.</p>
This is how Go is able to detect the sleep example. Sleep does not create a happens before edge between threads and so the race is detected regardless of how the threads are actually scheduled.</p>
However, locks don't fit cleanly into this model. Go forces locks to fit by modeling acquire/release as happens before relationships. If thread 0 acquires a lock first, the unlock call on thread 0 must happen before the lock call on thread 1.</p>
Here's a rough chart of what the graph looks like.</p>
Time↓  Thread 0          Thread 1         Main Thread</span></span>
═════ ═══════════       ═══════════       ═══════════</span></span>
  │   START <----------------------------- go increment(&wg, 0)</span></span>
  |                     START <----------- go increment(&wg, 1)         </span></span>
  │                     mutex.Lock()</span></span>
  |.                    counter++</span></span>
  │                     fmt.Printf() </span></span>
  │   mutex.Lock() <--- mutex.Unlock() </span></span>
  │   counter++         counter++ (RACE)</span></span>
  │   fmt.Printf()      wg.Done()-----┐     </span></span>
  │   mutex.Unlock()                  |</span></span>
  │   wg.Done() ------------------------> wg.Wait()</span></span>
  │                      </span></span>
</span>
Legend:</span></span>
B<-A : A must happen before B</span></span>
C->D : C must happen before D</span></span></code></pre>
In the racy schedule, thread 1 acquires the lock first, causing a happens before edge from the unlock on thread 1 to the lock thread 0. The guarded write on thread 0 cannot reach the unguarded write on thread 1, and vice versa, and the race is successfully detected.</p>
Now let's look at the "safe" schedule.</p>
Time↓  Thread 0          Thread 1         Main Thread</span></span>
═════ ═══════════       ═══════════       ═══════════</span></span>
  │   START <----------------------------- go increment(&wg, 0)</span></span>
  |                     START <----------- go increment(&wg, 1)         </span></span>
  │   mutex.Lock()      </span></span>
  │   counter++  </span></span>
  │   fmt.Printf()      </span></span>
  │   mutex.Unlock() -> mutex.Lock() </span></span>
  |                     counter++</span></span>
  │                     fmt.Printf()    </span></span>
  │                     mutex.Unlock() </span></span>
  │                     counter++ </span></span>
  │                     wg.Done()-----┐ </span></span>
  │    wg.Done() ------------------------> wg.Wait()</span></span>
</span>
Legend:</span></span>
B<-A : A must happen before B</span></span>
C->D : C must happen before D</span></span></code></pre>
The two guarded writes are safe because the guarded write on thread 1 is reachable from the guarded write on thread 0 via the happens before relationship between the lock and unlock.</p>
However, this also means the unguarded</em> write on thread 1 is reachable from the guarded write on thread 0. And so, Go does not report a race. While technically correct for this execution, the threads could just as easily have acquired the locks in a different order. The end result is a possible race was missed in executed code.</p>
Even so, Go's data race detector remains a best-in-industry tool. No other language has better tooling for easily getting useful reports on data races4</a></sup>.</p>
Modeling locks as synchronization points causes blind spots, but this was likely a design decision made to ensure high performance and no false positives5</a></sup>.</p>
Like any tool, Go's data race detector works best when you understand its boundaries, and I thought this was a neat non-obvious pattern that it can miss. Just because code is covered and go reports no races, does not necessarily mean the code is race free.</p>
^{1</sup>
a.k.a Thread Sanitizer</p>
</div>}^2</sup>Or an LLM</a></p>
</div>
^{3</sup>
As far as I know thread sanitizer is using some form of hyper optimized vector clocks.</p>
</div>}^4</sup>I've written a proof of concept data race detector for Rust</a></p>
</div>
^{5</sup>
Lockset analyses can be pretty cool though...</p>
</div>}

Solving Advent of Code at Compile Time with Rust Macros

2024-12-14T00:00:00+00:00

Intro
The Problem </a> </h2>
Go read the full problem description</a>.
Roughly summarized, given two columns of numbers, sort both columns, calculate the absolute difference between corresponding elements, and sum all differences together.
If you can bear to witness it, a python solution is shown below.
Warning: This code completes in a shocking O(n log n) time complexity and allocates memory with reckless abandon.
input = [ 3, 4, 4, 3, 2, 5, 1, 3, 3, 9, 3, 3, ] left = sorted(input[::2]) right = sorted(input[1::2]) print(sum(abs(l - r) for l, r in zip(left, right))) # 11</code></pre>
Not only is the python solution barely legible, it is also horribly slow at O(n log n). If we want to keep AI from taking over programming jobs, we need to set the bar higher. Anything above O(1) will soon be unacceptable 1</a>.
A noble attempt, but far from a perfect program.
Perfection
powerful</a> tool. But the title of the blog says "with Rust macros" so we will use declarative macros instead 2</a>.
Splitting the list
Here is the macro</a>:
macro_rules! day1 { // match two numbers at a time, storing them in left and right lists (@split [$($lcol:tt)*] [$($rcol:tt)*] $l:tt $r:tt $($tail:tt)*) => { day1!(@split [$($lcol)* $l] [$($rcol)* $r] $($tail)*) }; // base case when no numbers remain (@split [$($lcol:tt)*] [$($rcol:tt)*]) => { todo!() }; // entry ($($tail:tt)*) => { day1!(@split [] [] $($tail)*) }; }</code></pre> Without getting too much into the weeds with rust macros, a macro defined with macro_rules!</code> is a series of patterns and expansions. Think of it as a series of rules, each with two parts:
macro_rules! some_macro { (pattern1) => { expansion1 }; (pattern2) => { expansion2 }; /* ... */ }</code></pre>
Each rule says, "if you see this pattern, replace it with this expansion". The macro keeps applying these rules until there's nothing left to expand 3</a>. The pattern syntax can be complicated</a>, but we'll stick to the simplest patterns here.
Starting with a simple pattern ($a:tt) => { $a $a };</code>. This tells Rust:
match a single token (tt)4</a> and name it a</code>.</li> replace it by repeating the token twice.</li> </ul> So foo</code> becomes foo foo</code> and 1</code> becomes 1 1</code>. A series of tokens can also be matched using ($($tail:tt)*)</code>. The *</code> means zero or more, like in a regex. For example, given the input 1 2 3</code>: The expansion ($($tail)*)</code> expands back to 1 2 3</code></li> The expansion ($($tail:$tail)*)</code> becomes 1:1 2:2 3:3</code></li> </ul> Essentially, everything in the innermost parenthesis is repeated for each matched token. Lastly, patterns can also match tokens directly. The pattern (abc $($tail:tt)*)</code> will match any input that starts with the token abc</code>. With that crash course, that should be enough to understand the entry rule // entry ($($tail:tt)*) => { day1!(@split [] [] $($tail)*) };</code></pre> This is our setup phase. We Match 0 or more tokens (all/any input).</li> Wrap them in a new macro starting with @split</code>.</li> Add two empty lists, which will eventually hold our left and right columns.</li> Keep the original input tokens at the end.</li> </ul> The @split</code> allows us to control the next pattern to be matched by defining another rule with a pattern that starts with @split</code>. The @_name_</code> is a rust convention. It could be any set of tokens to explicitly match against. So day1!( 1 2 3 4)</code> expands to day1!(@split [] [] 1 2 3 4)</code>. The real magic happens in the next rule. // match two numbers at a time, storing them in left and right lists (@split [$($lcol:tt)*] [$($rcol:tt)*] $l:tt $r:tt $($tail:tt)*) => { day1!(@split [$($lcol)* $l] [$($rcol)* $r] $($tail)*) };</code></pre> This may look complicated, but is easy to understand by following an example. Starting with the expansion from the previous step day1!(@split [] [] 1 2 3 4)</code>, the pattern matches: The token literal @split</code></li> Two empty lists [] []</code> ([$($lcol:tt)*] [$($rcol:tt)*]</code>)</li> The first number 1</code> ($l</code>)</li> The second number 2</code> ($r</code>)</li> And all remaining tokens 3 4</code> ($($tail:tt)*</code>)</li> </ul> The rule then expands to day1!(@split [1] [2] 3 4)</code>. Broken down, the expansion creates a new macro that: Includes 1</code> at the end of the first list ([$($lcol)* $l]</code>)</li> Includes 2</code> and the end of the second list ([$($rcol)* $r]</code>)</li> Keeps the remaining tokens at the end 3 4</code> ($($tail)*</code>)</li> </ul> In other words, we are tracking the state of our columns in two lists and keeping not yet processed tokens at the end. This matches two tokens from the unprocessed input and the expansion places them into the correct list. Incrementally matching tokens from the tail and recursively processing the rest is a pattern called tt munching</a>. Storing the result to be matched by future rules is a pattern called push down accumulation</a>. This process continues until there are no more unprocessed input tokens. day1!(@split [] [] 1 2 3 4) // Match and split first pair (1,2) -> day1!(@split [1] [2] 3 4) // Match and split second pair (3,4) -> day1!(@split [1 3] [2 4]) // No more pairs to match. move to base case</code></pre> At this point, there are no more tokens for $l $r</code> to match, and the pattern for the current rule no longer applies. This leads to the final rule related to splitting the list. // base case when no numbers remain (@split [$($lcol:tt)*] [$($rcol:tt)*]) => { todo!() };</code></pre> At this point we have two lists of unsorted tokens ready for the next phase of our perfect program. Sorting a list</a> </h3> The last section was hopefully a gentle introduction to declarative macros in rust because this section is a little cursed. Next we want to sort a sequence from the previous step like 2 1 4 3</code> into 1 2 3 4</code> at compile time. The issue is macros can't do conditional logic, they only match patterns and expand tokens. There is no way to say if a > b, expand to 'b a', else 'a b'. At least not directly. You could brute force comparison by including a pattern for every possible pair being compared. (@sort 1 1) => { 1 1 }; (@sort 1 2) => { 1 2 }; (@sort 2 1) => { 1 2 }; (@sort 1 3) => { 1 3 }; (@sort 3 1) => { 1 3 }; /* ... */</code></pre> But that would be horribly hacky... Expanding on this idea, we can implement bubble sort as a declarative macro. Full Bubble Sort Macro</a> </h4> playground</a> macro_rules! bubble_sort { (@bubble [$($work:tt)*] [$($final:tt)*] 1 1 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 1] [$($final)*] 1 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 1 2 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 1] [$($final)*] 2 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 1 3 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 1] [$($final)*] 3 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 1 4 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 1] [$($final)*] 4 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 1 5 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 1] [$($final)*] 5 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 1 6 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 1] [$($final)*] 6 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 1 7 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 1] [$($final)*] 7 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 1 8 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 1] [$($final)*] 8 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 1 9 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 1] [$($final)*] 9 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 2 1 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 1] [$($final)*] 2 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 2 2 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 2] [$($final)*] 2 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 2 3 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 2] [$($final)*] 3 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 2 4 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 2] [$($final)*] 4 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 2 5 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 2] [$($final)*] 5 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 2 6 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 2] [$($final)*] 6 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 2 7 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 2] [$($final)*] 7 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 2 8 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 2] [$($final)*] 8 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 2 9 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 2] [$($final)*] 9 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 3 1 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 1] [$($final)*] 3 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 3 2 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 2] [$($final)*] 3 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 3 3 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 3] [$($final)*] 3 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 3 4 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 3] [$($final)*] 4 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 3 5 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 3] [$($final)*] 5 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 3 6 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 3] [$($final)*] 6 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 3 7 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 3] [$($final)*] 7 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 3 8 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 3] [$($final)*] 8 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 3 9 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 3] [$($final)*] 9 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 4 1 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 1] [$($final)*] 4 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 4 2 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 2] [$($final)*] 4 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 4 3 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 3] [$($final)*] 4 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 4 4 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 4] [$($final)*] 4 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 4 5 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 4] [$($final)*] 5 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 4 6 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 4] [$($final)*] 6 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 4 7 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 4] [$($final)*] 7 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 4 8 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 4] [$($final)*] 8 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 4 9 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 4] [$($final)*] 9 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 5 1 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 1] [$($final)*] 5 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 5 2 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 2] [$($final)*] 5 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 5 3 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 3] [$($final)*] 5 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 5 4 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 4] [$($final)*] 5 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 5 5 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 5] [$($final)*] 5 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 5 6 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 5] [$($final)*] 6 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 5 7 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 5] [$($final)*] 7 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 5 8 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 5] [$($final)*] 8 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 5 9 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 5] [$($final)*] 9 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 6 1 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 1] [$($final)*] 6 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 6 2 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 2] [$($final)*] 6 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 6 3 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 3] [$($final)*] 6 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 6 4 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 4] [$($final)*] 6 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 6 5 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 5] [$($final)*] 6 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 6 6 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 6] [$($final)*] 6 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 6 7 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 6] [$($final)*] 7 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 6 8 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 6] [$($final)*] 8 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 6 9 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 6] [$($final)*] 9 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 7 1 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 1] [$($final)*] 7 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 7 2 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 2] [$($final)*] 7 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 7 3 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 3] [$($final)*] 7 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 7 4 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 4] [$($final)*] 7 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 7 5 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 5] [$($final)*] 7 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 7 6 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 6] [$($final)*] 7 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 7 7 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 7] [$($final)*] 7 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 7 8 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 7] [$($final)*] 8 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 7 9 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 7] [$($final)*] 9 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 8 1 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 1] [$($final)*] 8 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 8 2 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 2] [$($final)*] 8 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 8 3 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 3] [$($final)*] 8 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 8 4 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 4] [$($final)*] 8 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 8 5 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 5] [$($final)*] 8 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 8 6 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 6] [$($final)*] 8 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 8 7 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 7] [$($final)*] 8 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 8 8 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 8] [$($final)*] 8 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 8 9 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 8] [$($final)*] 9 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 9 1 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 1] [$($final)*] 9 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 9 2 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 2] [$($final)*] 9 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 9 3 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 3] [$($final)*] 9 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 9 4 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 4] [$($final)*] 9 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 9 5 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 5] [$($final)*] 9 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 9 6 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 6] [$($final)*] 9 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 9 7 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 7] [$($final)*] 9 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 9 8 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 8] [$($final)*] 9 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] 9 9 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 9] [$($final)*] 9 $($tail)*) }; (@bubble [$($work:tt)*] [$($final:tt)*] $tail:tt) => { bubble_sort!(@bubble [] [$($final)* $tail] $($work)* ) }; (@bubble [] [$($final:tt)*]) => { todo!() }; // entry ($($sortme:tt)* ) => { bubble_sort!(@bubble [] [] $($sortme)* ) } }</code></pre> At the core, there are only two types of rules. First is a comparison rule that processes pairs of numbers: (@bubble [$($work:tt)*] [$($final:tt)*] 2 1 $($tail:tt)*) => { bubble_sort!(@bubble [$($work)* 1] [$($final)*] 2 $($tail)*) };</code></pre> This type of rule incrementally matches two numbers, adds the smaller of the two to our working list, and leaves the larger with the remaining numbers to compare. This is repeated until only one number remains, at which point we know it is the largest of the current batch. The second rule adds the largest number to the final sorted list, resets the working list, and continues with the remaining numbers. (@bubble [$($work:tt)*] [$($final:tt)*] $tail:tt) => { bubble_sort!(@bubble [] [$($final)* $tail] $($work)*) };</code></pre> Here is the process visualized on sample input: bubble_sort!(@bubble [] [] 4 3 2 1) // Initial state bubble_sort!(@bubble [3] [] 4 2 1) // 3 < 4, add 3 to work bubble_sort!(@bubble [3 2] [] 4 1) // 2 < 4, add 2 to work bubble_sort!(@bubble [3 2 1] [] 4) // 1 < 4, add 1 to work bubble_sort!(@bubble [] [4] 3 2 1) // 4 is largest, add to final bubble_sort!(@bubble [2] [4] 3 1) // 2 < 3, add 2 to work bubble_sort!(@bubble [2 1] [] 3) // 1 < 3, add 1 to work bubble_sort!(@bubble [] [4 3] 2 1) // 3 is largest, add to final bubble_sort!(@bubble [1] [4 3] 2) // 1 < 2, add 1 to work bubble_sort!(@bubble [] [4 3 2] 1) // 2 is largest, add to final bubble_sort!(@bubble [] [4 3 2 1]) // 1 is largest, add to final</code></pre> Finally a base case for when there is nothing left to sort. (@bubble [] [$($final:tt)*]) => { todo!() }</code></pre> At this point, final</code> contains the sorted sequence of tokens ready to be further processed. Putting it all together</a> </h3> Unfortunately we now have two problems: Our bubble_sort only handles one list at a time, but we have two lists.</li> We cannot nest macro expansions.</li> </ol> Let's see why nesting does not work. This looks logical: (@split [$($lcol:tt)*] [$($rcol:tt)*]) => { answer!( [bubble_sort!($($lcol)*)] [bubble_sort!($($rcol)*)] ) };</code></pre> But this doesn't work. Given the input [1 3 2] [4 6 5]</code>, we would like the bubble_sort macros to expand first, so that we get something like answer!([3 2 1] [6 5 4])</code></pre> But that is not how the rust macro system works. The compiler will recognize answer!</code> as a macro and look for rules within that macro that match the literal tokens [bubble_sort!($($lcol)*)] [bubble_sort!($($rcol)*)]</code>. There is no way to "pass" the expansion of one macro back to a higher macro. This is why we have been using push down accumulation. Macros cannot be nested. Instead of "passing" values back up, we keep intermediate state as tokens at the start and forward them down to the next macro expansion. Instead we will use another pattern, callbacks</a>, to get our sorted lists back "up" as input to another macro. the callback pattern involves passing the name of a macro to be expanded. macro_rules! duplicate_and_call { ($callback:tt $($tail:tt)*) => { $callback!($($tail)* $($tail)*) } } // Now when we call: duplicate_and_call!(foo 1 2 3) // It expands to: foo!(1 2 3 1 2 3)</code></pre> Calling duplicate_and_call!(foo 1 2 3)</code> expands to foo!(1 2 3 1 2 3)</code>. This effectively allows the expansion (1 2 3 1 2 3</code>) to be passed to another macro to be further processed. Now, every bubble_sort</code> rule is updated to also match a callback ($callback:tt</code>) and additional args for that callback (($($state:tt)*)</code>). These are simply forwarded unchanged until the terminal rule is reached. Now, the terminal rule can be written. (@bubble $callback:tt ($($args:tt)*) [] [$($final:tt)*]) => { $callback!( $($args)* [$($final)*] ) };</code></pre> Here's an example of how the bubble_sort</code> macro would be expanded wit the new terminal rule. bubble_sort!(foo (blah blah) 5 4 6) // initial input /* all the intermediate bubble_sort steps */ bubble_sort!(@bubble foo (blah blah) [] [6 5 4]) // terminal rule foo!(blah blah [6 5 4])</code></pre> The bubble_sort</code> macro takes as input a callback</code> name, arbitrary args, and an unsorted list. It will eventually pass the args followed by the sorted list wrapped in square brackets to the callback macro. We can use this to write a double_bubble</code> macro, which takes the two unsorted lists from split_list</code> and expands to two sorted lists using bubble_sort</code>. macro_rules! double_bubble { // match the two unsorted lists ([$($l:tt)*] [$($r:tt)*]) => { // sort the right list, and forward it to double_bubble!(@stage1 bubble_sort!(double_bubble (@stage1 [$($l)*]) $($r)*) }; // match the unsorted left list and the sorted right list (@stage1 [$($unsorted:tt)*] [$($sorted:tt)*]) => { // sort the left list and forward both to double_bubble!(@stage2 bubble_sort!(double_bubble (@stage2 [$($sorted)*]) $($unsorted)*) }; // Match two sorted lists (@stage2 [$($l:tt)*] [$($r:tt)*]) => { todo!() }; }</code></pre> Skipping the bubble_sort</code> expansions, here is how the double_bubble</code> macro is expanded. double_bubble!([2 1 3] [5 4 6]) // initial input -> bubble_sort!(double_bubble (@stage1 [2 1 3]) 5 4 6) // sort first list /* bubble sort expansions */ -> double_bubble!(@stage1 [2 1 3] [6 5 4]) // first list sorted -> bubble_sort!(double_bubble (@stage2 [6 5 4]) 2 1 3) // sort second list /* bubble sort expansions */ -> double_bubble!(@stage2 [6 5 4] [3 2 1]) // both lists sorted</code></pre> And now that we at last have our two sorted lists, we can emit our expression. The Answer</a> </h3> The last step is to emit an expression to compute the answer. Remember, the goal is to add the absolute difference between each pair between the lists. (@stage2 [$($l:tt)*] [$($r:tt)*]) => { $( { if $l > $r { $l - $r } else { $r - $l} } + )* 0 };</code></pre> That's it! Given an input like [3 2 1] [6 5 4]</code> the macro will expand to an expression: if 3 > 6 { 3 - 6 } else { 6 - 3 } + if 2 > 5 { 2 - 5 } else { 5 - 2 } + if 1 > 4 { 1 - 4 } else { 4 - 1 } + 0</code></pre> We get the absolute difference between each pair and add them up. All done! Except this expression is not evaluated anywhere. Even though I said no const at the start, I will use the const_str::format</a> crate to get constant string formatting and display the result in a compile time panic. const fn _main() -> usize { const answer: usize = day1! { 3 4 4 3 2 5 1 3 3 9 3 3 }; panic!("{}", format!("The answer is {answer}")); }</code></pre>$ cargo run -r Compiling advent-of-code-2024 v0.1.0 (/home/glados/Code/advent-of-code-2024) error[E0080]: evaluation of constant value failed --> src/main.rs:142:5 | 142 | panic!("{}", format!("The answer is {answer}")); | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the evaluated program panicked at 'The answer is 11', src/main.rs:142:5 ... For more information about this error, try `rustc --explain E0080`. error: could not compile `advent-of-code-2024` (bin "advent-of-code-2024") due to 1 previous error</code></pre> Here is the full macro on playground</a>. And now we really are all done. The perfect program complete. The compiler gives us the answer before we even ask the question. No runtime overhead, no bugs, no problems. The perfect program is the program that never was. ^{1 This is sarcasm if you were unsure. Python is great. </div>}^2Inspired by Aurorans Solis' RustConf 2023 talk, "Anything you can do, I can do worse with macro_rules</a>". </div> ^{3 Or it hits a recursion limit. </div> ^{4 Technically a token tree, which can be a group of tokens as well. </div>}}