Automatic1111 for Apple metal, 40% speed up sd1.5

I use Draw Things a lot on Apple hardware, and one thing has always bothered me about Automatic1111: it feels slower than it should.

Not unusably slow. Just slow enough that you notice it.

On my M3 Pro, a short five-step DPM++ SDE generation in Automatic1111 was typically landing somewhere around 8–10 seconds. Draw Things had already shown me that Stable Diffusion on Apple Silicon could feel much more immediate than that.

So I wanted to see how much of that gap was actually necessary.

There was one important constraint: I did not want to replace Automatic1111.

I wanted the same WebUI, checkpoints, LoRAs, samplers, extensions, API, prompt syntax, and general workflow. I wasn't interested in converting everything to Core ML and building another inference engine around it. The goal was much narrower:

How fast can Automatic1111 get if we make the parts that matter behave more like native Apple software?

The answer, at least for the workloads I'm running, is quite a bit faster.

The same class of generation that was taking roughly 8–10 seconds on my M3 Pro is now generally landing between 3 and 7 seconds. And 13-20, now landing 8-10 on my M1 Mac Mini.

Those are observed ranges across my current workloads, not a controlled benchmark claiming a universal 2x improvement. There is also an important distinction between the runtime improvements and NGMS, which actually reduces the amount of guidance work being performed.

Still, the difference in actual use is substantial.

More interesting than the final number, though, was what it took to get there.

It wasn't one optimization.

Start with the workload, not the benchmark

The workload I cared about was pretty specific:

• Stable Diffusion 1.x
• DPM++ SDE
• Karras
• 5 steps
• CFG around 1.15
• 384×640 and 512×512
• FP16 UNet on MPS
• FP32 VAE by default

That specificity matters.

Early on, DPM++ 2M looked like an easy way to shave off time. It was faster, but it didn't produce the result I wanted from the short schedule.

That isn't an optimization. It's a different workload.

This became the rule for basically everything that followed: if an optimization looks great in isolation but doesn't make the actual generation faster while preserving the result I'm trying to produce, it doesn't count.

Attention was the obvious place to start.

PyTorch's MPS backend has gotten substantially better, but there are still Stable Diffusion attention shapes where going directly to Metal makes sense.

The mistake would have been treating a custom Metal implementation as universally faster.

It isn't.

Instead, I added a Metal Flash Attention path specifically for the SD 1.x shapes where it actually won in testing.

The router looks conceptually like this:

if inference and fp16_mps and query_tokens >= 192 and head_dim in (40, 80, 160):
    return metal_flash_attention(q, k, v)

return pytorch_sdpa(q, k, v)

There are additional checks around masks, training, dropout, tensor layout, grouped-query attention, and supported types, but that's the basic idea.

Metal is not the default because Metal sounds faster. It gets the operation when we've measured that shape and it deserves it.

Everything else goes back through PyTorch.

That fallback is important. Automatic1111 supports far more configurations than my five-step SD 1.x workflow. I didn't want a faster fork that only worked if nobody touched anything.

The kernel wasn't the whole problem

Getting attention into Metal helped, but it exposed something more interesting.

The native extension was committing the MPS command buffer after every attention call.

Stable Diffusion calls attention over and over inside every UNet evaluation. With a short five-step generation, repeatedly submitting tiny chunks of work starts becoming a meaningful part of the total runtime.

So instead of treating the Metal kernel like its own little application, I integrated it into PyTorch's current MPS stream.

The extension ends PyTorch's current kernel coalescing, encodes the Metal Flash Attention operation into the current command buffer, and then lets the rest of the PyTorch MPS work continue from there.

The explicit commit after every attention call went away.

This ended up being one of the more important lessons from the entire project.

The fastest kernel still loses if you submit the command buffer after every call.

At these generation times, overhead matters. You're no longer just optimizing how quickly the GPU can multiply matrices. You're optimizing how often Python, PyTorch, MPSGraph, and Metal have to coordinate with each other.

There was also a wonderfully obvious reminder not to trust the timer: one of the early versions produced a green image.

It was fast.

It was also green.

The Metal path now runs an isolated attention-plus-projection correctness test before the WebUI enables it.

Unified memory changes the rules

The next problem was memory.

Apple Silicon doesn't have a discrete pile of VRAM sitting next to system RAM. The GPU and the rest of the machine are competing for the same physical memory.

That makes some traditional GPU assumptions fairly bad ones.

An attention matrix can technically fit in memory and still be a terrible idea if macOS is under pressure, the allocator starts thrashing, or the machine begins swapping.

So instead of using a fixed VRAM threshold, the fork estimates the cost of native attention against both total and currently available memory.

Conceptually:

attention_bytes =
    batch × heads × query_tokens × key_tokens × element_size

estimated_peak = attention_bytes × 2.5

budget =
    min(
        10% of total memory,
        20% of currently available memory,
        1.5 GiB
    )

If the estimated peak fits inside that budget, native SDPA can run.

If it doesn't, the request goes through the memory-bounded sub-quadratic path instead.

The chunk size for that fallback is dynamic too. An 8 GB Mac shouldn't make the same decision as a 32 GB Mac, and neither should behave as though Chrome, Xcode, or whatever else is running doesn't exist.

I don't count this as a blanket speed improvement. It's mostly about keeping performance predictable and avoiding the cases where an ostensibly fast operation causes enough memory pressure to make the whole generation slower.

Stop keeping every attention chunk around

I also changed how the sub-quadratic fallback handles K/V chunks.

The existing approach computes partial attention results, keeps the numerator, normalization weight, and maximum for each chunk, then stacks everything together at the end.

That's unnecessary.

Instead, the fork maintains a running maximum, normalization sum, and weighted output. Each new K/V chunk gets merged into that running state and can then be discarded.

The recurrence is basically:

new_max = max(running_max, chunk_max)

running_scale = exp(running_max - new_max)
chunk_scale = exp(chunk_max - new_max)

running_values =
    running_values × running_scale +
    chunk_values × chunk_scale

running_weights =
    running_weights × running_scale +
    chunk_weights × chunk_scale

This is the same general online-softmax idea that makes Flash Attention memory efficient.

Memory now scales around the current chunk instead of accumulating every partial result until the end.

I tested the forward results against PyTorch SDPA and also tested gradients in float64. Again, the goal wasn't just to make something clever. It had to be a safe fallback.

Some MPS workarounds have outlived the bugs

There was another category of optimization that was much less glamorous: deleting old workarounds.

Apple's PyTorch backend has changed a lot.

Automatic1111 accumulated defensive behavior for older MPS implementations, including cloning torch.narrow() results and pushing LayerNorm through FP32.

Those fixes made sense when the underlying MPS bugs existed. On newer versions of PyTorch, they can just become copies, allocations, conversions, and memory traffic.

So those behaviors are now gated by runtime version rather than applied indiscriminately.

There's still an A1111_MPS_FORCE_LEGACY_OPS=1 escape hatch if somebody needs the old behavior.

I also enabled PYTORCH_MPS_PREFER_METAL=1 because direct Metal matrix multiplication tested better for the SD 1.x projection sizes I was targeting, and removed the default sampling upcast so more of the short sampling path stays in FP16.

That last change is a real tradeoff. FP16 reduction order and removing the upcast can affect same-seed output.

I'm fine with that for this workflow, but it shouldn't be presented as free performance.

Fusing GroupNorm and SiLU

Once the unnecessary work was reduced, I went looking for operations that were both necessary and repeated constantly.

GroupNorm followed by SiLU is everywhere in the SD 1.x UNet.

Normally those are separate PyTorch operations. That means separate dispatches and an intermediate activation that gets written out and then immediately read back.

So I wrote a fused Metal kernel.

For compatible FP16 inference tensors, one 256-thread Metal threadgroup handles each batch/group pair. The kernel accumulates the sum and squared sum in FP32, reduces those into mean and variance, applies normalization and the affine parameters, applies SiLU, and writes the FP16 result.

One dispatch. No intermediate activation.

If the tensor isn't compatible, we're training, gradients are enabled, the dtype is wrong, or the native path fails, it goes straight back to:

F.silu(norm(input_tensor))

I deliberately stopped there.

It was tempting to start fusing entire residual blocks, but GroupNorm plus SiLU was a pair I could isolate, test, and prove.

As it turned out, that restraint mattered.

NGMS is different

There's one part of the final speedup that needs to be separated from the engine work.

NGMS, or Negative Guidance minimum sigma, can skip unconditional guidance during eligible portions of sampling.

With classifier-free guidance, the UNet is often doing conditional and unconditional work together. At a low CFG like 1.15 on a five-step schedule, skipping eligible unconditional work can remove a meaningful amount of computation.

That's obviously fast because the GPU isn't doing some of the work at all.

The fork defaults NGMS to 1.0 with all-steps behavior enabled for this tuned workflow.

But this isn't the same category as making attention or GroupNorm faster.

NGMS changes the denoising calculation. It can change composition and detail, and it's recorded in the PNG metadata when active.

So there are really two performance stories here.

The first is making the existing engine cheaper: Metal attention, fewer command-buffer submissions, better memory behavior, fewer obsolete conversions, and fused operations.

The second is asking the engine to do less work through NGMS.

Any controlled benchmark of this fork needs to show both.

The most useful optimizations were the ones I deleted

A lot of this project was trying things that sounded like they should work and then removing them.

Packed QKV projections were one example.

I implemented them. The resulting image was byte-identical in the test.

Performance went from 8.988 seconds to 9.011 seconds.

That's about 0.26% slower.

Gone.

The more ambitious experiment was moving entire residual blocks into MPSGraph.

The idea looked good on paper: GroupNorm, SiLU, 3×3 convolutions, timestep embedding, residual addition, and the optional skip convolution could all live inside one graph. Existing PyTorch MPS buffers could be bound directly, compiled graphs could be cached by shape, and compatible inference blocks could avoid a pile of individual dispatches.

The microbenchmarks were encouraging too.

Some mid and low-resolution blocks improved by 1–5%. The smallest blocks were as much as roughly 9% faster.

Then I ran the image.

The existing path had a median of 9.5556 seconds.

The MPSGraph version came in at 9.6533.

It was 1.02% slower.

So I deleted it.

That experiment also produced one of the more interesting crashes during development. My first implementation synchronously dispatched onto PyTorch's Metal queue and then called an MPSGraph function that synchronously entered the same queue again.

macOS killed it with:

dispatch_sync called on queue already owned by current thread

After fixing the nested dispatch, the graph worked correctly.

It was still slower.

That distinction is important. Correct code isn't necessarily useful code.

PyTorch's individual MPS convolutions are already pretty good. The graph overhead ate the dispatch savings, particularly because the largest spatial blocks, where most of the actual work happens, didn't improve enough.

Microbenchmarks nominate changes. Full generations elect them.

What actually survived

The resulting performance diff is surprisingly small.

The implementation is four code commits beyond the Automatic1111 dev base I started from and touches 20 of 329 tracked paths.

The pieces that survived were:

• shape-selective Metal Flash Attention
• deferred Metal command-buffer submission
• unified-memory-aware attention routing
• dynamic query chunk sizing
• streaming online softmax
• removal of obsolete MPS copies and FP32 detours
• direct Metal matmul preference
• less sampling upcasting
• fused GroupNorm plus SiLU
• NGMS for the tuned low-CFG workflow

Unsupported inputs still fall back to PyTorch.

The native extensions also self-test at startup. They're built against the active Python/PyTorch environment, validated in a subprocess, and only enabled if the tests pass.

That subprocess is more important than it sounds. Native GPU code doesn't always politely throw a Python exception when something goes wrong. Sometimes it takes the interpreter with it.

I'd rather lose the optimization than lose the WebUI.

The results

On the M3 Pro, the broad workload range that motivated this project moved from roughly 8–10 seconds to 3–7 seconds.

I'm intentionally calling that an observed range rather than a controlled benchmark. It spans workloads, and I don't have enough matched M3 Pro runs yet to pretend every second can be attributed cleanly.

I do have a cleaner development comparison from an M1 Mac mini.

Using the same model hash, tensor shape, sampler, schedule, step count, CFG, dimensions, Clip skip, and NGMS configuration:

That's about 32% lower latency, or roughly 1.47x the generation throughput.

The seeds differed, so I'm treating this as a matched compute-shape throughput comparison rather than an image-parity test. The later fused GroupNorm plus SiLU work also came after this particular comparison.

One other number is worth repeating because it describes the development process better than the winning benchmark does:

I spent time building the second one because the microbenchmarks said it should be faster.

It wasn't.

So it isn't in the fork.

Why this still isn't Draw Things

Draw Things has a fundamental advantage here.

It can own the entire execution environment.

It can design model representation, graph execution, memory lifetime, precision, scheduling, and UI behavior around Apple hardware.

Automatic1111 can't do that without giving up much of what makes Automatic1111 useful.

It's dynamic Python software. People monkey patch it. Extensions hook into it. Models and LoRAs get swapped while it's running. ControlNet gets inserted. Users run different VAEs, different model families, different resolutions, different attention implementations, and all sorts of configurations I haven't thought about.

That's the ecosystem I wanted to keep.

This project instead targets the seams where native Apple execution can enter and leave without requiring Automatic1111 to become a different application.

We've gotten a meaningful amount of performance that way.

There is probably more available, but the next gains get harder.

I want better stage-level timing around prompt encoding, the UNet, VAE decode, and postprocessing. Mixed-precision VAE decoding is interesting. Channels-last layouts across the UNet are interesting. A static whole-UNet MPSGraph is interesting too, but at that point we're getting much closer to maintaining a second execution engine.

That's a different tradeoff.

For now, this is still Automatic1111.

Same checkpoints. Same LoRAs. Same UI. Same extensions. Same general workflow.

It just spends a lot less time waiting at the boundaries between Python, PyTorch, MPSGraph, and Metal.