DoomShift: How Much Adaptation Do World Models Need Under Domain Shift?

Rohan Nagabhirava,  Keerthana Chirumamilla,  Changliu Liu
Carnegie Mellon University
CoRL 2026 Workshops: PhysWM and The Action Gap (TAG)
Paper Code Dataset Models

Rollouts on arena 7, a map the model never saw in training: each starts from 32 true frames and follows the recorded actions for 16 tics (shown from tic 4). The zero-shot SD 1.4 U-Net keeps turning, firing and moving with the actions, but within a few tics replaces the arena's grey walls and brown floor with the dark stone of its training maps. After LoRA adaptation on eight episodes of this map, it keeps the new scene (scene PSNR against the true frame).

Abstract

World models have become capable simulators, but they are evaluated almost exclusively on held-out rollouts from the domain they were trained on. A small domain shift, such as a new map of the same game, lighting, textures or enemies, can make their outputs drift even when the underlying physics and game engine are unchanged. With DoomShift, a controlled version of this shift in Doom, we measure what breaks and how much adaptation is needed to repair it. We train three diffusion backbones, initialized from pretrained diffusion image models, on four Doom maps and evaluate them on 13 unseen maps. All three preserve their response to controls but fail to reproduce the new scene: scene PSNR drops by 2.8–3.3 dB and LPIPS rises by 0.126–0.145 across the three backbones. This consistent gap shows that action-conditioned behavior transfers more reliably than scene appearance, independent of architecture or model scale. Yet the lost appearance is relatively cheap to recover: rank-16 adapters trained on only eight episodes with 2% of the original training updates recover 72–76% of the LPIPS rise and about half of the lost PSNR. Together, these results suggest that adapting a world model to a new environment may require restoring the scene's appearance rather than relearning the underlying game.

A controlled scene shift

Seventeen Doom maps share one physics and game engine and differ only in layout, textures and lighting. We record 2,412 episodes of the Arnold agent playing deathmatches (about 96 hours of play and 12.1M frames at every engine tic), train on 4 maps and hold out 13. Three pretrained diffusion models become next-tic world models under one recipe: the SD 1.4 U-Net (860M), PixArt-α (628M) and SD 3.5 Medium (2.27B), each predicting the next frame from 32 context frames and the newest action.

Method overview: data, models and training, adaptation and decoder fine-tuning, evaluation against the raw frame

(a) Data, (b) models and training, (c) adaptation and decoder fine-tuning, (d) evaluation against the raw frame.

Zero-shot evaluation on unseen maps

Training-map and unseen-map predictions, zero-shot and adapted, against ground truth

(a) On the training maps the prediction is close to the ground truth; (b) on an unseen map it is not, until adapted on eight episodes.

On the 13 unseen maps every backbone keeps its response to the turn control, while scene PSNR drops by 2.8–3.3 dB and LPIPS roughly doubles. Every unseen map has worse LPIPS than every training map.

BackboneScene PSNR (dB) ↑
train → unseen		LPIPS ↓
train → unseen		Turn-response ↑
train → unseen
SD 1.4 U-Net25.20 → 22.300.158 → 0.3030.86 → 0.81
PixArt-α25.18 → 22.380.159 → 0.2850.84 → 0.79
SD 3.5 Medium25.38 → 22.090.126 → 0.2620.84 → 0.81

Medians over maps, one tic ahead, against the raw frame. Turn-response: flip the last turn and count the windows whose predicted horizontal shift reverses direction; 0 for a model that ignores actions and 0.89 for a perfect model, because 11% of recorded turns move the view too little to measure in one tic.

Adaptation cost

Per-map scene PSNR and LPIPS, zero-shot and adapted, and median recovery against adapter updates

(a, b) Scene PSNR and LPIPS per unseen map, zero-shot (open) and after 4k adaptation updates (filled). (c, d) Median statistics against adapter updates.

  • Rank-16 LoRA adapters on the attention layers, trained on 8 episodes of the new map.
  • 4k updates: 2% of the pretraining updates and about 1% of its compute.
  • They recover 72–76% of the LPIPS rise and about half of the lost PSNR, and half of the LPIPS rise within a median of 150–250 updates.
  • Full fine-tuning of the U-Net recovers more (88% vs. 72%) but needs 205× more trainable parameters and forgets 2.7× more on the training maps (1.08 vs. 0.40 dB).

BibTeX

@inproceedings{nagabhirava2026doomshift,
  title     = {DoomShift: How Much Adaptation Do World Models Need Under Domain Shift?},
  author    = {Nagabhirava, Rohan and Chirumamilla, Keerthana and Liu, Changliu},
  booktitle = {CoRL 2026 Workshop on Bringing Physics Simulation and World Models Together for Robotic Manipulation},
  year      = {2026}
}