DualWorldBench

What is DualWorldBench?

A benchmark for executable pickup-and-delivery planning.

DualWorldBench evaluates agent planning under controlled symbolic-temporal and grounded-topological complexity. The task interface is shared: an agent receives task specifications and, when applicable, a grounded scene, then outputs a sequential plan composed of move, pickup, and deliver actions.

DualWorldBench task overview with task specification, SimWorld scene, and sequential action plan. — The agent must produce a topology-aware action plan while satisfying pickup-before-delivery, capacity, and task-matching constraints.

Why is this problem important?

Delivery planning exposes two missing capabilities in current benchmarks.

01

Real agents face coupled combinatorial constraints.

A delivery agent must jointly handle task order, rewards, deadlines, pickup-before-delivery precedence, capacity limits, and travel cost. This makes planning difficulty grow at the sequence level rather than as isolated subtasks, but many existing evaluations still use simplified settings with weak temporal and combinatorial pressure.

02

Planning must integrate multi-source evidence.

In realistic scenes, symbolic task descriptions and visual topology are not trivially aligned: two locations close in coordinates may be far apart in reachable distance. Existing benchmarks rarely isolate whether agents can combine perception with structured constraints to produce actionable plans.

What do we do?

We decompose planning into TextWorld, SimWorld, and DualWorld-VQA.

The benchmark is integrated because it uses a shared pickup-and-delivery interface, and decomposable because each component isolates a different source of difficulty.

TextWorld

Procedurally generated symbolic pickup-delivery tasks with task composition, deadlines, rewards, capacity constraints, and online task release. Mode A uses homogeneous Type-A tasks; Mode M mixes Type-A/B/C tasks. L1 is static with non-binding deadlines, while L2 introduces online releases and deadlines.

SimWorld

Grounded scene layouts evaluated under text, image, and multimodal inputs across open, T1, T2, indoor, and campus scene groups. All tasks are available at the start and share a unified reward; navigable distances come from scene graphs.

DualWorld-VQA

1,665 diagnostic questions across 13 scenes probe visual perception, spatial topology, and logical planning under pickup-delivery constraints.

Evaluation Results

Three experimental questions.

Q1 / Q2 / Q4: How do planners scale with combinatorial difficulty?

TextWorld varies task count, task composition, and online release; SimWorld varies task cardinality under grounded scenes.

Q3: Can MLLMs align textual and visual evidence into feasible plans?

SimWorld compares text-only, image-only, and multimodal inputs under the same executable interface.

Q5: Are grounded plans robust across scene topology?

Scene groups test whether agents can transfer from open layouts to corridor, indoor, and campus-scale topology.

TextWorld: symbolic-temporal planning

TextWorld evaluates 14 LLMs and three solver baselines using cumulative reward R and simulated completion time T. Higher R and lower T are better; T is not model inference runtime. Ranks are computed separately for LLMs and solver baselines.

Table 1. TextWorld results under varying task counts.

Method	Rank	Average		n=5		n=8		n=10		n=12		n=15
Method	Rank	R	T	R	T	R	T	R	T	R	T	R	T
Open-source LLMs
Qwen3.6-27B	9	251.41	5061.46	99.33	1969.57	213.17	5552.39	249.12	5377.14	312.94	5158.18	382.49	7250.02
Llama-2-13B	12	212.33	5515.20	100.28	2396.66	209.73	5471.45	202.32	5403.88	298.18	5912.93	251.15	8391.05
Qwen3-VL-8B	13	68.81	3305.46	19.23	1449.30	121.67	4481.61	35.69	2902.33	135.69	4653.41	31.79	3040.67
Llama-3-8B	14	19.33	3223.40	26.14	1640.78	13.94	3526.92	50.24	3609.83	6.33	3237.74	0.00	4101.73
Proprietary LLMs
Gemini-2.5-Flash	1	265.73	4568.84	111.10	1875.90	213.24	5231.71	279.18	4175.72	333.12	4934.41	392.01	6626.44
Gemini-3-Pro	2	264.36	4310.73	114.77	1761.01	213.24	4843.20	278.53	3991.18	323.13	4716.06	392.14	6242.17
GPT-5.6-Sol	3	264.33	5013.85	114.77	2011.85	213.24	5384.76	278.53	4785.25	323.13	5665.16	392.01	7222.23
Claude-Opus-5	4	264.10	4347.24	114.77	1745.07	213.24	5058.77	277.35	4009.72	323.13	4695.04	392.01	6227.59
DeepSeek-v4-Flash	5	263.60	4711.47	111.10	1789.44	213.24	5437.33	278.53	4051.14	323.13	5406.74	392.01	6872.71
Qwen3.7-Plus	6	263.24	4560.71	114.77	1863.30	213.24	5094.50	273.90	4165.28	322.27	5185.82	392.01	6494.65
Claude-Sonnet-5	7	260.16	4780.29	114.54	1782.60	196.24	5408.39	278.43	4275.89	319.52	5634.78	392.05	6799.79
Claude-4.5-Haiku	8	253.01	4900.80	111.10	1927.15	204.95	5333.92	266.26	4818.51	304.06	5532.84	378.70	6891.58
GPT-4o	10	251.05	5245.35	114.77	2226.95	189.94	5808.08	272.77	4805.22	303.95	5662.43	373.82	7724.05
GPT-5.4-Mini	11	246.81	5207.10	111.10	2219.29	198.84	5532.03	248.54	4925.06	321.70	5662.36	353.88	7696.76
Combinatorial Optimization Solver
DFS	3	264.33	4702.45	114.77	2028.71	213.24	5349.95	278.53	4344.86	323.13	5186.65	392.01	6602.05
Online DP	2	264.36	4083.25	114.77	1758.71	213.24	4742.61	278.53	3806.96	323.13	4479.11	392.14	5628.85
Offline DP	1	278.17	4015.28	126.20	1718.67	230.27	4602.62	290.00	3794.25	334.65	4406.57	409.71	5554.29

Table 2. TextWorld results at task count 15.

Method	Rank	Average		A-L1		M-L1		A-L2		M-L2
Method	Rank	R	T	R	T	R	T	R	T	R	T
Open-source LLMs
Qwen3.6-27B	8	382.49	7250.02	302.97	5291.29	558.94	5612.23	251.02	10166.85	417.01	7929.71
Llama-2-13B	12	251.15	8391.05	302.97	6165.02	558.94	6820.17	142.68	10580.02	0.00	9999.00
Qwen3-VL-8B	13	31.79	3040.67	10.04	453.02	60.08	455.02	10.04	4102.59	47.00	7152.04
Llama-3-8B	14	0.00	4101.73	0.00	4.00	0.00	4.00	0.00	9246.89	0.00	7152.04
Proprietary LLMs
Gemini-3-Pro	1	392.14	6242.17	302.97	3614.01	558.94	3806.19	289.72	9619.76	416.92	7928.71
Claude-Sonnet-5	2	392.05	6799.79	302.97	5500.38	558.94	4149.32	289.36	9619.76	416.92	7929.71
Claude-Opus-5	3	392.01	6227.59	302.97	4366.61	558.94	3107.06	289.21	9619.76	416.92	7816.91
Qwen3.7-Plus	4	392.01	6494.65	302.97	3684.40	558.94	4745.75	289.21	9619.76	416.92	7928.71
Gemini-2.5-Flash	5	392.01	6626.44	302.97	4307.42	558.94	4649.88	289.21	9619.76	416.92	7928.71
DeepSeek-v4-Flash	6	392.01	6872.71	302.97	4973.04	558.94	5256.33	289.21	9619.76	416.92	7641.72
GPT-5.6-Sol	7	392.01	7222.23	302.97	5093.83	558.94	6359.43	289.21	9619.76	416.92	7815.91
Claude-4.5-Haiku	9	378.70	6891.58	302.97	4877.98	558.94	5535.86	262.07	9619.76	390.80	7532.73
GPT-4o	10	373.82	7724.05	302.97	6020.76	558.94	6346.16	241.98	10273.35	391.38	8255.92
GPT-5.4-Mini	11	353.88	7696.76	302.97	5615.12	558.94	6324.61	255.85	10685.64	297.76	8161.68
Combinatorial optimization solvers
DFS	3	392.01	6602.05	302.97	4187.31	558.94	4903.47	289.21	9619.76	416.92	7697.67
Online DP	2	392.14	5628.85	302.97	2573.66	558.94	2680.27	289.72	9619.76	416.92	7641.72
Offline DP	1	409.71	5554.29	302.97	2573.66	558.94	2680.27	289.72	9529.04	487.22	7434.19

At n=15, Average is the unweighted mean of A-L1, M-L1, A-L2, and M-L2 at seed 1. A delivery that violates Type-C priority receives R=0 and T=9999 before averaging.

At n=15, Qwen3.6-27B earns R=382.49, compared with 31.79 for Qwen3-VL-8B and 0.00 for Llama-3-8B. Online release requires models to revise plans while managing deadlines and heterogeneous priorities.

SimWorld: grounded-topological planning

SimWorld evaluates six VLMs across 13 scenes with text-only, image-only, and multimodal inputs. Feasible (F) is the fraction of plans without execution violations; Completion (C) measures task coverage. Normalized Cost (NC) divides travelled distance by the optimal route length over exactly the delivered task subset. Higher F/C and lower NC are better; NC should be read alongside coverage.

Table 3. SimWorld scaling results under varying task counts.

Method	Modality	Average			n=5			n=8			n=10
Method	Modality	F	C	NC	F	C	NC	F	C	NC	F	C	NC
Qwen3-VL-8B	Multi.	0.989	0.974	1.90	0.984	0.946	1.56	0.984	0.976	1.95	1.000	1.000	2.20
Qwen3-VL-8B	Image	0.836	0.904	1.90	0.778	0.895	1.58	0.857	0.907	1.95	0.873	0.911	2.17
Qwen3-VL-8B	Text	1.000	0.544	1.46	1.000	0.527	1.28	1.000	0.619	1.56	1.000	0.486	1.53
Qwen3.7-Plus	Multi.	1.000	1.000	1.27	1.000	1.000	1.13	1.000	1.000	1.30	1.000	1.000	1.38
Qwen3.7-Plus	Image	0.984	0.995	1.54	0.984	0.997	1.32	0.984	0.996	1.56	0.984	0.992	1.74
Qwen3.7-Plus	Text	1.000	0.972	1.26	1.000	1.000	1.12	1.000	1.000	1.23	1.000	0.916	1.42
Claude-Haiku-4.5	Multi.	0.762	0.961	2.16	0.921	0.987	1.57	0.698	0.964	2.49	0.667	0.930	2.43
Claude-Haiku-4.5	Image	0.175	0.707	3.19	0.270	0.752	2.69	0.159	0.708	3.48	0.095	0.660	3.40
Claude-Haiku-4.5	Text	0.905	0.981	1.93	1.000	1.000	1.49	0.810	0.972	1.95	0.905	0.971	2.35
Claude-Opus-5	Multi.	0.519	0.945	1.74	0.730	0.968	1.36	0.429	0.933	1.78	0.397	0.935	2.08
Claude-Opus-5	Image	0.561	0.865	1.45	0.635	0.902	1.30	0.571	0.859	1.46	0.476	0.835	1.59
Claude-Opus-5	Text	0.698	0.971	1.38	0.857	0.994	1.23	0.683	0.972	1.38	0.556	0.948	1.52
GPT-5.4-mini	Multi.	0.640	0.934	1.81	0.778	0.952	1.52	0.698	0.948	1.88	0.444	0.900	2.04
GPT-5.4-mini	Image	0.127	0.695	2.03	0.143	0.698	1.95	0.175	0.742	1.98	0.063	0.644	2.17
GPT-5.4-mini	Text	0.995	0.966	1.78	1.000	1.000	1.55	0.984	0.998	1.85	1.000	0.900	1.95
GPT-5.6-Sol	Multi.	0.984	0.998	1.13	0.984	0.997	1.08	0.984	1.000	1.16	0.984	0.997	1.17
GPT-5.6-Sol	Image	0.989	0.996	1.39	0.984	0.994	1.25	1.000	1.000	1.39	0.984	0.995	1.53
GPT-5.6-Sol	Text	0.968	0.995	1.13	0.968	0.994	1.07	0.937	0.992	1.13	1.000	0.998	1.20

Table 4. SimWorld scene-wise results across five scene groups.

Method	Modality	Open			T1			T2			Indoor			Campus
Method	Modality	F	C	NC	F	C	NC	F	C	NC	F	C	NC	F	C	NC
Qwen3-VL-8B	Multi.	1.000	0.783	1.66	0.973	0.979	1.91	1.000	0.984	1.91	1.000	1.000	1.73	1.000	0.987	2.16
Qwen3-VL-8B	Image	0.889	0.950	1.92	0.880	0.915	1.91	0.773	0.885	1.91	0.733	0.905	1.76	1.000	0.920	1.97
Qwen3-VL-8B	Text	1.000	0.853	1.83	1.000	0.462	1.37	1.000	0.533	1.43	1.000	0.808	1.64	1.000	0.557	1.60
Qwen3.7-Plus	Multi.	1.000	1.000	1.24	1.000	1.000	1.23	1.000	1.000	1.30	1.000	1.000	1.29	1.000	1.000	1.29
Qwen3.7-Plus	Image	0.667	0.894	1.20	1.000	1.000	1.57	1.000	1.000	1.57	1.000	1.000	1.35	1.000	1.000	1.60
Qwen3.7-Plus	Text	1.000	1.000	1.20	1.000	0.968	1.28	1.000	0.977	1.23	1.000	0.920	1.23	1.000	1.000	1.28
Claude-Haiku-4.5	Multi.	0.333	0.908	2.68	0.640	0.947	2.36	0.880	0.971	1.97	0.867	0.977	2.06	0.933	0.993	1.92
Claude-Haiku-4.5	Image	0.000	0.653	2.12	0.133	0.679	2.85	0.027	0.693	4.10	0.533	0.760	2.05	0.867	0.900	2.08
Claude-Haiku-4.5	Text	0.889	0.931	2.04	0.907	0.985	1.72	0.933	0.984	2.03	0.867	0.977	1.92	0.800	0.983	2.41
Claude-Opus-5	Multi.	0.778	0.956	1.20	0.520	0.926	1.49	0.440	0.966	2.18	0.600	0.930	1.47	0.667	0.950	1.34
Claude-Opus-5	Image	0.111	0.678	1.33	0.560	0.878	1.45	0.640	0.893	1.44	0.467	0.785	1.57	0.533	0.857	1.46
Claude-Opus-5	Text	0.667	0.944	1.30	0.667	0.972	1.39	0.707	0.964	1.32	0.800	0.993	1.37	0.733	0.993	1.64
GPT-5.4-mini	Multi.	0.556	0.897	1.59	0.573	0.917	1.71	0.680	0.945	1.94	0.533	0.925	1.71	0.933	0.993	1.90
GPT-5.4-mini	Image	0.111	0.747	1.73	0.093	0.684	2.10	0.120	0.699	2.03	0.200	0.687	1.96	0.267	0.705	1.93
GPT-5.4-mini	Text	1.000	0.975	1.77	1.000	0.969	1.79	1.000	0.964	1.78	0.933	0.953	1.66	1.000	0.967	1.89
GPT-5.6-Sol	Multi.	1.000	1.000	1.08	0.973	0.996	1.12	0.987	0.999	1.13	1.000	1.000	1.19	1.000	1.000	1.19
GPT-5.6-Sol	Image	0.778	0.956	1.11	1.000	1.000	1.36	1.000	1.000	1.43	1.000	0.987	1.36	1.000	0.993	1.54
GPT-5.6-Sol	Text	1.000	1.000	1.09	0.933	0.988	1.13	0.987	0.998	1.11	1.000	1.000	1.20	1.000	1.000	1.21

Qwen3.7-Plus achieves F=C=1.000 with multimodal input at every task count. GPT-5.6-Sol keeps multimodal F=0.984 from n=5 to n=10, while NC rises from 1.08 to 1.17. GPT-5.4-mini drops from F=0.778 to 0.444 over the same range.

Performance across the two worlds

TextWorld comparison at task count 15 across four symbolic regimes. — TextWorld at n=15. Reward is normalized as R / max R and completion time as min T / T; higher radar scores are better.

SimWorld feasibility and normalized route cost across scene groups. — SimWorld scores are averaged across the three modalities; NC is inversely normalized within each scene. Scene effects depend on visual grounding capability.

DualWorld-VQA: diagnostic examples and results

VQA items are organized into three layers. Perception-level questions test visual geometric relations; topology-level questions introduce wall constraints and valid walking routes; planning-level questions require grounded pickup-delivery decisions under capacity and precedence constraints.

Layer 1

Visual Perception

Example: order START, D1, D2, P1, P2 from left to right.

Layer 2

Spatial Topology

Example: decide whether path A->B or A->C is longer when walls block straight-line shortcuts.

Layer 3

Logical Planning

Example: find the visit order that minimizes total travel distance under capacity constraints.

Perception contains 185 axis-sorting, 185 anchor-choice, and 185 pair-choice questions. Topology and planning each contain 555 questions. Planning success requires a feasible route within a relative distance tolerance of the reference.

Table 5. DualWorld-VQA accuracy by question family.

Model	Overall ↑	Axis ↑	Anchor ↑	Pair ↑	Topology ↑	Planning ↑
Qwen3-VL-8B	0.335	0.162	0.622	0.535	0.566	0.000
Qwen3.7-Plus	0.843	1.000	1.000	1.000	0.793	0.737
Claude Haiku 4.5	0.227	0.005	0.232	0.162	0.530	0.018
Claude Opus 5	0.799	1.000	1.000	0.995	0.778	0.622
GPT-5.4 Mini	0.392	0.422	0.589	0.654	0.559	0.063
GPT-5.6-Sol	0.864	0.968	0.989	0.995	0.775	0.832

Accuracies are fractions from 0 to 1, matching the appendix. The figure reports percentages and combines Axis, Anchor, and Pair into Visual Perception.

Failure attribution across the two worlds

TextWorld decomposes the reward gap into undelivered tasks, lateness, and priority violations. SimWorld assigns one root cause per run in the order grounding, capacity, state tracking, then precedence; violation-free runs are complete or incomplete.

TextWorld earned reward and reward-loss composition by model. — Stronger models earn over 92% of attainable reward. Lateness vanishes in L1; only one of 280 runs violates Type-C priority.

SimWorld root-cause shares pooled across input modalities for six models. — Precedence and state-tracking errors dominate invalid plans. Grounding and capacity errors are uncommon; modality effects vary by model.

Table 6. TextWorld reward-gap attribution.

Model	Average R ↑	Earned % ↑	Not delivered % ↓	Late % ↓	Late L2 % ↓	Priority % ↓
Gemini-2.5-Flash	265.73	93.4	0.0	6.6	13.5	0.0
Gemini-3-Pro	264.36	92.9	0.0	7.1	14.5	0.0
GPT-5.6-Sol	264.33	92.9	0.0	7.1	14.5	0.0
Claude-Opus-5	264.10	92.8	0.0	7.2	14.7	0.0
DeepSeek-v4-Flash	263.60	92.6	2.2	5.1	10.5	0.0
Qwen3.7-Plus	263.24	92.5	0.0	7.5	15.3	0.0
Claude-Sonnet-5	260.16	91.4	0.0	8.6	17.5	0.0
Claude-4.5-Haiku	253.01	88.9	0.0	11.1	22.7	0.0
Qwen3.6-27B	251.41	88.4	3.1	8.6	17.6	0.0
GPT-4o	251.05	88.2	0.0	11.8	24.1	0.0
GPT-5.4-Mini	246.81	86.7	0.0	13.3	27.1	0.0
Llama-2-13B	212.33	74.6	0.2	15.8	32.3	9.4
Qwen3-VL-8B	68.81	24.2	69.5	6.3	13.0	0.0
Llama-3-8B	19.33	6.8	83.6	9.6	19.6	0.0

Pooled over four scenarios and five task counts at seed 1. Earned + Not delivered + Late + Priority = 100% before rounding. Late L2 is the within-L2 share; Late L1 is zero for every model.

Table 7. SimWorld failure attribution by model and modality (% of runs).

Modality	Model	Precedence	State tracking	Capacity	Grounding	Incomplete	Success
Text-only	GPT-5.6-Sol	3.2	0.0	0.0	0.0	0.5	96.3
Text-only	GPT-5.4-mini	0.0	0.5	0.0	0.0	5.3	94.2
Text-only	Claude-Opus-5	16.4	13.2	0.5	0.0	0.0	69.8
Text-only	Claude-Haiku-4.5	5.8	3.2	0.0	0.5	7.4	83.1
Text-only	Qwen3.7-Plus	0.0	0.0	0.0	0.0	5.3	94.7
Text-only	Qwen3-VL-8B	0.0	0.0	0.0	0.0	67.7	32.3
Multimodal	GPT-5.6-Sol	1.1	0.5	0.0	0.0	0.5	97.9
Multimodal	GPT-5.4-mini	24.3	10.6	0.0	1.1	2.6	61.4
Multimodal	Claude-Opus-5	20.6	23.8	1.6	2.1	0.5	51.3
Multimodal	Claude-Haiku-4.5	15.3	7.4	0.5	0.5	1.1	75.1
Multimodal	Qwen3.7-Plus	0.0	0.0	0.0	0.0	0.0	100.0
Multimodal	Qwen3-VL-8B	1.1	0.0	0.0	0.0	4.8	94.2
Image-only	GPT-5.6-Sol	0.5	0.0	0.5	0.0	1.1	97.9
Image-only	GPT-5.4-mini	73.0	11.1	2.1	1.1	1.1	11.6
Image-only	Claude-Opus-5	34.9	5.8	2.6	0.5	0.0	56.1
Image-only	Claude-Haiku-4.5	63.5	15.9	1.1	2.1	5.3	12.2
Image-only	Qwen3.7-Plus	0.5	0.0	1.1	0.0	0.0	98.4
Image-only	Qwen3-VL-8B	16.4	0.0	0.0	0.0	27.5	56.1

189 runs per model and modality. Each run has one root cause, so rows sum to approximately 100% after rounding. Success requires both a legal plan and full task completion.

Key Findings

Scheduling, action consistency, and route optimization remain challenging.

01

High reward still leaves room for better routes.

Gemini-2.5-Flash leads LLMs in TextWorld with R=265.73 and T=4568.84. Offline DP reaches R=278.17 with T=4015.28.

02

Coverage gives way to scheduling.

Stronger models earn over 92% of attainable reward; lateness causes most residual loss. Weaker models mainly leave tasks undelivered.

03

Modality effects depend on the model.

Qwen3.7-Plus reaches F=C=1 with multimodal input. GPT-5.6-Sol remains reliable across modalities, while weaker models struggle with image-only planning.

04

Topology interacts with visual grounding.

Scene-dependent feasibility varies most for weaker visual planners. Stronger models remain reliable across most layouts; larger task sets still increase route cost.

05

VQA distinguishes several failure modes.

Planning accuracy spans 0.0% to 83.2%. Wrong task-node sets, constraint violations, and suboptimal routes require different improvements.

The current benchmark evaluates temporal and topological complexity separately. Combining both within shared instances is a direction for future work.

Citation

Cite DualWorldBench.

@misc{chen2026dualworldbench,
  title  = {DualWorldBench: Can Agents Plan Deliveries across Symbolic and Grounded Worlds?},
  author = {Chen, Yang and Wen, Bin and Peng, Danyang and You, Hong-Jie and Shang, Zi-Qiao and Guo, Lan-Zhe},
  year   = {2026},
  note   = {Camera-ready version}
}