The Mobile Unified Multimodal Understanding (MUMU) Challenge 2026 is a benchmark competition designed to advance research on efficient unified multimodal architectures for mobile-oriented visual understanding.

Unlike existing task-specific benchmarks, this challenge requires participants to design a single unified model that simultaneously performs semantic tagging, open-vocabulary object detection, and image caption generation under constrained computational budgets.

The challenge emphasizes algorithmic efficiency, architectural unification, and deployability on resource-constrained environments, while evaluation is conducted on PC-side compute statistics rather than requiring on-device execution.

• Promote unified vision-language modeling
• Encourage efficient multi-task representation learning
• Benchmark trade-offs between performance and complexity
• Advance mobile-friendly multimodal perception systems

Data splits include a training set of approximately 1.2M images, a 20,000-image validation set, a 20,000-image public test set, and a 30,000-image private final test set. Novel detection categories are hidden in private evaluation.

• Official Baseline Code: Microsoft dstoolkit-finetuning-florence-2
• Official Pre-trained Model: Microsoft Florence-2-base

Model Constraints

• Single unified model only; Adapter / LoRA is allowed within the unified architecture.
• No task-specific independent models. All task heads must be integrated within the unified model.
• Total Parameter Count ≤ 0.5B.
• Peak Inference Memory ≤ 8GB (PC-side)

Evaluation Metrics

• Task A: Macro-F1
• Task B: Novel-aware mAP@[0.5:0.95]
• Task C: 0.4 CIDEr + 0.3 SPICE + 0.3 CLIPScore
• Final Score = (0.3A + 0.4B + 0.3C) × E
• Efficiency factor: E = 1 / (1 + alpha * PeakMemory + beta * Params)

Ranking Rules

• Primary ranking: Final Score
• Tie-break 1: Lower peak inference memory
• Tie-break 2: Lower parameter count
• Tie-break 3: Higher detection performance

Submissions are evaluated on Codabench. Upload a single ZIP archive containing both the predictions used for automatic scoring and the reproducible model package used for manual verification. All files must sit at the ZIP root exactly as shown below — do not wrap them in an outer directory.

The scoring program reads only predictions.json. It checks that the reproducibility files exist, but it never imports or executes participant code.

submission.zip
predictions.json         # scored automatically
model/
  weights.pt             # .pth, .bin, .safetensors or sharded weights
  config.yaml
inference.py
requirements.txt
README.md

Before your first upload

Email your team name, Codabench username and affiliation to changliu73@outlook.com. Registrations are approved manually, and these three fields must match the values you enter in the Codabench fact sheet when you upload. No submission file repeats them.

predictions.json

{
  "schema_version": "1.0",
  "model_info": {
    "parameters_m": 120.0,
    "gflops_224": 18.5,
    "peak_memory_gb": 4.2
  },
  "predictions": [
    {
      "image_id": "image_filename_without_extension",
      "tags": {
        "quality": ["quality_good"],
        "scene": ["indoor", "office"],
        "event": ["conference"]
      },
      "detections": [
        { "label": "person", "score": 0.9321, "bbox_xyxy": [15.8, 47.1, 211.6, 426.9] }
      ],
      "caption": "A person speaks into a microphone during an indoor conference."
    }
  ]
}

• image_id is the image filename stem. Every test image must appear exactly once; missing or unknown IDs are rejected.
• tags must contain all three namespaces — quality, scene, event — each a list drawn from the published Task A vocabulary. Empty lists are allowed; unknown labels and duplicates are rejected.
• detections: label is free-form lowercase text, score is finite and in [0, 1], and bbox_xyxy gives absolute pixel coordinates [x1, y1, x2, y2] in the original image. At most 300 detections per image.
• caption must be non-empty, a single line, English, at most 300 characters and at most 30 evaluator tokens.

Evaluator tokens are counted after NFKC normalisation and lowercasing: Unicode word characters, with internal apostrophes and hyphens kept inside a single token, and standalone punctuation dropped. Accented and non-Latin letters count as ordinary letters, so João is one token.

model_info

The model_info block at the top of predictions.json declares your model's resource usage. These three numbers are the only model metadata a submission carries — nothing else has to be filled in by hand.

• parameters_m — total parameter count in millions. Enters the efficiency factor E.
• gflops_224 — GFLOPs for one 224×224 forward pass. Enters the efficiency factor E.
• peak_memory_gb — peak inference memory. A hard eligibility limit; it does not enter the score.

Declared values above the published model constraints are rejected automatically; accepted values remain subject to organizer verification.

Reproducibility package

A unified predictor API is required. inference.py should initialize from a weight path and return tags, detections and a caption for each input image in a single inference call. Top Teams are reproduced and inspected manually: a submission is disqualified if the predictions cannot be reproduced from the included checkpoint within the announced tolerance, if the implementation uses independent task-specific models, or if any resource constraint is exceeded.

Prize Distribution

• 1st Place Overall: 6,000 CNY
• 2nd Place Overall: 3,000 CNY
• 3rd Place Overall: 1,000 CNY

Additional Awards

• Best Efficiency Award
• Best Unified Architecture Award
• Best Student Team Award