[Question] 如何正常运行LLaDA1.5，现在出来的都是MASK

Required prerequisites

I have read the documentation https://tilelang.com.
I have searched the Issue Tracker that this hasn't already been reported. (comment there if it has.)

Questions

from diffulex import Diffulex, SamplingParams
from transformers import AutoTokenizer



# Initialize the Diffulex engine
model_path = "/root/models/LLaDA-1.5"
llm = Diffulex(
    model_path,
    model_name="llada",  # or "dream", "llada", etc.
    tensor_parallel_size=1,
    data_parallel_size=1,
    gpu_memory_utilization=0.7,
    max_model_len=2048,
    decoding_strategy="d2f",  # or "d2f", "fast_dllm"
    mask_token_id=126336,  # model-specific mask token ID
enforce_eager = True,
kv_cache_layout= "unified",
use_lora = False,
    add_block_threshold=0.1,
    semi_complete_threshold=0.9,
decoding_threshold=0.95,
block_size=32,
buffer_size=4,
)

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

# Set sampling parameters
sampling_params = SamplingParams(
    temperature=0.0,
    max_tokens=128,
)



# Prepare prompts
prompts = [
    "Question: What is the capital of France? Answer:",
    "Question: Explain quantum computing in simple terms. Answer:",
]

# Generate responses
outputs = llm.generate(prompts, sampling_params)
#print(outputs.convert_to_text(tokenizer))
print(outputs.to_benchmark_format())
# Process results
#for output in outputs:
#    print(f"Generated text: {output['text']}")
#    print(f"Number of diffusion steps: {output['n_diff_steps']}")
#    print(f"Token IDs: {output['token_ids']}")

如果按照Readme来[rank0]: TypeError: 'GenerationOutputs' object is not iterable

print(outputs.to_benchmark_format()) 可以输出内容，但是

Diffulex Generating: 100%|███████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:11<00:00,  5.89s/it, tpf=1.09tok/step, ttft=1.12s, tpot=0.04s, ptps=114.47tok/s, dtps=52.07tok/s, tps=47.28tok/s]
[08:57:57] INFO     --------------------------------
           INFO     Generation Outputs Summary:
           INFO     --------------------------------
           INFO     Total Tokens: 512 toks
           INFO     Total NFEs: 254 nfes (steps)
           INFO     Total Time: 11.718297283630818 sec
           INFO     TPF: 1.09 tok/step
           INFO     TTFT: 1.12 sec
           INFO     TPOT: 0.04 sec
           INFO     Throughput: 47.28 tok/sec
           INFO     Prefill Throughput: 114.47 tok/sec
           INFO     Decode Throughput: 52.07 tok/sec
           INFO     --------------------------------
None
[{'text': '', 'full_text': '<|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|>', 'token_ids': [126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463], 'nfe': 254}, {'text': '', 'full_text': '<|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|><|mdm_mask|>', 'token_ids': [126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463, 126463], 'nfe': 254}]

结果都是MASK的空值，我已经根据benchmark yaml改了llada的配置参数，请问能否提供llada的运行配置。

[rank0]: │ /root/models/Diffulex/diffulex/model/llada.py:195 in __init__                                                                                                                                                                            │
[rank0]: │                                                                                                                                                                                                                                          │
[rank0]: │   192 │   ) -> None:                                                                                                                                                                                                                     │
[rank0]: │   193 │   │   super().__init__()                                                                                                                                                                                                         │
[rank0]: │   194 │   │   self.config = config                                                                                                                                                                                                       │
[rank0]: │ ❱ 195 │   │   self.transformer = nn.Moduledict(                                                                                                                                                                                          │
[rank0]: │   196 │   │   │   dict(                                                                                                                                                                                                                  │
[rank0]: │   197 │   │   │   │   wte=VocabParallelEmbedding(config.embedding_size or config.vocab_size, config.d_model),                                                                                                                            │
[rank0]: │   198 │   │   │   │   emb_drop=nn.Dropout(config.embedding_dropout),        
[rank0]: AttributeError: module 'torch.nn' has no attribute 'Moduledict'

你们是不是自己都没跑过代码，这个Moduledict, ModuleDict大小写写错了跑不通，自己没测试过吗？

@WuNein

@WuNein 感谢您的关注与反馈。

目前我们尚未正式支持 LLaDA 1.5。主要原因是，Diffulex 当前针对的是 block-wise dLLM 路线，暂不支持 full-attention dLLM。现阶段代码中仅保留了对 D2F-LLaDA-Instruct 的早期适配，但这一部分后续没有作为主测模型持续维护，因此目前不能保证其在最新代码上的可用性与稳定性。

就我们已经做过的完整测试而言，目前覆盖最充分的是 SDAR 系列：

buffer size：1 ~ 4
block size：4 / 8 / 16 / 32
对应 page size：4 / 8 / 16 / 32

我们优先完善 SDAR 支持，是因为该系列覆盖了多种块大小组合，能够帮助我们更系统地验证和优化整个运行时状态机，尤其是：

不同 page size 的兼容性
page size ≠ block size 时的行为正确性

由于系统仍在快速开发和重构过程中，我们暂时无法保证每次优化后都对所有模型完成完整回归测试；相关 CI 和自动化测试流程也仍在持续完善中。这一点确实会带来部分旧适配路径失效或行为变化的问题，我们也在持续改进。

接下来我们会尽快重新测试并整理以下模型的支持状态：

D2F-LLaDA
D2F-Dream
Fast-dLLM-v2

同时，我们也正在推进对以下模型的支持：

LLaDA 2.0
LLaDA 2.1
LLaDA-DMax

感谢大家的理解与关注，我们后续会在文档和测试覆盖范围上补充得更清楚，尽量减少当前支持状态与实际可用性之间的信息落差。

@WuNein

@WuNein BTW，若您需要跑 LLaDA 1.5，可以尝试用 D2F 的代码来做蒸馏训练，训出 Block Causal 能力，然后用 D2F 策略来跑。

@drewjin

@drewjin 好的，不过麻烦readme里面 https://github.com/SJTU-DENG-Lab/Diffulex#:~:text=model_name%3D%22fast_dllm_v2%22%2C%20%20%23%20or%20%22dream%22%2C%20%22llada%22%2C%20etc. 写一下
普遍有Non-Block不支持的情况

@WuNein

@WuNein 再次感谢反馈，我们目前正在加速完善文档、CI、SKILL 等的支持工作。

目前的支持应该比较完善了

Navigation Menu

Repository navigation

[Question] 如何正常运行LLaDA1.5，现在出来的都是MASK #39

Required prerequisites

Questions

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Question] 如何正常运行LLaDA1.5，现在出来的都是MASK #39

Description

Required prerequisites

Questions

Activity

WuNein commented on Apr 14, 2026

drewjin commented on Apr 15, 2026

drewjin commented on Apr 15, 2026

WuNein commented on Apr 16, 2026

drewjin commented on Apr 16, 2026

drewjin commented on Jun 26, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions