Questions about HPSv3 scorer correctness, Figure 2 sample quality, and DiffusionNFT baseline

[Shi-wang-MIT]

Hi, thanks for releasing the code. I have a few questions regarding the implementation and the reported qualitative results.

1. HPSv3 scorer implementation appears to miss image normalization

I think there may be a serious correctness issue in the current HPSv3 scorer.

In src/hpsv3_scorer.py, the differentiable preprocessing path calls:

self.ip._preprocess(images01[i:i + 1], do_rescale=False)

However, in the official HPSv3 differentiable image processor, _preprocess() does not resolve do_normalize=None, image_mean=None, and image_std=None to the processor defaults. That resolution only happens in the public preprocess() / preprocess_tensor() path.

As a result, although disabling rescaling is appropriate for an input already in [0, 1], the image does not appear to receive the required CLIP mean/std normalization before being fed into the Qwen2-VL vision encoder.

This seems potentially quite significant: the resulting HPSv3 scores, and especially the image-space reward gradients used for optimization, may not correspond to the official HPSv3 scorer.

Have you checked numerical parity between this implementation and the official

HPSv3RewardInferencer.reward(...)

on exactly the same image/prompt pairs?

It would be helpful if you could provide a simple parity test comparing the raw HPSv3 scores from the released scorer against the official implementation.

2. Figure 2 / teaser image quality

I also have a question about the qualitative results in the paper.

The samples shown in Figure 2 / the main qualitative figure appear substantially higher quality than what I obtain from the released implementation and than some of the other reported qualitative results.

Could you clarify exactly how these images were generated?

In particular, were they generated using exactly the same released checkpoint and inference configuration? It would be useful to provide the corresponding:

• prompts,
• random seeds,
• checkpoints,
• sampling steps,
• CFG/guidance settings,
• resolution, and
• any sample-selection or curation procedure.

This would make the qualitative comparison much easier to reproduce.

3. DiffusionNFT baseline outputs are consistently blurry

Finally, I am having difficulty reproducing a reasonable DiffusionNFT baseline using the released implementation.

The images generated by the provided DiffusionNFT baseline are consistently very blurry / low quality in my runs. This seems unusual enough that I am concerned there may be an implementation or inference-configuration issue with the baseline.

Could you clarify whether you verified this implementation against the original DiffusionNFT implementation?

In particular, could you provide the exact DiffusionNFT training and inference configuration used for the paper, as well as some representative baseline generations? It would also be useful to confirm that DiffusionNFT and DiffusionOPSD are evaluated using the same sampling resolution, number of steps, guidance settings, and other inference hyperparameters.

Thanks — I would appreciate any clarification on these points, especially the HPSv3 preprocessing issue, since that may affect both the reported HPSv3 evaluation numbers and optimization results.

[Shi-wang-MIT]

@ldkong1205 @weichow23 For vis

[weichow23]

Thank you for your question 🌹. I will review the code and relevant materials, and I will get back to you immediately after completing the experiment.

[weichow23]

Thank you for raising this important question🙏. We truly appreciate you pointing out these issues, and we will release an updated version of the paper 🚀.

1. HPSv3 scorer

You are correct about the preprocessing mismatch. Our differentiable scorer calls the private _preprocess(..., do_rescale=False) path of the official HPSv3 implementation, but does not explicitly pass the processor's normalization defaults. Avoiding a second [0,1] rescaling is appropriate; omitting mean/std normalization is not.

Backbone	Methods (in order)	Originally reported HPSv3	Official HPSv3 re-score
SD3.5-M	ReFL / DiffusionNFT / DiffusionOPSD	9.33 / 9.14 / 13.34	13.53 / 11.92 / 14.93
Z-Image-Turbo	FlowGRPO / ReFL / DiffusionNFT / DiffusionOPSD	7.11 / 13.77 / 1.58 / 14.44	11.64 / 15.30 / 8.96 / 15.50

We ran a paired parity check on 112 identical saved image/prompt pairs with the same HPSv3 checkpoint. The mean absolute difference in raw scores was 3.77; for example, SD3.5-M/ReFL image #0000 scored 9.858 with our released path versus 15.930 with HPSv3RewardInferencer.reward(...), while Z-Image-Turbo/DiffusionNFT image #0000 scored -1.112 versus 10.584. We then used the official public reward API to re-score 1,000 saved DrawBench generations for each of the seven HPSv3-specific method/backbone rows. No images were regenerated, and no checkpoints were changed. The resulting means are in the table above. On this evaluation of the existing checkpoints, DiffusionOPSD remains the highest-scoring method on both backbones. The margins change: its lead over ReFL falls from 4.016 to 1.400 on SD3.5-M and from 0.672 to 0.199 on Z-Image-Turbo.

This re-evaluation is not a substitute for correcting training. The same reward path supplied scalar rewards in HPSv3-specific training, and DiffusionOPSD additionally used image-space reward gradients. We will fix the preprocessing call, release a direct numerical parity test, rerun the HPSv3-specific comparisons with the corrected scorer and matched training budgets, and update the paper and released results. The other reward-specific runs do not use the HPSv3 scorer.

2. Figure 2

Figure 2 was assembled from a large pool generated with our released Z-Image-Turbo pointwise LoRA and the Z-Image-Turbo inference path. The relevant released inference entry point is scripts/native_eval.py, which loads the LoRA through --lora and calls the zimage_rollout implementation. The native settings are BF16, 1024 x 1024, nine FlowMatchEuler steps and guidance scale 0, consistent with the Z-Image-Turbo model card and our released inference instructions.

A distinction matters here: scripts/native_eval.py is the reproducible public evaluation entry point and uses fixed per-batch seeds; that fixed-seed evaluation protocol is not a record of Figure 2's exploratory sampling. We did not set or preserve per-image seeds for the qualitative pool. We generated many candidates and manually selected the Figure 2 montage, but did not retain a per-tile mapping to the original prompt, seed and source file. We therefore cannot provide the exact per-tile metadata or recreate the identical montage pixel-for-pixel. We apologize for this missing provenance and will describe the figure explicitly as a curated qualitative illustration in the revision, rather than imply that it is an uncurated random sample. With the released checkpoint and inference code, we expect comparable-quality generations to be reproducible, while the precise Figure 2 composition is not.

3. DiffusionNFT baseline

For SD3.5-M, we ported the official DiffusionNFT implementation and its SD3.5-M configuration. Our released preset uses FP16, 512 x 512 resolution, ten-step DPM-Solver++ 2M training, 48 prompt groups with 24 samples per group, guidance scale 1, and the matched 100-update budget. The held-out SD3.5-M comparison uses 40-step flow sampling at 512 x 512 and guidance scale 1 for all methods; the training implementation and baseline launcher are released.

The official DiffusionNFT repository does not provide a Z-Image-Turbo recipe. For Turbo we kept DiffusionNFT's training objective but adapted the released Z-Image preset and trainer to the model's native regime: BF16, 1024 x 1024, nine-step FlowMatchEuler sampling, guidance 0, and the same 100-update budget. Within each backbone, DiffusionNFT and DiffusionOPSD are evaluated on the same prompts with the same resolution, sampler, step count and guidance. The Z-Image-Turbo qualitative comparison also uses shared initial latents across methods and the complete native schedule (paper, Appendix C.1). Its softness is therefore not a consequence of truncating inference to fewer steps.

Our additional NFT-only six-column contact sheet compares the same fixed prompts across PickScore, HPSv2.1 and VLM-Pointwise checkpoints for each backbone. The SD3.5-M NFT images do not show universal blur, whereas some Z-Image-Turbo NFT settings are noticeably softer. This supports a backbone/recipe-specific concern, but does not establish its mechanism. Because Turbo is a few-step distilled model, we hypothesize that NFT's forward-noising and endpoint-fitting objective may disturb the denoising dynamics calibrated by distillation. We did not extensively tune a new NFT recipe (just refer to the official specifications of both.) for Turbo and cannot rule out a stronger model-specific configuration. We would welcome a verified stronger port, rerun the comparison and update the baseline. We discussed this point in Issue #6.

DiffusionOPSD also outperforms DiffusionNFT based on the official SD3.5-M configuration. Importantly, the SD3.5-M result does not depend on the Z-Image-Turbo adaptation: under the official DiffusionNFT recipe and matched 100-update protocol, DiffusionOPSD outperforms DiffusionNFT in all ten reward-specific comparisons in Table 1