FLIRT-style signature libraries for NeverD.
<format>/<arch>/<bitness>/*.pat
Supported formats: elf, pe, macho. Architectures: x86 and arm, each
at 32 or 64 bits, so Windows ARM64 images use pe/arm/64 and Windows
ARM32 (Thumb-2) images use pe/arm/32.
NeverD copies this tree to build/bin/signatures/ at build time. Use
neverd sigs --auto to apply the set that matches the loaded binary. It reads
the .pat files of the matching directory and ignores other files, such as the
<name>.sources.json provenance records that sit next to generated files. For a PE image whose Rich header names the Visual
Studio release of its linker, it reads only that release's vs<year>.pat and
the files that belong to no release (winsdk.pat, masm32.pat,
mingw32-zlib.pat). A static runtime library comes from the linker's own
release, and older releases state some of the same bytes under other names.
Without a usable Rich header it reads every file of the directory.
A file larger than 50 MB, GitHub's recommended limit, is written in parts of
even size: <name>.pat, <name>.part2.pat and so on. NeverD reads the parts
as one library, and the Rich header chooses a release's parts together.
<leading bytes> <crc length> <crc16> <function length> :0000 <name> [^<offset> <name>]... [<tail bytes>]
Each line states the bytes of one library function and gives it its names at
offset 0: one, or every symbol an ELF or Mach-O library gives the routine
(glibc's puts is also _IO_puts), of which NeverD shows the one with the fewest
leading underscores, then the shortest:
- The leading bytes are the first 32 bytes of the function, or all of it when it is shorter.
- The CRC16 covers the bytes after them, up to the first byte a linker may rewrite.
- The tail states the bytes after the CRC span, starting where the CRC span ends.
- Bytes that a relocation rewrites are
..wildcards, and so are the bytes an ELF linker may rewrite around one: the opcode of a GOT load it relaxes to aleaor a direct call, or a whole TLS access sequence in a static link. In Mach-O code they are also themovqopcode of a GOT load ld64 relaxes toleaq, and every arm64 instruction a linker optimization hint names (see macOS (macho/) signatures). - A line states at least 16 bytes exactly. A shorter claim matches far more code than the function it came from.
- Each
^<offset> <name>names a routine the function branches to directly: at<offset>starts the rel32 field of an x86 or x64callorjmp, or the ARM64B/BLor Thumb-2B.W/BL/BLXinstruction. An ARM-stateB/BL/BLXis stated one byte past its instruction, at an odd offset that no Thumb-2 instruction has, so the offset says which instruction set the branch is in. The name is the symbol the object's relocation names: an undefined symbol, or a function the object defines; a call a linker may rewrite, such as__tls_get_addrin a TLS sequence it relaxes, is not stated. - Several
^<offset>with one offset name the routines that one branch may reach, any of which confirms it: a symbol and the alternate name an/alternatenamedirective of the library gives it, which a COFF link resolves the symbol to where no object defines it (the x86 CRT's__except_handler4calls__filter_x86_sse2_floating_point_exception, and a program without the SSE2 filter reaches__filter_x86_sse2_floating_point_exception_default), or the routines that builds of the same bytes call there (the release CRT'sfreewhere the debug CRT calls_free_dbg). - NeverD follows each branch in the image. A match whose branch reaches a
routine NeverD names as none of the routines the branch may reach, or in
an ELF or Mach-O image the stub of another import, is dropped; one whose
branches all reach routines they name is confirmed. What the matches then
name is what their callers' branches are checked against in turn, until
nothing new is named. A branch to a routine that only jumps on -- a
linker's thunk, or a routine that tail-calls another, such as the release
CRT's
free, which jumps to_free_base-- is followed on, and what it reaches confirms the reference but contradicts nothing. - NeverD releases older than these references reject such lines. One before NeverSight/NeverD#208 reads an odd offset as a Thumb-2 branch, and one before NeverSight/NeverD#214 reads the references at one offset as separate branches, each of which the others contradict.
These rules are NeverD's own:
neverd-sigmaker
produces the lines, and neverd::sigs::PatternGenerator defines how many
bytes each relocation rewrites.
A name is the linkage name the library's symbol table spells, byte for byte:
?Close@CFile@@UEAAXXZ, _ZNSt6thread4joinEv, x86 _memcpy, Mach-O _deflate. Names are never
demangled, sanitized, prefixed or truncated.
A line is kept only if its bytes identify one routine. Lines that state the same bytes under names they share are one routine's, under the symbols each build of the library defines, and become one line with all of their names. When lines state the same bytes under names with nothing in common, all of them are dropped, because any one name would be a guess, unless the file's lines name different routines at a branch they all make: then the one whose branch NeverD confirms is taken. NeverD may apply every file of a directory together, so this holds across the directory. A line is also dropped when a routine of another name, at least as long, states every byte the line states: NeverD compares a line only as far as the line's own length and accepts any byte where the line has a wildcard, so the line would name that routine too.
| File | Built from |
|---|---|
vs2015.pat |
MSVC v140 (14.0): x86 and x64 with ATL/MFC, ARM32 without |
vs2017.pat |
MSVC v141 14.16 with ATL/MFC (x86, x64, ARM32, ARM64) |
vs2019.pat |
MSVC v142 14.29 with ATL/MFC (x86, x64, ARM32, ARM64) |
vs2022.pat |
MSVC v143 14.44 with ATL/MFC (x86, x64, ARM32, ARM64) |
vs2026.pat |
MSVC 14.50 and 14.51 with ATL/MFC (x86, x64, ARM64; VS 2026 has no ARM32 toolchain) |
winsdk.pat |
Windows SDK 10.0.17763, 18362, 19041, 20348, 22000, 22621 and 26100: the Universal CRT and the user-mode libraries (ARM32 through 10.0.22621) |
vs2005.pat |
Visual Studio 2005 Team Suite (8.0.50727.42) with ATL/MFC, the Platform SDK, the Windows CE x86 libraries and the DIA SDK (x86, x64) |
vs2008.pat |
Visual Studio 2008 Professional (9.0.21022) with ATL/MFC, the Windows SDK 6.0A, the Windows CE x86 libraries, the CRT libraries built from source and the DIA SDK (x86, x64) |
vs2010.pat |
Visual Studio 2010 Professional (10.0.30319) with ATL/MFC, the Windows SDK 7.0A and the DIA SDK (x86, x64) |
vs2012.pat |
Visual Studio 2012 Professional (11.0.50727) with ATL/MFC (x86, x64, ARM32) |
vs2013.pat |
Visual Studio 2013 Professional (12.0.21005) with ATL/MFC and the Windows SDK 7.1A (x86, x64, ARM32) |
masm32.pat |
The MASM32 SDK 8.2, 9, 10 and 11: masm32.lib, fpu.lib and datetime.lib built from their sources with the SDK's own assembler, and the prebuilt debug.lib (x86) |
mingw32-zlib.pat |
zlib 1.3 compiled without optimization, as the code of rizin's mingw32-zlib lines reads, by Ubuntu's MinGW-w64 GCC (x86, x64) |
The Visual Studio 2005 to 2013 libraries come from Microsoft's installation
media, which no current Windows image can install; they are unpacked on
Ubuntu instead (see below). They are the release-to-manufacturing builds of
the same media rizin's files were made from. Microsoft no longer publishes a
Visual Studio 2010 disc; its libraries come from the Internet Archive's copy
of the MSDN Professional disc, en_visual_studio_2010_professional_x86_dvd_509727.iso,
whose SHA-1 f0ed50712d83bf0eda7d284da76df49e4c88cef7 is the one MSDN listed
for it and the one rizin's sigdb-source records. (The Windows SDK 7.1, which
Microsoft does publish, carries the same Visual C++ 2010 CRT libraries byte
for byte, but no ATL/MFC.) Every file is built from its libraries alone: no
line rizin had for it remains. See
Lines imported from rizin.
Two workflows produce the generated files:
msvc-libraries.ymlruns on GitHub-hosted Windows images, one job per row of.github/msvc-matrix.json:- It installs each toolset or SDK the image lacks. Most come from the image's own Visual Studio installer. An SDK that installer no longer offers comes from Chocolatey or from Microsoft's standalone SDK installer.
- It packs each architecture's static libraries into a reproducible
tar.zst, with a manifest that gives the size and SHA-256 of every file, and keeps the archives as assets of anmsvc-libs-*release. - It builds the programs in
validation/with the same toolset: statically linked, with linker maps. - One Ubuntu job per row of
.github/msvc-legacy-matrix.jsondownloads an older release's installation medium from Microsoft (for Visual Studio 2010, the Internet Archive's copy of the MSDN disc), checks its SHA-256, unpacks the Windows Installer packages that hold the libraries with 7-Zip, cabextract and msitools, and packs the same kind of archive (scripts/collect_legacy_media.py). - One Ubuntu job per row of
.github/masm32-matrix.jsonbuilds a MASM32 SDK release's libraries. The SDK ships them as sources, which its installer assembles with the SDK's ownML.EXEandLINK.EXE; the job runs the same make files under Wine and clears the time stampsLINKwrites, so two builds give the same archive (scripts/collect_masm32.py). - One Ubuntu job per row of
.github/mingw-matrix.jsoncompiles a C library rizin built with MinGW itself, from its release archive, with Ubuntu's MinGW-w64 cross compilers, and archives it in ar's deterministic mode, recording the compiler and header packages (scripts/collect_mingw_library.py). - One Ubuntu job per row of
.github/elf-matrix.jsoncollects the static libraries of an ELF file's packages; see Linux and Android (elf/) signatures. - One Ubuntu job per row of
.github/homebrew-matrix.jsoncollects the static libraries of a formula's Homebrew bottles; see macOS (macho/) signatures.
msvc-signatures.ymlbuildsneverd-sigmakerat a pinned NeverD revision and runs NeverD'sbuild_msvc_signatures.pyover everymsvc-libs-*release, one architecture at a time:- A file with library archives behind it is rebuilt from them alone.
- Each architecture keeps only objects built for it (
--machine), and lines cover every byte of every function. - Ambiguous lines are dropped across the directory, as described above.
- Each file is read back with the loader's parser.
- A manual run with
publishset commits the result. Withmigrate_importedset, it first runs the migration of the imported files again.
Both workflows run from Actions → Run workflow. The library releases are
drafts unless publish_release is set. The archives hold unmodified Microsoft
libraries, which stay under the license terms of Visual Studio and the Windows
SDK. They are kept so that every line traces back to the exact bytes it came
from, not to redistribute the libraries.
| File | Built from |
|---|---|
ubuntu-libc6.pat |
glibc 2.3.2 to 2.36: libc.a, libpthread.a, librt.a, libanl.a, libutil.a and libresolv.a of 338 libc6-dev packages (x64) |
ubuntu-libgcc-7.pat to ubuntu-libgcc-12.pat |
GCC 7 to 12's libgcc.a and libgcc_eh.a (x64) |
ubuntu-libstdc++-5.pat to ubuntu-libstdc++-12.pat |
GCC 5 to 12's libstdc++.a, libsupc++.a and libstdc++fs.a (x64) |
ubuntu-libc++-7.pat to ubuntu-libc++-15.pat |
LLVM 7 to 15's libc++.a and libc++fs.a (x64) |
ubuntu-musl.pat |
musl 0.9.14 to 1.2.3's libc.a (x64) |
ubuntu-openssl.pat |
OpenSSL 0.9.7 to 3.0.5's libcrypto.a and libssl.a (x64) |
ubuntu-zlib.pat |
zlib 1.2.1 to 1.2.13's libz.a (x64) |
ubuntu-libsodium.pat |
libsodium 1.0.0 to 1.0.18's libsodium.a (x64) |
ubuntu-libseccomp.pat |
libseccomp up to 2.5.4's libseccomp.a (x64) |
android-ndk.pat |
The Android NDK r9d to r25b: bionic for every API level, the C++ runtimes (libc++, GNU libstdc++, STLport, gabi++) and the toolchains' Android runtimes (libgcc, compiler-rt, libunwind, libatomic, OpenMP), but none of the libraries its toolchains run on the host (x86, x64, ARM32, ARM64) |
fedora-zlib.pat |
zlib 1.3 compiled without optimization by Fedora 38's GCC 13.2.1 and clang 16.0.6 (x86, x64) |
Every file but fedora-zlib.pat is built from the packages rizin's
sigdb-source records for it: each ubuntu-* file from its library's -dev
package in every Ubuntu release from 4.10 to 22.10 that carried one, and
android-ndk.pat from the NDK releases. One Ubuntu job per row of
.github/elf-matrix.json
(scripts/collect_elf_packages.py)
downloads them, Ubuntu's packages from Launchpad and the NDK from Google,
checks each against the SHA-1 sigdb-source records, and archives the static
libraries it holds by their objects' ELF class and machine, in
msvc-libs-* releases like the Windows libraries. A library whose bytes
another package already supplied is stored once; the manifest still lists
the package.
fedora-zlib.pat has no package behind it: rizin's lines came from
signature-builds-zlib,
which configured zlib 1.3 with CMake and no build type and compiled it with
a Fedora machine's gcc and clang. One job per row of
.github/fedora-matrix.json
(scripts/collect_fedora_library.py)
builds it the same way with the compilers Fedora 38 had when the lines were
made, unpacked from the packages Koji keeps and checked against their
SHA-256. Of rizin's 561 lines, that build reproduces 538, and 15 state bytes
that several of its routines share. The other 8 state a relocated field, or
a CRC over one, as rizin read it with the object's relocations applied at its
own load address (0x08000000), which no program's bytes match.
An ELF image has no Rich header to name the release of its libraries, so
neverd sigs --auto reads every file of its elf/ directory, and ambiguous
lines are dropped across the whole directory, as described above. The ELF
lines state their branches as the Windows ones do since NeverSight/NeverD#208,
so lines that differ only in the routines they call are kept and told apart:
r25b's ARM32 ctype_byname<wchar_t>::do_toupper and do_tolower are the same
bytes calling towupper and towlower, and NeverD takes the one whose call
reaches the routine it names: the routine NeverD names there, or in a
dynamically linked program the import whose PLT stub it is.
| File | Built from |
|---|---|
homebrew-zlib.pat |
zlib 1.3.2's libz.a from Homebrew's arm64_sonoma, arm64_sequoia, arm64_tahoe and arm64_golden_gate bottles (arm64), and from its sonoma bottle (x64) |
homebrew-xz.pat |
xz 5.8.4's liblzma.a from the arm64_sequoia, arm64_tahoe and arm64_golden_gate bottles (arm64) |
homebrew-pcre2.pat |
PCRE2 10.49's libpcre2-8.a, libpcre2-16.a, libpcre2-32.a and libpcre2-posix.a from the arm64_sequoia, arm64_tahoe and arm64_golden_gate bottles (arm64) |
One Ubuntu job per row of
.github/homebrew-matrix.json
(scripts/collect_homebrew_bottles.py)
downloads a formula's bottles from Homebrew's package registry on ghcr.io,
by the SHA-256 Homebrew publishes for each bottle, which the row pins. It
archives the static libraries the row names by their objects' Mach-O CPU
type, in msvc-libs-* releases like the other libraries. A library whose
bytes another bottle already supplied is stored once; the manifest still
lists the bottle. Each file's <name>.sources.json lists the bottles it was
built from, by tag and SHA-256, so the file can be traced to them without a
release. The zlib.pat, lzma.pat and pcre2.pat that stood here before
had no record of what they were built from; these files replace them.
ld64 and lld do more to arm64 code than fill in relocated fields. An object's
linker optimization hints (LC_LINKER_OPTIMIZATION_HINT) name the
instructions of an address computation, which the linker may fold into
fewer. It then rewrites the load or store at the end of the computation too,
although no relocation covers it: in the arm64_sequoia probe below, lld
turned zlib's ldr x8, [x8, #0x20] into ldr x8, [x8, #0x2b8]. The lines leave
every instruction a hint names unstated, so deflate and PCRE2's
compile_regex match. On x86-64 they turn a GOT load of a routine the
program defines from movq into leaq, two bytes ahead of the relocated
field, which the lines leave unstated too: in the sonoma x64 probe lld
rewrote 3 of zlib's 9 GOT loads. In both probes every other byte that
changed between an object and the program lies in a relocated field.
PCRE2's bottles are built with LLVM's machine outliner, which moves
instruction sequences several functions repeat into OUTLINED_FUNCTION_<n>
routines and numbers them anew in each object. The number says nothing about
the code another build numbers alike, so no line is named after such a
routine and no reference names one (NeverSight/NeverD@518dee92): the routine
only ends the function before it.
A Mach-O image has no Rich header either, so neverd sigs --auto reads every
file of its macho/ directory, and ambiguous lines are dropped across the
whole directory. PCRE2 builds each routine three times, for 8-, 16- and
32-bit code units, and many of the three are the same bytes.
scripts/evaluate_probes.py runs
neverd sigs --json --no-debug over the validation programs and compares
every name NeverD would apply with the linker map of the same image:
- named: library functions that NeverD names as the map does.
- wrong: addresses NeverD would give a name the map does not give them. This includes the program's own functions: STL templates the program instantiates compile to the same bytes as library code.
- disputed: addresses where the loaded lines disagree and no branch reference settles which is right, so NeverD names nothing. An address where exactly one of the names comes from a match whose references NeverD confirmed gets that name, and counts as named or wrong like any other.
validation/ holds three programs, each linked statically with a map: an
MFC program and a CRT program built with /O2 /MT, and the CRT program built
with /Od /MTd. They were built with Visual Studio 2015 (x86, x64), 2017, 2019
and 2022 (x86, x64, ARM32, ARM64), and 2026 with both its default toolset and
14.50 (x86, x64, ARM64): 60 programs. NeverD chooses the files as
neverd sigs --auto does, by the program's Rich header. Named is the share of
the map's library functions; wrong is the share of the names NeverD applies.
Measured with NeverD at
4428327d.
| Architecture | Programs | Named, optimized | Named, /Od |
Wrong, optimized | Wrong, /Od |
Disputed |
|---|---|---|---|---|---|---|
| x86 | 18 | 54% | 57% | <0.1% | 1.8% | 3,322 |
| x64 | 18 | 62% | 59% | <0.1% | 2.0% | 2,709 |
| ARM32 | 9 | 60% | 61% | <0.1% | 1.3% | 1,453 |
| ARM64 | 15 | 65% | 66% | <0.1% | 1.5% | 2,330 |
Across all 60 programs NeverD names 170,024 of 284,642 library functions
(60%), 550 names are wrong (0.3%), and 9,814 addresses are disputed. A
disputed address is one where lines that differ only in their branches all
match and none of the branches settles which: NeverD leaves it unnamed. At
71 other such addresses a confirmed branch settles it, and 70 of those
names are right. Counting these needs a NeverD that reports each match's
confirmed flag (NeverSight/NeverD#164); with an older one the script
counts them as disputed. Since NeverSight/NeverD#208 what the branches settle
is what the branches of the routines' callers are checked against in turn,
which names 1,431 more of these programs' library functions. Since
NeverSight/NeverD#212 a routine that only jumps on no longer contradicts
the matches that call it: 2,605 more, which the references had dropped
because operator delete jumps to free and malloc to _malloc_base.
The alternatives of NeverSight/NeverD#214 name 489 more. None of the three
names more wrongly.
Before the Rich header chose the files, before the covering rule and the
branch references, and before the Visual Studio 2005 to 2013 files were
rebuilt from their libraries, it named 133,078 (47%) and 2,678 names were
wrong (2.0%).
- Optimized builds (
/O2,/MT, with or without MFC) are the common case, and 53 of their names are wrong. On ARM these are mostly short routines that share their code with other routines except for what a relocation rewrites, such as MFC initializers that differ only in the message they register. /Odbuilds compile many template instantiations to the same bytes, so more of their names are wrong: the program's own instantiations take the names of the library's. The branch references drop such a name when the instantiation calls a routine NeverD names otherwise, which it cannot do when the routines it calls are the program's own and unnamed.- Rebuilding
vs2010.patfrom the Visual Studio 2010 disc, with ATL/MFC, cost these programs 30 names and made 5 wrong. Releases give some short routines' bytes different names -- ATL'sT2BSTRand the Universal CRT's_wcslenare the same code up to the routine they call -- so every file of the directory drops them, as described above. With the debug CRT's_wcslenunnamed, the/Odprograms'tcslenwrappers match_aligned_free's line, and its reference has no name to contradict. - x86 coverage is bounded by NeverD's function discovery, which since
Find packed MSVC hotpatch entries and start them at the no-opalso finds the functions MSVC packs directly after aret. Since NeverSight/NeverD#204 signatures are also tried at every function NeverD's detector finds, not only at those the image's tables state; the optimized x86 programs went from 28% named to 54%. - The Visual Studio 2005 to 2013 files have no validation programs with
maps, but the setup programs on their media are linked statically with
their release's runtime, and the Rich header chooses the file of the
linker's release (VS 2005, VS 2008, and VS 2010 SP1 for the VS 2012 and
2013 installers). Microsoft's symbol server keeps the public PDBs of some
of them, which
scripts/pdb_truth.pyturns into whatevaluate_probes.pycompares with. A public PDB names only public symbols, so a name at a static function cannot be checked either way.- VS 2008's setup program: NeverD names 300 of the 492 functions the PDB
names that the release's libraries define. Of its other names, 43 are at
static functions, 2 differ from the PDB only in decoration (an ATL
header function the program compiled itself with
/Gzor/Zc:wchar_t-), and 1 is wrong: MFC'sCOccManager::OnEventat the program's own callback of the same bytes. - VS 2010's: 277 of 429. 37 are at static functions, 15 differ only in
decoration, 5 name the MFC instantiation of an ATL
CStringTmember whose code is the same, and 2 are wrong. - VS 2005's: 298 of 480. 42 are at static functions, 2 differ only in decoration, and none is wrong. Its linker placed read-only data in the code section, where NeverD's function discovery alone finds 85 of the program's 747 functions; since NeverSight/NeverD#204 signatures are also tried at the functions NeverD's detector finds there.
- The symbol server has no PDB for the VS 2012 and 2013 setup programs.
- VS 2008's setup program: NeverD names 300 of the 492 functions the PDB
names that the release's libraries define. Of its other names, 43 are at
static functions, 2 differ from the PDB only in decoration (an ATL
header function the program compiled itself with
validation/elf/ holds C and C++ programs linked statically against the
very packages the ELF files were made from, as
scripts/build_elf_probes.py builds them, and
Android programs built with NDK r25b's own clang. Each is compared with the
symbol table of the same program before it was stripped: a function is a
library's when one of the archives the program was linked with defines one of
its names. For an ELF image NeverD reads every file of the directory. Before
is what the lines imported from rizin, moved to the rules above, named in the
same programs, as measured with NeverD at
b08a55bb
(ARM) and
278e2f2e
(the others). The files as they are now were measured with NeverD at
4428327d.
| Programs | Count | Library functions | Named before | Named | Wrong before | Wrong | Disputed before | Disputed |
|---|---|---|---|---|---|---|---|---|
glibc 2.27, 2.31, 2.35, -O2 and -O0 |
6 | 6,265 | 3,206 (51%) | 4,974 (79%) | 117 | 14 | 79 | 0 |
| GNU libstdc++ 11, 12 | 2 | 9,509 | 2,915 (31%) | 4,454 (47%) | 176 | 5 | 54 | 618 |
| zlib 1.2.11 | 1 | 1,182 | 653 (55%) | 968 (82%) | 25 | 0 | 14 | 0 |
| OpenSSL 3.0 | 1 | 10,726 | 5,278 (49%) | 6,862 (64%) | 1,510 | 2 | 287 | 479 |
| musl 1.2.2 | 1 | 160 | 0 | 94 (59%) | 0 | 0 | 0 | 0 |
| NDK r25b x64 | 1 | 1,850 | 767 (41%) | 1,291 (70%) | 16 | 2 | 14 | 13 |
| NDK r25b x86 | 1 | 1,942 | 177 (9%) | 1,482 (76%) | 10 | 0 | 0 | 17 |
| NDK r25b ARM64, C and C++ | 2 | 5,794 | 465 (8%) | 3,854 (67%) | 4 | 0 | 0 | 90 |
| NDK r25b ARM32, C | 1 | 1,907 | 116 (6%) | 1,255 (66%) | 4 | 0 | 0 | 13 |
| NDK r25b ARM32, C++ | 1 | 4,117 | -- | 2,233 (54%) | -- | 1 | -- | 104 |
- Most of the names wrong before were OpenSSL's: its
d2i_*,i2d_*,*_freeand per-cipher routines are the same code up to the object a relocation names, and rizin's lines named one of them. The rebuilt file drops such lines, as described above; of the 1,510 addresses named wrong before, 2 still are. - bionic's AArch64 assembly syscall stubs begin with a
bti clanding pad and open their unwind entry after it. NeverD before NeverSight/NeverD#191 started each such function at its second instruction, where the stub of an older NDK without the landing pad matches: 365 of its 373 wrong names on these programs were the right names 4 bytes late. - musl builds its library without unwind tables, and NeverD's function discovery does not follow the calls from the entry point: it finds 2 of the program's 167 functions. Since NeverSight/NeverD#204 signatures are also tried at every function NeverD's detector finds, which names 94.
- NeverD decodes the bytes after every call of a stripped ARM32 program as
code. After a call that does not return, such as
bl abort, they are the caller's literal pool, which NeverD recognizes by the load that reads it. The ARM32 C++ program also needs to know which calls do not return: an ARM__memcpy_chkends in one and Thumbwcslenfollows at once. NeverD infers that since NeverSight/NeverD#203, and loads it; the rizin lines were never measured on it. - The disputed addresses are routines whose lines the files keep because
their branches tell them apart, where this program's branches cannot: C++
charandwchar_tinstantiations whose whole call chains are the same bytes (basic_istream<char>and<wchar_t>'s constructors callbasic_ios<char>::initand<wchar_t>::init, which call the same routine), and OpenSSL'sd2i_*,i2d_*and*_freewrappers, which call*_itroutines that differ only in the data they point to. Before, the files dropped these lines, and NeverD named nothing there either.
validation/macho/ lists programs that link every object of one macOS
release's bottles, as
scripts/build_macho_probes.py builds them
with clang and LLVM's ld64.lld against a text stub of libSystem. There is
one program per bottle tag, binding its imports through chained fixups, and
the arm64_sequoia bottles are linked once more with classic dyld
information. Each is compared with the linker map of the same program before
it was stripped. Before is what the zlib.pat, lzma.pat and pcre2.pat
this directory held named in the same programs. Both were measured with
NeverD at
6228f29b.
| Programs | Count | Library functions | Named before | Named | Wrong before | Wrong | Disputed before | Disputed |
|---|---|---|---|---|---|---|---|---|
zlib (arm64_sonoma) |
1 | 146 | 2 (1%) | 119 (82%) | 0 | 0 | 0 | 0 |
zlib, xz and PCRE2 (arm64_sequoia; chained fixups and dyld information) |
2 | 6,528 | 134 (2%) | 2,332 (36%) | 1,222 | 0 | 198 | 46 |
zlib, xz and PCRE2 (arm64_tahoe, arm64_golden_gate) |
2 | 6,556 | 118 (2%) | 2,328 (36%) | 1,164 | 0 | 194 | 46 |
zlib (sonoma, x64; chained fixups and dyld information) |
2 | 288 | -- | 238 (83%) | -- | 0 | -- | 0 |
- Half of the library functions are PCRE2's
OUTLINED_FUNCTION_<n>: 1,656 of thearm64_sequoiaprogram's 3,264. No line names them, as described above. Of the program's other 1,608 functions, 1,166 are named (73%). - There was no x64 file before.
- The files before were made from other builds of the libraries. Most of the names they got wrong were outlined fragments named by the number another build gave them.
- The disputed addresses are routines whose lines the files keep because
their branches tell them apart, where the program's branches cannot:
PCRE2's 8-, 16- and 32-bit
*_createand*_freeroutines, and xz'slzma_easy_encoder_memusageandlzma_easy_decoder_memusage. - Chained fixups and dyld information give the same result: NeverD checks references through the stubs either kind binds.
The first commit of this repository imported pattern files from rizin's
sigdb-source (LGPL-3.0): the PE
vs2005–vs2022, winsdk, masm32 and mingw32-zlib files, and every ELF
file. Those lines followed rizin's conventions, not the rules above:
- Names were rewritten. Characters outside
[A-Za-z0-9_.]became_, and PE names were cut at 125 characters:?AfxRegisterClass@@YAHPAUtagWNDCLASSW@@@Zbecame_AfxRegisterClass__YAHPAUtagWNDCLASSW___Z, and the stdcall_TimeSpan@24became_TimeSpan_24. ELF C++ names were demangled, and some were turned intomethod.<class>.<member>. None of this can be undone from the text alone:_stands for any of_,?,@and$. - Some lines were named after sections, labels or data objects
(
.text_tii_131,_LN116,obj.once.9977). - Many tails were stated from the end of the leading bytes, with the CRC span
written as wildcards. NeverD compares a tail from the end of the CRC span,
so such a line was checked
crclenbytes out of place and hardly ever matched. - Many lines stated bytes that other routines share under other names. In the ELF files, which the same packages explain, 18,780 lines did.
sigdb-source records where each line came from: the SHA-1 of the medium or
package, and a pattern file per library object. For Visual Studio 2005 to
2013 those were the MSDN Professional discs. Their libraries are the
release-to-manufacturing builds on the media Microsoft still publishes, and
for Visual Studio 2010 the very disc sigdb-source names, so the PE files for
Visual Studio 2005 to 2022 and the Windows SDK are now built from collected
libraries, as are masm32.pat and mingw32-zlib.pat, and every ELF file:
from the packages sigdb-source records, and fedora-zlib.pat from the
compilers its lines were made with.
scripts/migrate_imported_signatures.py
accounts for every imported line. It starts from the text of the import
commit every time, reads every file the import held, and compares each line
with every function of the libraries it can read: the collected MSVC and SDK
libraries for pe/, and the collected ELF libraries for elf/. For each
line:
- A tail stated from the end of the leading bytes is moved to start after the CRC span. A line that then states fewer than 16 bytes is removed.
- A line whose bytes do not identify one routine is removed: they agree in full with library functions of several names, they open a longer function of another name, or they are a function's other than the one the imported name spells.
- A line whose bytes agree with a library function of exactly one name takes that name, when the imported name is its rizin spelling or names nothing (a section or a label).
- Failing that, a line takes the one library name that rizin's conversion
spells the same way. The names include every routine a library's symbol
table defines, also those
neverd-sigmakerwrites no line for. - A PE C name is kept as it is: every decorated C++ name holds
@@, which rizin spelled__, so a name with no__past its leading underscores was never rewritten. On 32-bit x86,_name_Nis also how rizin spelled the stdcall_name@N; it becomes that only when the routine ends inret N. - A line named after a section, label or data object, which no library function explained, is removed.
- rizin filed some ELF lines under the other pointer width: 32-bit code among the 64-bit NDK lines, and some 64-bit code among the ARM32 ones. A line whose bytes this width's libraries do not state, but a function of the other width's libraries does, is that width's. So is one whose name only the other width's libraries define, when both widths' files are built from the very build the import was made from: rizin read some objects with their relocations applied, so their bytes match no function. When that width's file is built anew from that build, the line is left out, as that file holds the routine; otherwise it moves to that directory.
- Every file the import held is now built from collected libraries alone,
so no imported line stays in any of them; the report still sorts each
one. A line is reproduced when a collected function of its name states
every byte the line states, also when the function runs on past the
line's end only into the NOPs or INT3s that pad its section: rizin
measured a function to its last instruction,
neverd-sigmakerto the next symbol. When the collected libraries are the very build the import was made from (a Visual Studio release's RTM libraries, the MASM32 SDK's libraries built from its sources with its own assembler), a line whose routine they define at all is superseded: the lineneverd-sigmakermade for that routine, or its decision to make none, stands. A library asset states which it is (reproduces_import): the ELF packages rizin recorded are, and so isfedora-zlibbuilt with the Fedora compilers rizin's lines were made with, butmingw32-zlibcompiled with Ubuntu's MinGW-w64 GCC and the Windows SDK's libraries are other builds. Any other line is not reproduced. - A line with no linkage name is unresolved.
The import held 866,083 lines: 339,431 in pe/x86/32, 311,102 in
pe/x86/64 and 215,550 in elf/. 522,494 are reproduced by the collected
libraries and 109,000 superseded by what they define, and 1,893 are the
other pointer width's code. 190,136 are removed as rules 2 and 6 say:
165,269 whose bytes several library routines share, 23,621 whose bytes open
a longer routine, and 1,246 naming no function. 18,677 have no linkage name,
and 23,883 name a routine no collected library reproduces: labels, C++/CLI
code, objects only the runtime DLLs link, test harnesses, the DIA SDK, which
no asset collects, and rizin's reading of routines the libraries state
themselves. None of them is in any file: every file is built from its
libraries alone. reports/ lists, per file the import held,
every line removed, not reproduced or unresolved.