diff --git a/doc/developers_guide.md b/doc/developers_guide.md new file mode 100644 index 0000000..92a4dba --- /dev/null +++ b/doc/developers_guide.md @@ -0,0 +1,656 @@ +# AFF developer guide + +This document is a reference guide to the AFFLIBv3 library. It is intended to +be read by software developers seeking to use the library in their own +programs. It is *not* intended for end users. + +> This guide originates from https://forensics.wiki/ by the same author of +> AFFLIBv3 and was copied with permission. + +End users will be better off reading the "AFF Users Guide". This guide gives an +introduction to the AFF format and explains the tools included with the +library. The default imager, aimage, is covered in this guide. + +This version of documentation reflections version 2.3.0 of the library. + +## Introduction + +Using the AFF is (hopefully) a rather transparent process. The library can +handle plain files, such as those generated by dd, AFF files, and EnCase (E01) +images. It is not necessary for your program to know which kind of file it is +processing, the data is extracted and presented to the program transparently. +All you need to do is change the basic file operations to their AFF +equivalents. These simple substitutions allow you to use AFF or Encase files +just like dd images. + +## Compiling AFF programs + +The library currently supports the following compilers: + +* gcc +* g++ +* MinGW +* Microsoft Visual C++ + +When compiling an AFF program you must include the AFF header: + +```text +#include +``` + +During linking you must include the AFF library itself and the SSL library. +Additionally, if you're using a C compiler, you will need the C++ standard +library. Some examples using several compilers are shown below: + +Using g++: + +```bash +g++ file.c -lafflib -lssl +``` + +Using gcc: + +```bash +gcc file.c -lafflib -lssl -lstdc++ +``` + +Using MinGW: + +```text +gcc.exe file.c -lafflib -lssl -lstdc++ +``` + +Using Microsoft Visual C++: + +```text +LINK.EXE file.obj afflib.lib +``` + +### Testing for the AFF using Automake + +If you're using the GNU Automake and other tools to check for the AFF, we +recommend the following tests to determine if the AFF is present on the system: + +```text +########################################################### +## Test for AFF Support +########################################################### +AC_ARG_ENABLE([aff], + AC_HELP_STRING([--enable-aff],[Use AFF for reading files (default yes)]), + [enable_aff=$enableval], [enable_aff=yes]) +if test "${enable_aff}" = "yes" ; then + AC_CHECK_HEADERS([afflib.h],,[enable_aff=no]) + AC_CHECK_LIB([aff],[af_open],,[enable_aff=no]) +fi +if test "${enable_aff}" = "yes" ; then + AC_DEFINE([USE_AFF],1,[Use the AFF library to read files]) + + AC_MSG_RESULT([Found Advanced Forensics Format (AFF), good!]) + AC_MSG_RESULT([Checking for AFF specific features...]) + + AC_CHECK_LIB([aff],[af_open_stdin], + AC_DEFINE([USE_AFF_STDIN],1,[The AFF can be used to open standard input])) +fi +``` + +These tests will define two variables in the config.h file for you. First, the +value USE_AFF is defined the AFF is found on the development system. Second, +the value USE_AFF_STDIN is defined if the function af_open_stdin is supported. +That function was not present in early versions of the library. + +## Your First AFF program + +The first program we will examine the command line arguments to determine if +they are valid AFF files. If so, it will attempt to open them and display the +size of their uncompressed data. The complete source code is below. + +```text +#include +#include +#include +#include +#include + +#define FALSE 0 +#define TRUE 1 + +int process_file(char *fn) +{ + AFFILE *af = af_open(fn,O_RDONLY,0); + if (NULL == af) + { + perror(fn); + return TRUE; + } + + printf ("The uncompressed size of this file is %"PRId64" bytes\n", + af_get_imagesize(af)); + + af_close(af); + return FALSE; +} + + +int main(int argc, char **argv) +{ + if (argc < 2) + { + printf ("Usage: %s [FILES]\n", argv[0]); + return EXIT_FAILURE; + } + + argc++; + argv--; + + while(*argv) + { + process_file(*argv); + ++argv; + --argc; + } + return EXIT_SUCCESS; +} +``` + +We will go through the important parts of the program. + +```text +#include +#include +``` + +These two lines include the necessary headers to use the AFF. Although fcntl.h +is not actually required, it provides several macros that are highly +recommended. + +```text +AFFILE * af = af_open(fn,O_RDONLY,0); +if (NULL == af) +{ + perror(fn); + return TRUE; +} +``` + +This line attempts to open the given file using the af_open command. The +O_RDONLY is one of many macros defined in fcntl.h that can be used when +opening files. The final parameter specifies whether or not the file +must be created. This normally only applies when writing AFF files, so +it's safe to leave it as zero here. The function returns a pointer to an +AFFILE structure. There are some public data fields in that structure +defined in afflib.h. + +```text +printf ("The uncompressed size of this file is %"PRId64" bytes\n", + af_get_imagesize(af)); +``` + +This line uses one of the miscellaneous functions included with the AFF +to display the size of the uncompressed image stored in the file. +Compare the value displayed by this line to the size of the AFF image on +the disk and you can see how much the library compresses. The example +file zeros.aff shows an extreme case! + +```text +af_close(af); +``` + +This final line closes the file handle and destroys the AFFILE +structure. + +## AFF Concepts + +### Segments + +AFF files can be viewed as a series of name/value pairs. + +### Pages + +### AFF Return Values + +The following status codes can be returned by AFF functions. + +| Code | Description | +| --- | --- | +| AF_ERROR_EOF | The end of the AFF file has been reached | +| AF_ERROR_DATASMALL | The buffer provided for the requested AFF segment is not large enough to hold the data that would be returned. | +| AF_ERROR_TAIL | There is no tail or an error occurred trying to read the tail of an AFF segment. This indicates that the file is corrupt (probably truncated). | +| AF_ERROR_SEGH | There is no head or an error occurred trying to read the head. This indicates that the AFF file is corrupt. | +| AF_ERROR_NAME | Invalid segment name | +| AF_ERROR_INVALID_ARG | The argument was invalid | +| AF_ERROR_NO_AES | This version of the library was compiled without AES support and the program is attempting to read an AES-encrypted segment. | +| AF_ERROR_AES_TOO_SMALL | The decrypted AES segment is too small for the amount of data that it was supposed to contain. The segment is probably corrupt. | +| AF_ERROR_KEY_SET | A key was previously set. (You can only have one AES encryption key per file.) | + +## AFF Function Reference + +### Stream Functions + +#### af_open + +```text +AFFILE * af_open(const char *filename, int flags, int mode); +``` + +The flags parameter controls what the program should be allowed to do +with the file. There are a set of handy macros defined in fcntl.h that +you should use. When reading files, using O_RDONLY is usually +sufficient. When writing an AFF file, it is best to use O_CREAT\|O_RDWR. + +The mode parameter is only used when creating a new file (RBF - and only +on \*nix?). It controls the + +It should be noted here that when writing AFF files, the af_open command +does more than just open a file handle. When writing, the af_open +command causes the default AFF header to be written to the file on the +disk. + +TODO: describe what happens, when opening for write, if the file already exists? + +See [open(3)](https://linux.die.net/man/3/open) + +#### af_open_stdin + +```text +AFFILE * af_open_stdin(void); +``` + +A special case of af_open, this function an AFF handle on standard +input. Note that this function did not appear until AFF version 2.3. +Also note that if you are working on the Windows platform (or plan to +support the Windows platform) you need to explicitly tell the compiler +to treat standard input as a stream of binary data. For example, using +MinGW you can set: + +```text +int _CRT_fmode = _O_BINARY; +``` + +somewhere in your code. See for +more + +#### af_popen + +```text +AFFILE * af_popen(const char *command, const char *type) +``` + +This function is currently on supported on UNIX variants, +[Cygwin](cygwin.md) and MinGW (which are really both Unix +variants). + +See [popen(3)](https://linux.die.net/man/3/popen) + +#### af_close + +```text +int af_close(AFFILE *af); +``` + +Closes an AFF file handle and frees all resources associated with it. +You can use this function regardless if you opened the file with af_open +or af_popen. + +#### af_read + +```text +int af_read(AFFILE *af, unsigned char *buf, size_t count); +``` + +Similar to `fread`, this function reads `count` bytes from the file and +stores them in `buf`. `buf` should already be allocated to hold this +amount of data. The function returns the number of bytes read or -1 on +error. Note the number of bytes read is size_t, not uint64_t, because +size_t defines the maximum data segment size on the current platform. + +**RBF - This function is supposed to read a `size_t` number of bytes. +Should it also return a `size_t`?** + +See also [read(3)](https://linux.die.net/man/3/read) + +#### af_seek + +```text +uint64_t af_seek(AFFILE *af, int64 pos, int whence); +``` + +Sets the file pointer into the *decompressed data*. The new position, +given in bytes, is obtained by adding `pos` bytes to the position +specified by `whence`. If `whence` is set to SEEK_SET, SEEK_CUR, or +SEEK_END, the offset is relative to the start of the file, the current +position, or the end of the file, respectively. On success this function +returns zero. **RBF - On error this function returns minus one +(uint64_t) ??** and sets errno appropriately. + +##### Errors + +EBADF: The stream is not seekable. (e.g. a stream that was opened with +af_popen) + +EINVAL: The `whence` argument was not one of SEEK_SET, SEEK_CUR, or +SEEK_END. + +See also [fseek(3)](https://linux.die.net/man/3/fseek) and +[fseeko(3)](https://linux.die.net/man/3/fseeko) + +#### af_tell + +```text +uint64 af_tell(AFFILE *af); +``` + +Similar to ftello, this function returns the current value of the file +position indicator for the file pointed to by `af`. On an error, this +function returns -1 and sets errno appropriately. + +See also [ftell(3)](https://linux.die.net/man/3/ftell) and +[ftello(3)](https://linux.die.net/man/3/ftello) + +#### af_eof + +```text +int af_eof(AFFILE *af); +``` + +Returns TRUE if the file pointer into the decompressed data is at the +end of the decompressed data. Otherwise, returns FALSE. + +#### af_reopen + +### Writing AFF Files + +#### af_enable_writing + +```text +void af_enable_writing(AFFILE *af, int flag); +``` + +This function must be executed before attempting to write data (using +af_write) to an AFF file. You do not need to execute this function, +however, before calling af_update_seg. If flag is TRUE, enables the user +to write to the file AF. + +#### af_set_callback + +```text +void af_set_callback(AFFILE *af, + void (*wcb)(struct affcallback_info *)); +``` + +Sets a callback function that is called during each of the four phases +of the segment write operation. + +#### af_enable_compression + +```text +void af_enable_compression(AFFILE *af, int type, int level); +``` + +Enables compression on the given AFF file. The type of compression must +be an AFF compression algorithm as shown in Table 2. The compression +level can be any integer between AF_COMPRESSION_MIN and +AF_COMPRESSION_MAX. These values are currently these are one and nine, +respectively, but could change in future versions. + +For example, to enable maximum LZMA compression: + +```text +af_enable_compression(af,AF_COMPRESSION_ALG_LZMA,AF_COMPRESSION_MAX); +``` + +| Flag | Algorithm | +| --- | --- | +| AF_COMPRESSION_ALG_NONE | No compression | +| AF_COMPRESSION_ALG_ZLIB | zlib | +| AF_COMPRESSION_ALG_LZMA | LZMA. Dramatically slower than zlib, this algorithm can produce image files that are significantly smaller. | + +#### af_compression_type + +```text +int af_compression_type(AFFILE *af); +``` + +Returns the compression algorithm being used by the file in question. +The algorithms are given in Table 2. + +#### af_write + +```text +int af_write(AFFILE *af, unsigned char *buf, size_t count); +``` + +Note that you must call af_enable_writing on af before using this +function. + +#### af_badflag + +```text +const unsigned char * af_badflag(AFFILE *af); +``` + +Returns the pattern used to identify bad sectors in this file. + +#### af_is_badsector + +```text +af_is_badsector(AFFILE *af, unsigned char *buf); +``` + +Returns TRUE if buf is the pattern used to mark a bad sector, Otherwise, +returns FALSE. + +#### af_stats + +```text +void af_stats(AFFILE *af, FILE *f); +``` + +This function writes statistics about the current usage of the given AFF +file. This includes the file's name and the number of pages read, pages +written, pages compressed, pages decompressed, cache hits, cache misses, +and bytes copied into the cache. This function is generally used only +for debugging the AFF. + +Sample output: + +```text +AFSTATS for filename.aff +Pages read: 2 +Pages written: 0 +Pages compressed: 0 +Pages decompressed: 1 +Cache hits: 0 +Cache misses: 1 +Bytes copied: 10848768 +``` + +#### af_set_cache_size + +### Segment Functions + +#### af_rewind_seg + +```text +af_rewind_seg(AFFILE *af); +``` + +Returns the segment pointer to the beginning the file. This function +should be used before any segments are examined. + +#### af_get_seg + +```text +int af_get_seg(AFFILE *af, const char *name, unsigned long *arg, unsigned char *data, size_t *datalen); +``` + +#### af_get_next_seg + +```text +int af_get_next_seg(AFFILE *af, char *segname, size_t segname_len, unsigned long *arg, unsigned char *data, size_t *datalen); +``` + +#### af_update_seg + +```text +int af_update_seg(AFFILE *af, const char *name, unsigned long arg, const void *value, unsigned int vallen); +``` + +**RBF - Why aren't these arguments in C99 type? Arg, especically, needs +to be denoted as being 32 bits** + +**RBF - If the sizes of segments when read is returned in a size_t, why +isn't the length size_t when being passed in for writing? And why the +different name for the length parameter** + +##### Proposed new prototype + +```text +int af_update_seg(AFFILE *af, const char *name, uin32_t arg, const void *value, size_t datalen); +``` + +This function is used both to set and change the values of segments in a +file. The name of the segment to be updated is given in name. If the +segment does not exist, it will be created. The value arg is 32-bits of +application specific data. If your segment is only going to store 32 +bits of data or less, you can easily store the value in the arg +parameter. In this case, set data to NULL and datalen to zero. + +Programmers do not need to call af_enable_writing before using this +function. af_update_seg returns zero on success and -1 on error. When an +error occurs the value errno is set to indicate the problem. Possible +values for errno include: ENOTSUP The operation is not supported by the +filesystem. This happens when the programmer attempts to set segments in +a file that doesn't support metadata (e.g. AF_IDENTIFY_RAW). + +Examples: + +Updating a small value: + +```text +int status = af_update_seg(af,"Luggage Combination",12345,NULL,0); +if (status) +{ + perror(af_filename(af)); + return -1; +} +``` + +Updating a large value: + +```text +status = af_update_seg(af,"List of problems solved by MacGuyver", 0, buf_macguyver,strlen(buf_macguyver)); +if (status) +{ + perror(af_filename(af)); + return -1; +} +``` + +#### af_del_seg + +```text +int af_del_seg (AFFILE *af, const char *name); +``` + +Deletes the first segment of the given name from the file. Returns zero +on success, -1 on error. + +### Miscellaneous Functions + +#### af_filename + +```text +const char * af_filename(AFFILE *af); +``` + +Returns the filename associated with an opened file. Sample: + +```text +printf ("The current file being read is %s.\n", af_filename(af)); +``` + +#### af_identify + +```text +int af_identify(AFFILE *af); +``` + +Returns the type of file associated with the given handle. The types are +defined in Table 3. + +| Type | Description | +| --- | --- | +| AF_IDENTIFY_RAW | A raw file, sometimes also called a dd image or flat file | +| AF_IDENTIFY_AFF | A standard AFF archive | +| AF_IDENTIFY_AFD | A directory of AFD files, or a split AFF file | +| AF_IDENTIFY_EVF | An EnCase file read with the original AFF EnCase implementation; now deprecated. | +| AF_IDENTIFY_EVD | An EnCase file read with the original AFF EnCase implementation; now deprecated. | +| AF_IDENTIFY_SPLIT_RAW | A split raw file | +| AF_IDENTIFY_AFM | A raw file with metadata | +| AF_IDENTIFY_EWF | An EnCase file read with libewf. | + +Sample usage: + +```text +const char * af_identity_to_str(int id) +{ + switch(id) { + case AF_IDENTIFY_RAW: return "Raw file"; + case AF_IDENTIFY_AFF: return "Standard AFF image"; + case AF_IDENTIFY_AFD: return "Split AFF image"; + case AF_IDENTIFY_EVF: + case AF_IDENTIFY_EVD: return "Deprecated EnCase image"; + case AF_IDENTIFY_SPLIT_RAW: return "Split Raw file"; + case AF_IDENTIFY_AFM: return "Raw file with metadata"; + case AF_IDENTIFY_EWF: return "EnCase image"; + default: return "Unknown"; + } +} + +int identify_filetype(AFFILE *af) +{ + printf ("The file is type: %s\n", af_identify_to_str(af_identify(af))); + return FALSE; +} +``` + +#### af_get_imagesize + +```text +int64 af_get_imagesize(AFFILE *af); +``` + +Returns the number of bytes in the decompressed data. On an error, +returns -1. Sample: + +```text +int64_t data_size = af_get_imagesize(af); +if (-1 == data_size) +{ + fprintf (stderr,"%s: Unable to determine size of decompressed data", af_filename(af)); + return TRUE; +} +printf ("The decompressed data is 0x%"PRIx64" bytes\n", data_size); +``` + +## Segment Names + +Although segment names and values can be defined by the user at any +time, here is a list of the segments that are included in the AFF +library. Those values set by aimage are noted. + +**RBF - Which segment names are required for the file to be a valid +AFF?** + +**RBF - How does the programmer enumerate all of the segment names?** + +| Name | Aimage | Description | +| --- | --- | --- | +| AF_IGNORE | | Ignore segments with a zero length name | +| AF_PAGESIZE | Y | The size of each page, in bytes | +| AF_BADFLAG | Y | Data that is inserted into the image file to mark a bad sector. | +| AF_DMESG | Y | The results of the 'dmesg' command are stored in the AFF file. | +| AF_ACQUISITION_COMMAND_LINE | Y | The command line used to run the disk imaging program. | +| AF_ACQUISITION_ISO_COUNTRY | Y | The ISO country code of the computer used to create the image; set manually at imaging time. | +| AF_PAGE_MD5 | Y | The MD5 hash of the current page | +| AF_ACQUISITION_SECONDS | Y | The number of seconds used to create the image file. |