Chainmail is a Sendmail® milter, thus you need to have Sendmail® properly installed and its milter support enabled. Configuration and installation of Sendmail® is beyond the scope of this document; read libmilter/README file in Sendmail® distribution for instructions on how to enable milter functionality.
If you intend to compile milteroo and chainmail from sources, you need to install libmilter headers and library. How to do this is distribution-dependent. In Red Hat® and similar systems, you need to install sendmail-devel package.