DEMIME(1)

NAME
demime - Removes mime attachments and other cruft from e-mail

SYNOPSIS
 demime [-[no]d] [--[no]debug] [--[no]quiet] [-p] [--[no]preservefrom]
  [-x(=| )string] [--prefixchar(=| )string] [-i(=| )/path/to/file]
  [--debuginput(=| )/path/to/file] [-j(=| )/path/to/file]
  [--junkmail_file(=| )/path/to/file] [-[no]a] [--[no]adremove]
  [--[no]7bit] [-7] [-8] [--8bit] [--[no]eliderfc822] [--[no]forwarderror]
  [-w(=| )(0|1|2)] [--warnings_to_syslog(=| )(0|1|2)] [--] [relay|-]

DESCRIPTION
There are two major features of demime - mime removal and advertising signature removal.

  Mime Removal
demime reads a piece of e-mail from standard input. It is designed to be invoked directly as an alias program in /etc/aliases or by using majordomo's wrapper program. It attempts to remove all mime cruft from the piece of mail, including alternative sections and attachments and output simple plain text, rendered as well as it possibly can.

It is meant for the mailing list manager who wants to see an end to attachments and unreadable cruft on their mailing list. They can put it into the input stream to make their mime troubles go away.

It can also be used by an individual user who wants to remove all attachments before they read mail. On at least one of the mailing lists I read, people are constantly sending huge attachments and alternate sections. I filter all those through demime using maildrop (although procmail can be used as well).

Basically, mime is fine if you are sending to another like mailer. If you are using Eudora, and another Eudora user sends you mail, you are likely to interpret things in exactly the same manner. But if you are a Eudora 3 user and a Netscape user sends you html mail, it is likely that the mail will appear right justified because of some bug or other. Cross-client mime is just not ready for prime time.

Also, the Majordomo Mailing List Manager inserts whatever is in the input stream into the digest after removing most of the headers. Specifically, such headers as 'Content-type' are removed, leaving readers no way to decode those sections. This means that digest readers frequently have to skip attachment after attachment and it becomes difficult if not impossible for them to make heads or tails out of what comes from the digest - they also have no visual clue, unless they read very carefully, as to when they are in a quoted message, alternative section and so forth.

Finally, mime can hide trojan horses. File attachments to messages can contain viruses, and some mailers have been shown to be subject to attack from unruly javascripts which are imbedded in html sections.

Because of the above, sending mime to mailing lists is probably not a good thing to do. It is quite unlikely that your recipients will interpret your mime attachments in the way you mean them to, unless they happen to have exactly the right mailer.

Microsoft uses various forms of attachments to, I believe, provide formatting hints. These attachments are frequently provided as uuencoded files right in stream, although they may be mime as well. Those attachments are stripped out by demime. To folks not using Microsoft mailers, these attachments are useless overhead.

  Advertising Removal
Common patterns for footers added by such as Juno and Hotmail are detected and those signature blocks are removed. This behavior can be inhibited by setting the $AD_REMOVE variable in the demime program itself to 0, or by specifying -noa or --noadremove. See also "FILES" for the location and format of the file that allows you to control the matching.

  Parameters

-p | --[no]preservefrom
    Controls whether demime passes the Unix style "From" line through, if it exists. Normally defaults to true, except in special situations, should probably be left as true.

-f | --[no]forwarderror
    Controls whether certain errors are bounced back to the origin or forwarded. Typically, the sort of error that is handled this way is an error where no section can be interpreted by demime, or when the format of the mime is screwed up such that the parsing simply will not complete. noforwarderror is the default - demime produces an error on standard output, and returns a code that tells the MTA that there has been a failure. forwarderror attempts to produce the output and the error message to standard output. If you were using demime in a pipe, using the -f flag might be more appropriate.

-x(=| )string | --prefixchar(=| )string
    String, which normally defaults to '|', is used by demime to indicate that demime should start a pipe and pass the processed mail to the pipe. This is explained under $::MAJIC_PIPE_PREFIX_CHAR in the explaination of "target positional parameter", below.

-d | --debug
    Runs demime in debugging mode. Currently the only effect of this is to force things which might go to syslog to always go to stderr.

-i(=| )/path/to/file | --debuginput(=| )/path/to/file
    For debugging, the input file can be specified on the command line. Setting this also sets -d.

-j(=| )/path/to/junkmail/file | --junkmail_file(=| )/path/to/junkmail/file
    Defaults to /usr/pkg/etc/demime_junkmail.cf. The path to the file where the ad removal parameters are kept.

-[no]a | --[no]adremove
    Controls whether advertising removal is attempted. As distributed, the default is to remove advertising, and it can be negated.

-[no]7bit | -7 | -8bit | -8
    Normally, demime strips output to seven bit. If you typically use a character set which requires that eight bit characters be passed through, set -8, -no7bit or -8bit to turn off the stripping. If you are using plain ascii on a US English mailing list, you probably want to keep the default and strip body output to seven bits.

--[no]eliderfc822
    When a multipart/mixed or a multipart/alternative contains a message/rfc822 or message/news internal section, there exists a question as to what to do with that contained section. It might be the right thing to elide the entire contained section (if you have people who are web TV subscribers who can only reply by attaching the entire original message). It might be the right thing to treat the message as an internal plain text section and to demime it. The default action of the script is to demime the contained message. The alternative is to elide the contained mime section.

-w(=| )(0|1|2) | --warnings-to-syslog(= )(0|1|2)
    Normally, warnings go to syslog. If debug is set, warnings go to stderr. That behaviour can be overridden if you set this. 0 says that warnings should always go to stderr, 1 says that warnings should go to syslog if stderr is not a terminal and to stderr if there is a tty and 2 says to always send warnings to stderr. Normally defaults to 0 if -d is set and 1 if -d is not set.

--uselynx
    This operand causes demime to use the lynx browser to interpret the html sections. The default for this is nouselynx if HTML::FormatText is installed and yes if it is not installed. Your version of lynx must be recent enough to support the --stdin command line argument.

    Lynx is called with these arguments: lynx --stdin --dump --force_html --hiddenlinks=ignore --localhost --image_links --nolist --noredir --noreferer --realm

    It must be in your path for this to work.

--[no]quiet
    This parameter, if specified, will result in some messages that were output to indicate where stuff was elided not being produced. That is, the attachments will be deleted silently instead of having notations put into the message body that an attachment was deleted.

target positional parameter: relay|-|>&=d|'|pipe as argument'
    This required option indicates the mailing address that should get the reformatted message. If you want the message on stdout, use '-' as the address. The address to send to will not be read from the mail file, for security reasons.

    You may specify a list of addresses. Simply insure that they are passed as one token and separated by semicolons. If you use sendmail, an alias such as:

     realuser: "| demime '-;\realuser;otheruser' | other_program"

    may be useful. It will deliver both to the next program in the pipe and to the mail file belonging to the real user that you are aliasing.

    Finally, specifying such as

     realuser: "| demime '-;\realuser;>&=5' 5>>/tmp/trackfile | other_program"

    might be useful for debugging, although hardly as a reliable log since no locking against interlacing is performed. Note that the >&=digit syntax is checked for specifically and that you must arrange to have the file descriptor opened for demime if you use this syntax. Another possibility would be

     realuser: "| demime '-;\realuser;>&=2' | other_program"

    to put a copy of the message on standard error as well.

    If you specify a pipe like

     realuser: "|demime - | other_program"

    and there is a demime failure, other program will have been invoked. It might get just a null input, in which case, it might do the wrong thing. Demime's return code will be ignored as well. In other words, demime might fail to decode the message and return something descriptive on stderr, or might simply want the MTA to requeue, but the return code will be hidden.

    If this possibility bothers you, you can specify the logical equivalent of the above as

     realuser: "|demime  '| other_program'"

    or, as a more complex example:

     realuser: "|demime '-;root;| some_other_program'"|other_program

    Which will cause the MTA to start demime and other_program, and then, when output is ready, demime will send it to standard output, mail it to root, and invoke the pipe and send it to that program as well. I'm now using the simpler of the above two cases to invoke the majordomo programs via wrapper to preserve the demime failure codes, if any.

    This causes demime to simply put whatever you specify as a single token and open its own pipe to it. Note that if there are any shell metacharacters that it will invoke a shell, as per typical perl. You may not specify a pipe with a ';' in it, as the semicolon split is done earlier. If demime fails before it has anything to write, the other_program will never be invoked. Upon normal failure, demime collects return codes from any copies of sendmail it involes, any pipes it invokes, (with special treatment for the first pipe specified) and then any internal return codes.

    smrsh can't deal with the | as a pipe character. Edit the script and change $MAJIC_PIPE_PREFIX_CHAR to some other leading character(s), say '==' (or set it with -x above).

    Then you can specify the alias as

     realuser: "|demime -x '==' '== other_program'"

    and demime will convert it internally to

     realuser: "|demime '| other_program'"

    and then run it as a pipe, as above. This is only a requirement to get around a limitation in smrsh for sendmail.

    If the first pipe has died on a signal, EX_TEMPFAIL will be returned.

    If the first pipe has returned a return code, then that return code will be returned by demime.

    If any other program has returned a non-zero return code, that return code will be returned by demime.

    Demime's internal return code is returned. This may be EX_OK (0) or some other code that indicates a temporary or permanent failure to the MTA. If demime has not yet tried to produce any output, none of the other possibilities will apply.

  What demime will do when faced with different types of input:

text/plain or no content type in mail header
    The content will be passed through without reformatting. A scan will be done to determine if there are uuencoded files instream If found, they will be replaced by a note. Content-type: text all by itself on the header line will be converted to text/plain. Some versions of elm incorrectly send 'text' as a content-type.

text/enriched
    A very simple formatter which is built into demime will attempt to do the mimimum amount of formatting possible. exerpt, lt, nl, and np tags will be respected (np is executed as a double nl. Comments and params will be elided and all other tags will be ignored.

text/html
    The HTTP::FormatText class will be used to format the input into simple text. No additional formatting will be done.

message/rfc822
    An email which is a single message/rfc822 will be expanded such that this piece of e-mail is interpreted as if it were at the top level. If the message/rfc822 has sections, they will be interpreted as if it were at top level. This only is done if the message/rfc822 is the type of the entire body. If there is a multipart/mixed where one of the sections is message/rfc822, this section will be elided, and not considered for interpretation.

    This is only done because some CCMail users produce this structure if they forward a message and then change content. It does not quite seem that this is the right thing to do, but we are trying to be liberal with what we accept while eliminating as much cruft as possible. In general, it seems wrong to recursively flatten included rfc822 sections in a multipart/mixed. So far, the ones I've seen seem to be cruft.

    A message/rfc822 in a multipart/mixed may be flattened or elided. This is dependent on the setting of the EXPAND_MULTIPART_RFC822_SECTION configuration variable in the script header. If expanded, it will be treated as a top level message/rfc822. This switch also determined whether a multipart/related will be flattened or elided.

message/delivery-status
    Treated as text/plain.

message/news
    Treated exactly like message/rfc822.

multipart/alternative
    If a text/plain section is available, it will be displayed. If no text/plain is available, but a text/html is available, it will be interpreted and displayed instead. If no text/plain or text/html is available, but a text/enriched is available, it will be displayed. If none of these are available, a message indicating that the mail cannot be interpreted will be displayed and the mail will be returned to the sender (by returning EX_NOPERM from sysexits.h), assuming that it is being invoked from a sendmail alias. The non-displayed alternetive sections will be silently ignored - that is, no inline message will be displayed. At top level, a header will be added indicating which alternative section was selected.

multipart/mixed or multipart/parallel
    Any items that are text/html, text/enriched, or text/plain will be interpreted inline as if they were stand-alone, one after the other in the order presented. If a multipart/alternative is presented inside of a top level multipart/mixed, that item wll be interpreted according to the rules for a top level multipart/alternative (the most renderable section will be rendered), except that failure to find an interpretable section will result in an inline message being inserted into the output stream rather than a mailbounce, and no header indication will be made of which section was selected for rendering.

    A section of message/rfc822 may be expanded or elided, depending on the setting of EXPAND_MULTIPART_RFC822_SECTION. It is likely that these sections should generally be elided for typical mailing list usage.

    Other mime types will be elided from the output stream and replaced by a message that this has been done unless demime is in quiet mode.

multipart/signed or multipart/related
    At top level, treated like a multipart/mixed. Renderable sections are rendered, and unrenderable sections, specifically the PGP signature, are elided. If the switch that turns on desending into a message/rfc822 is set, the program will descend into any multipart/related it finds, rendering text sections (including one of alternates) and eliding other sections.

Unknown
    If the program does not recognize the top level mime type of the mail then a message will be printed and the mail will be bounced.

FILES
If the installer has left $AD_REMOVE = 1 in the program header (the default) the program will try to read the configuration file named in the variable $junkmail_file, which is set to /usr/pkg/etc/demime_junkmail.cf in the distribution. This file contains lines in the following format:

 [prefix_match]
 /regex/
 /regex/
 [suffix_match]
 /regex/
 /regex/
 [prefix_match_x]
 /regex/
 /regex/
 [center_match_x]
 /regex/
 [suffix_match_x]
 /regex/
 /regex/
 [prefix_match_y]
 /regex/
 /regex/
 [suffix_match_y]
 /regex/
 /regex/
 [prefix_match_z]
 /regex/
 /regex/

The regular expressions must be valid perl regular expressions. Blank lines, lines starting #, ; or //.. are considered comments and ignored.

Prefix_match is meant to contain regular expressions that match "introducers". Many of the advertising signatures use lines of underscores or dashes as introducers to their automatically added advertising signatures. Suffix_match lines contain lines of things following introducers, such as yahoo or bigfoot advertising dreck. The use of introducers is designed to make the possibility of a false match (against someone who uses a legitimate signature that uses, say, a yahoo address) less likely.

More than one [prefix_match] and [suffix_match] section may be specified, so that you can logically group the sections together so as to show that you are eliding particular sections of .signatures. However, the program will mash all [prefix_match] sections and all [suffix_match] sections together. If you have decided to work without "introducers", then specify the [prefix_match] section only.

If you specify a [prefix_match_X] where X can be anything you want, and a [suffix_match_X], where X corresponds with the X specified for [prefix_match_X] this will be treated as a separate tree. The prefix in this list must preceed the suffixes in this list for this group to work and cause the section to be elided. There can be more than one group, of course, that is, you can have [prefix_match_yahoo] with [suffix_match_yahoo] and [prefix_match_altavista] with [suffix_match_altavista].

The effect of this is that once a prefix matches, the next line is checked to see if it matches any suffix in that section. If it does, then the blank lines before the prefix, the prefix, and all lines as long as they match any regexp in the suffix are elided. The eliding happens when the first line does not match the suffix section for that prefix, and that line is then checked against the list of prefixes.

If there is no [center_match] or [suffix_match] for a group, any line that matches any prefix regex is elided.

If a [center_match_X] is part of a match group, then things act a little differently. A pattern in [prefix_match_X] must match for things to kick off. Once a section with a center is active, no sections without a center are looked at. Then a pattern in [center_match_X] must match. Then a pattern in [suffix_match_X] must match. If all this happens, before end of file then the section will be elided when the first suffix pattern matches. However, while a prefix/center/suffix combo is active, there is the possibility of a runaway. That is, because a prefix could match an unrelated line, and not every line in a prefix/center group must match, a false match againse a prefix could cause the rest of the line to not be checked. Until a suffix match occurs, every prefix from every match group without a center is checked against that line. If any of those prefixes match, the program decided that this is a false indication and does not delete the group - it throws away its current state and starts over with that line, checking that line against all prefixes.

If a prefix from more than one prefix match section (without a center) matches, the suffixes for those prefixes are all checked against the next lines. If no suffix for that prefix matches, it is simply removed from the "check" list. But, at this time, if any suffix matches, the first pattern to not match closes the match. The last pattern that contains a matching suffix (that is, the last closed match group) defines the high limit of the group of lines to be elided.

Similarly, if more than one prefix match/center match/suffix match section matches, they are put in an active list and they are all matched against subsequent lines. If the suffixes match before the centers match, they are elided from the match list. But when the first one "closes", (that is, the prefix, center, and suffix match cycle completes) the eliding will be done and the pending match state for all matches will be reset.

Regular expressions must start and end with a "/" (which is actually parsed off so that the patterns can be run without evaling a literal. If you need to specify a modifier, use the (?i) extension syntax (where, as an example, (?i) at the beginning of the pattern makes it case insensitive). See perlre for details on the extension syntax.

The program assembles the mail file section as an array of lines and then works forward through the lines.

As it marches forward, (ignoring all blank lines or lines that consist only of the normal quote sign, the >) it matches all lines, against the regular expressions in [prefix_match], and, if specified, [prefix_match_group1], [prefix_match_group2] etc. If none of these match, it goes on to the next line. All of the prefixes that have a matching regular expression are made "active". If none are active at the end of the cycle, the next line is checked against the prefixes.

This is important because every line, pretty much, is checked against all of the prefixes. The execution speed of this process is directly proportional to the number of prefix regexps (and to a lesser extent, the number of sections in the control file). It is assumed that suffixes will be matched against fewer lines than the prefixes are - but that is up to the user. General enough prefixes will cause the suffixes to be checked a lot.

If there are match sections that have 'center' sections activated on the same line as sections that do not have 'center' sections, the ones without 'center' sections are ignored.

ERROR REPORTING
Some errors, such as mime types that are not parsable, are reflected directly to the mail sender as a bounce. This is slightly unruly, but seems to be the right place to vector such things. The program will put descriptive messages on STDERR and return with EX_NOPERM as described above.

Other errors, such as errors in the format of regular expressions in the control file should not cause mail to bounce but should be reported somewhere. By default, perl's normal warn statement is used by the inner routines to report such problems. This is optionally intercepted and converted to a syslog message. The default behaviour is to report to syslog if STDERR is not a tty device, and to report to STDERR if it is a tty device. This means that the following unexpected behaviour can occur, if the following stage is run from the console:

 demime < mail_input 2>&1 | less

This is, of course, a syntax error. However, the message regarding that is sent to syslog. To change this behaviour, change the WARNINGS_TO_SYSLOG variable by editing the perl script (or set the --warnings_to_syslog command line variable. Setting it to 0 will always send warnings to STDERR, setting it to 1, the default, will work as described above, and setting it to 2 will force all warnings to syslog, even if demime is running from a terminal. If the -d flag is set, warnings always go to STDERR.

Any messages from a 'die' at top level are sent to STDERR. If warnings are being sent to syslog, the error is logged to syslog as well. These errors are typically errors that will not allow the program to continue.

If you need to elide the unix "From " line for archive separation, edit the script and find the variable $PRESERVE_UNIX_FROM_LINE and set it to 0, or set the --nopreservefrom command line switch.

BUGS
There are clearly many more mime types and so forth than I am parsing or am ever likely to parse. Nesting is an issue - I so not parse down into a tree except for the limited case of a multipart/alternative inside of a multipart/mixed or included sections inside of a top level message/rfc822. I suspect that I should parse a mixed inside of a mixed, except that I've never been presented with a test case where a mailer produced a mixed in a mixed that I wanted to keep in the output stream. I do descend more generally into multipart/related and mixed as of 99c, if the user turns on the options that aggressively descend into message/rfc822

This program should be restructured such that it is completely recursive such that it can parse arbitrarialy nested structures, just for neatness sake, even if that ends up not being a good idea. Some structuring has been done, but more needs to be done.

I don't deal with digests at all. For that matter, I am not sure if I should. If someone sends in a message/digest, what should I do? Try to flatten it and convert it to RFC 1153 format by flattening each message as if it were at top level, and eliding headers as appropriate? Is anyone even doing message/digest from a mailing list manager? If someone has a sample they could send me, I'd appreciate it.

People keep on inventing new mime types. The message/signed type probably should be treated like plain text but is not.

Of course, the first bug is mime itself. Mime should have been made transparently downward compatible with existing plain text mail systems or not done at all. That should have, for example, included the ability to verify receiving capability before sending such that only mail that could have been understood would ever be sent. Complex, but the alternative was to break all mail systems, which was the choice that was made.

SEE ALSO
HTML::FormatText. mime(1).

COPYRIGHT
Copyright (c) 1998, 1999 Nick Simicich. All Rights Reserved.

This program is free software; you can redistribute it and/or modify it under the same terms as perl itself. You may not, however, copy code from this module to your own programs without crediting the author.

There is no warranty on this code, nor is there an implied warranty of suitability for purpose. Use at your own risk.

AUTHOR
Nick Simicich <njs@scifi.squawk.com>

If you shoot mime, do you need to do so silently?

AVAILABILITY
The latest version of this package is likely to be available from http://scifi.squawk.com/demime.html