22

✏️ Editing: FAQ.Messages_Selection.txt

#!/bin/cat
$Id: FAQ.Messages_Selection.txt,v 1.18 2021/07/04 23:05:24 gilles Exp gilles $

This document is also available online at
https://imapsync.lamiral.info/FAQ.d/
https://imapsync.lamiral.info/FAQ.d/FAQ.Messages_Selection.txt

=======================================================================
                Imapsync tips to select messages.
=======================================================================

Questions answered in this FAQ are:

Q. What messages imapsync syncs by default?

Q. Is there a way we can specify a date range to sync emails? 
   If yes, can you please share an example?

Q. Is there a way we can specify an age to sync emails? 
   If yes, can you please share some examples?

Q. I want to sync messages based on their UID.

Q. Can I migrate only mails with attachments?

Q. How can I move messages marked \Deleted from all folders to
   a dedicated folder?

Q. What are the selection criteria available with --search option?

=======================================================================
Q. What messages imapsync syncs by default?

R1. By default, Imapsync syncs all messages, except duplicates.
    If you want to sync duplicates, add the option --syncduplicates

R2. You can alter the default "all messages selected" by using one or 
    more of the following options:

Extract from the README https://imapsync.lamiral.info/README

OPTIONS/message selection

--maxsize      int  : Skip messages larger  (or equal) than  int  bytes
     --minsize      int  : Skip messages smaller (or equal) than  int  bytes

--maxage       int  : Skip messages older than  int days.
                           final stats (skipped) don't count older messages
                           see also --minage
     --minage       int  : Skip messages newer than  int  days.
                           final stats (skipped) don't count newer messages
                           You can do (+ zone are the messages selected):
                           past|----maxage+++++++++++++++>now
                           past|+++++++++++++++minage---->now
                           past|----maxage+++++minage---->now (intersection)
                           past|++++minage-----maxage++++>now (union)

--search       str  : Selects only messages returned by this IMAP SEARCH
                           command. Applied on both sides.
                           For a complete set of what can be search see
                           https://imapsync.lamiral.info/FAQ.d/FAQ.Messages_Selection.txt

--search1      str  : Same as --search but for selecting host1 messages only.
     --search2      str  : Same as --search but for selecting host2 messages only.
                           So --search CRIT equals --search1 CRIT --search2 CRIT

--maxlinelength int : skip messages with a line length longer than  int  bytes.
                           RFC 2822 says it must be no more than 1000 bytes but
                           real life servers and email clients do more.

--useheader    str  : Use this header to compare messages on both sides.
                           Ex: Message-ID or Subject or Date.
     --useheader    str    and this one, etc.

--usecache          : Use cache to speed up next syncs. Not set by default.
     --nousecache        : Do not use cache. Caveat: --useuid --nousecache creates
                           duplicates on multiple runs.
     --useuid            : Use UIDs instead of headers as a criterion to recognize
                           messages. Option --usecache is then implied unless
                           --nousecache is used.

=======================================================================
Q. Is there a way we can specify a date range to sync emails? 
   If yes, can you please share an example?

R. Yes, a date range is possible with the --search option.

imapsync ... --search "SENTSINCE 1-Jan-2010" 
  
or 
  
  imapsync ... --search "SENTBEFORE 31-Dec-2010"

or
  
  imapsync ... --search "SENTSINCE 1-Jan-2010 SENTBEFORE 31-Dec-2010"

Months are specified like this:
  
   Jan
   Feb
   Mar
   Apr
   May 
   Jun
   Jul 
   Aug
   Sep
   Oct 
   Nov
   Dec

=======================================================================
Q. Is there a way we can specify an age to sync emails? 
   If yes, can you please share some examples?

R. Yes, with the --maxage or the --minage option.

Have in mind that the options --maxage and --minage can NOT be 
   combined successfully with the --search command. If the --search 
   option is present on the command line then --maxage and --minage 
   are ignored.

Example.1 Sync only messages less than 2 days old:

imapsync ... --maxage 2

Example.2 Sync only messages more than 2 days old:

imapsync ... --minage 2

Example.3 Sync only messages more than 30 days old and less than 365 days old:

imapsync ... --minage 30 --maxage 365

Example.4 Sync only messages less than 30 days old or more than 365 days old:

imapsync ... --maxage 30 --minage 365

Full explanation:

--maxage      <int>    : Skip messages older than <int> days.
                         final stats (skipped) don't count older messages
                         see also --minage
--minage      <int>    : Skip messages newer than <int> days.
                         final stats (skipped) don't count newer messages
                         You can do (+ are the messages selected):
                         past|----maxage+++++++++++++++>now
                         past|+++++++++++++++minage---->now
                         past|----maxage+++++minage---->now (intersection)
                         past|++++minage-----maxage++++>now (union)

Caveat.1 By default,

Option --maxage is implemented as a --search SENTSINCE 
RFC 3501 says: SENTSINCE <date>
Messages whose [RFC-2822] Date: header (disregarding time and
timezone) is within or later than the specified date.

Option --minage is implemented as a --search SENTBEFORE
RFC 3501 says: SENTBEFORE <date>
Messages whose [RFC-2822] Date: header (disregarding time and
timezone) is earlier than the specified date.

If --noabletosearch is on then --minage and --maxage deal
with the internal dates given by a FETCH imap command but 
not the Date: header. Internal date is the arrival date 
in the mailbox. Same remark for --noabletosearch1 and
--noabletosearch2 but only for one side then.

=======================================================================
Q. I want to sync messages based on their UID.

R. First have in mind that UIDs are unique only per folder, so work this
   way only with one folder at a time, with --folder option.

To show UIDs, there is the --debugLIST parameter.

imapsync ...  --debugLIST

To sync only a part of all messages, selected by UIDs 
from 10000 to 11000:

imapsync ... --search1 "UID 10000:11000"

To sync from INBOX only 3 messages UIDs 20000 20002 20004:

imapsync ... --search1 'OR OR UID 20000 UID 20002 UID 20004' --folder INBOX

To sync all messages from INBOX except 3 messages 
UIDs 20000 20002 20004:

imapsync ... --search1 'NOT OR OR UID 20000 UID 20002 UID 20004' --folder INBOX

If you search n UIDs then you have to put n-1 OR in the search line,
that's IMAP.

=======================================================================
Q. Can I migrate only mails with attachments?

R. Use:

imapsync ... --search "HEADER Content-Disposition attachment"

imapsync ... --search "HEADER Content-Type multipart/mixed"

or more generally:

imapsync ... --search "OR HEADER Content-Disposition attachment HEADER Content-Type multipart/mixed"

The last one doesn't work on all imap servers, not on Gmail at least (tested 2020/06/16)

=======================================================================
Q. How can I move messages marked \Deleted from all folders to
   a dedicated folder?

R. To move \Deleted messages from all folders to a specific folder, 
   let's call it Trash, use:

imapsync ... --search DELETED --regextrans2 "s/.*/Trash/"

=======================================================================
Q. What are the selection criteria available with --search option?

R. The list of search criteria are listed below, an excerpt from RFC3501.

https://tools.ietf.org/html/rfc3501#section-6.4.4
...
      The SEARCH command searches the mailbox for messages that match
      the given searching criteria.  Searching criteria consist of one
      or more search keys.  The untagged SEARCH response from the server
      contains a listing of message sequence numbers corresponding to
      those messages that match the searching criteria.
      When multiple keys are specified, the result is the intersection
      (AND function) of all the messages that match those keys.  For
      example, the criteria DELETED FROM "SMITH" SINCE 1-Feb-1994 refers
      to all deleted messages from Smith that were placed in the mailbox
      since February 1, 1994.  A search key can also be a parenthesized
      list of one or more search keys (e.g., for use with the OR and NOT
      keys).

In all search keys that use strings, a message matches the key if
      the string is a substring of the field.  The matching is
      case-insensitive.

The defined search keys are as follows.  Refer to the Formal
      Syntax section for the precise syntactic definitions of the
      arguments.

<sequence set>
         Messages with message sequence numbers corresponding to the
         specified message sequence number set.

ALL
         All messages in the mailbox; the default initial key for
         ANDing.

ANSWERED
         Messages with the \Answered flag set.

BCC <string>
         Messages that contain the specified string in the envelope
         structure's BCC field.

BEFORE <date>
         Messages whose internal date (disregarding time and timezone)
         is earlier than the specified date.

BODY <string>
         Messages that contain the specified string in the body of the
         message.

CC <string>
         Messages that contain the specified string in the envelope
         structure's CC field.

DELETED
         Messages with the \Deleted flag set.

DRAFT
         Messages with the \Draft flag set.

FLAGGED
         Messages with the \Flagged flag set.

FROM <string>
         Messages that contain the specified string in the envelope
         structure's FROM field.

HEADER <field-name> <string>
         Messages that have a header with the specified field-name (as
         defined in [RFC-2822]) and that contains the specified string
         in the text of the header (what comes after the colon).  If the
         string to search is zero-length, this matches all messages that
         have a header line with the specified field-name regardless of
         the contents.

KEYWORD <flag>
         Messages with the specified keyword flag set.

LARGER <n>
         Messages with an [RFC-2822] size larger than the specified
         number of octets.

NEW
         Messages that have the \Recent flag set but not the \Seen flag.
         This is functionally equivalent to "(RECENT UNSEEN)".

NOT <search-key>
         Messages that do not match the specified search key.

OLD
         Messages that do not have the \Recent flag set.  This is
         functionally equivalent to "NOT RECENT" (as opposed to "NOT
         NEW").

ON <date>
         Messages whose internal date (disregarding time and timezone)
         is within the specified date.

OR <search-key1> <search-key2>
         Messages that match either search key.

RECENT
         Messages that have the \Recent flag set.

SEEN
         Messages that have the \Seen flag set.

SENTBEFORE <date>
         Messages whose [RFC-2822] Date: header (disregarding time and
         timezone) is earlier than the specified date.

SENTON <date>
         Messages whose [RFC-2822] Date: header (disregarding time and
         timezone) is within the specified date.

SENTSINCE <date>
         Messages whose [RFC-2822] Date: header (disregarding time and
         timezone) is within or later than the specified date.

SINCE <date>
         Messages whose internal date (disregarding time and timezone)
         is within or later than the specified date.

SMALLER <n>
         Messages with an [RFC-2822] size smaller than the specified
         number of octets.

SUBJECT <string>
         Messages that contain the specified string in the envelope
         structure's SUBJECT field.

TEXT <string>
         Messages that contain the specified string in the header or
         body of the message.

TO <string>
         Messages that contain the specified string in the envelope
         structure's TO field.

UID <sequence set>
         Messages with unique identifiers corresponding to the specified
         unique identifier set.  Sequence set ranges are permitted.

UNANSWERED
         Messages that do not have the \Answered flag set.

UNDELETED
         Messages that do not have the \Deleted flag set.

UNDRAFT
         Messages that do not have the \Draft flag set.

UNFLAGGED
         Messages that do not have the \Flagged flag set.

UNKEYWORD <flag>
         Messages that do not have the specified keyword flag set.

UNSEEN
         Messages that do not have the \Seen flag set.

...

search          = "SEARCH" [SP "CHARSET" SP astring] 1*(SP search-key)
                    ; CHARSET argument to MUST be registered with IANA
...

search-key      = "ALL" / "ANSWERED" / "BCC" SP astring /
                  "BEFORE" SP date / "BODY" SP astring /
                  "CC" SP astring / "DELETED" / "FLAGGED" /
                  "FROM" SP astring / "KEYWORD" SP flag-keyword /
                  "NEW" / "OLD" / "ON" SP date / "RECENT" / "SEEN" /
                  "SINCE" SP date / "SUBJECT" SP astring /
                  "TEXT" SP astring / "TO" SP astring /
                  "UNANSWERED" / "UNDELETED" / "UNFLAGGED" /
                  "UNKEYWORD" SP flag-keyword / "UNSEEN" /
                    ; Above this line were in [IMAP2]
                  "DRAFT" / "HEADER" SP header-fld-name SP astring /
                  "LARGER" SP number / "NOT" SP search-key /
                  "OR" SP search-key SP search-key /
                  "SENTBEFORE" SP date / "SENTON" SP date /
                  "SENTSINCE" SP date / "SMALLER" SP number /
                  "UID" SP sequence-set / "UNDRAFT" / sequence-set /
                  "(" search-key *(SP search-key) ")"

seq-number      = nz-number / "*"
                    ; message sequence number (COPY, FETCH, STORE
                    ; commands) or unique identifier (UID COPY,
                    ; UID FETCH, UID STORE commands).
                    ; * represents the largest number in use.  In
                    ; the case of message sequence numbers, it is
                    ; the number of messages in a non-empty mailbox.
                    ; In the case of unique identifiers, it is the
                    ; unique identifier of the last message in the
                    ; mailbox or, if the mailbox is empty, the
                    ; mailbox's current UIDNEXT value.
                    ; The server should respond with a tagged BAD
                    ; response to a command that uses a message
                    ; sequence number greater than the number of
                    ; messages in the selected mailbox.  This
                    ; includes "*" if the selected mailbox is empty.

seq-range       = seq-number ":" seq-number
                    ; two seq-number values and all values between
                    ; these two regardless of order.
                    ; Example: 2:4 and 4:2 are equivalent and indicate
                    ; values 2, 3, and 4.
                    ; Example: a unique identifier sequence range of
                    ; 3291:* includes the UID of the last message in
                    ; the mailbox, even if that value is less than 3291.

sequence-set    = (seq-number / seq-range) *("," sequence-set)
                    ; set of seq-number values, regardless of order.
                    ; Servers MAY coalesce overlaps and/or execute the
                    ; sequence in any order.
                    ; Example: a message sequence number set of
                    ; 2,4:7,9,12:* for a mailbox with 15 messages is
                    ; equivalent to 2,4,5,6,7,9,12,13,14,15
                    ; Example: a message sequence number set of *:4,5:7
                    ; for a mailbox with 10 messages is equivalent to
                    ; 10,9,8,7,6,5,4,5,6,7 and MAY be reordered and
                    ; overlap coalesced to be 4,5,6,7,8,9,10.

See also the excellent "Introduction to IMAP" by Nick Babcock
https://nbsoftsolutions.com/blog/introduction-to-imap

See also "WITHIN Search Extension to the IMAP Protocol"
https://tools.ietf.org/html/rfc5032

...
   search-key =/ ( "OLDER" / "YOUNGER" ) SP nz-number
                  ; search-key defined in RFC 3501

4.  Example

C: a1 SEARCH UNSEEN YOUNGER 259200
   S: a1 * SEARCH 4 8 15 16 23 42

Search for all unseen messages within the past 3 days, or 259200
   seconds, according to the server's current time.
   
=======================================================================
=======================================================================

MR SKB