I use Thunderbird to receive email using POP3. I have Thurnderbird configured to leave email on the server. Lets say one day I uses POP3 to retrieve (RETR) 10 email messages, then I logout for the night. Overnight 10 more messages are sent to my mailbox. When I fire up Thunderbird the next morning, the STAT command should show 20 messages. However, Thunderbird should not download the first 10 messages; it should start at message 11 (or the unique identifier or UID for message 11). Thunderbird will send a POP3 UIDL command, then compare the UID's to the UID of the last message Thunderbird retrieved yesterday. It will find that the last UID matches the UIDL list for message 10, then Thunderbird will RETR 11, RETR 12, and so on.
In my case, the POP3 STAT command shows that I have 5379 messages on the POP server. I have already received about 5000 of them. For some reason Thunderbird wants to download all 5379 messages instead of starting at 5001. I am trying to debug this and was looking for the UID that Thunderbird thinks was the last message retrieved.
Does anyone know where Thunderbird (on Windows) stores the last UID, which it will use to compare to the UIDL (list)?
Is there a way to manually set it so I can force Thunderbird to start retrieving somewhere close to 5001?
Thunderbird has a file called popstate.dat that contains the UID, an timestamp (epoch), and a flag. The flag indicates the action that Thunderbird is to perform for the associated message.
Evidently, Thunderbird does not actually work like I described above. I think Thunderbird does the following. It sends a POP3 UIDL command to get a list of the UID's stored on the POP server. It then compares this list to the UID's stored in popstate.dat. Any UID's that are not already in popstate.dat are new messages to be retrieved. The UIDL command previously returned the message number and the associated UID. Thunderbird must then do an POP3 RETR command using the message number associated with the UID's that it has not yet retrieved. Thunderbird must also look at the flag in popstate.dat and take any actions for the associated message. For example, the flag d tells Thunderbird to delete the associated message. The f flag means that Thunderbird has only a truncated part of the message and should retrieve the full message.
At some point Thunderbird updates the popstate.dat with the new messages. I think this happens in a batch update to popstate.dat after all the actions have been completed. That is, if there are 10 new messages to retrieve, popstate.dat is not updated until all 10 messages have been retrieved.
I think my problem resides on the server. Apparently our infrastructure upgraded to a new version of the POP server and new UIDs were assigned on the new version. My popstate.dat had all the old UIDs. The UIDL to the new POP server send a list of 5000+ UIDs that Thunderbird did not have listed in popstate.dat. So, Thunderbird proceeded to download all 5000+ messages. If the new POP server had retained the old UID's then Thunderbird would have seen that I already had retrieved most of the 5000+ messages and would have just downloaded the ones that I did not have. I think most people in my organization use Outlook and do not use POP3, and however the version update was done to the POP server did not cause a problem for those users. Seems like some extra care was needed to ensure the new server had the same UIDs as the old server. Live and learn!
Which version of server software there is?
http://courier.sourcearchive.com/documentation/0.60.0-2/pop3dserver_8c-source.html
00718 ** The UIDL of the message is really just its filename, up to the first MDIRSEP character
May be you just need to replace first part of UIDL in popstate.dat to new ones?
There is an extension for Thunderbird to find duplicates and delete them. e.g. based on message-id (which is usually generated by the original sender of the mail and thus hasn't changed with your infrastructure update).
Perhaps not the answer the OP is exactly looking for, but it may probably help him.
I have had similar problems and to resolve them I have created a small php script to rebuild the popstate file.
You can find it here:
https://github.com/Magentron/rebuild_thunderbird_popstate
Simple PHP script to rebuild the Thunderbird's popstate.dat file. Primarily
useful when Thunderbird crashes and after starts retrieving all messages
that have been kept on the mail server. Default file that will be generated
is popstate.dat.GENERATED. If you change this to the popstate.dat file in
the appropriate Thunderbird directory, please make sure that Thunderbird is
closed, otherwise you might have more problems.
Please note that no attempt has been made to make this the best code I ever
wrote...
NB: Windows support is there, but never tested!
Usage
usage: rebuild_popstate.php [-d] [-i n] [-s] [-v] [-f file] server [ port ]
-c CRLF flag, use when talking to Windows servers
-d debug flag
-f output filename (if popstate.dat, Thunderbird needs to be closed!)
-i ignore the last n messages (for if you don't have them yet)
-s use for secure POP3 (SSL/TLS)
-v verbose flag
Related
After having done various research I cannot find the best way to achieve this:
I have a Node Express server, providing services to various users
I would like any user to be able to send an email to user-id#mydomain.com
Upon reception, I would like to trigger a specific task in the server, to process the email body.
Option 1
I was first hoping to find some kind of SMTP node package that I could simply embed in the server, and configure it with the various email addresses to be accepted, and with a callback function to trigger the task whenever an email arrives. Does this exist?
Option 2
Another option would be to install a SMTP server (ideally in a Docker container, and in any case on Linux), to handle the storage of each user mailbox. My Node server would then periodically check each user mailbox via POP3 or IMAP, and trigger the task whenever an email is found. But this seems a bit overkill to me:
I don't need to store the emails once the task is performed
This would be less responsive than having a callback like in the first option, and would require a periodic check of all users, whereas in practice, such emails will arrive very sporadically.
In this approach, what would you recommend as a dockerized SMTP server, and as a POP3/IMAP node package to retrieve and process emails?
Option 3?
Would there be any other approach?
Any recommendation welcome!!
Many thanks!
It looks like the Node smtp-server package can do the job. Combined with mailparser I can retrieve a Javascript object with the email structure fully parsed.
If processing emails in an outlook mailbox, is it possible to use some kind of ID to record the last message processed avoiding the need to iterate through all messages every time?
I'm trying to write a quick 'attachment stripper' to free up mailbox space and don't want to have to iterate through every message in a mail folder each time the code is ran. At a pinch I could use the mail folder name and the message date, but what happens if a folder is renamed, or messages moved? (Besides, I REALLY hate dates!!! formats to start with, and I have recently experienced randomly changing environment settings in my work environment - a citrix remote desktop - so now have a near phobia of the damn things!)
I was hoping there was some kind of message ID I could use but can see neither the MAPI ID nor the Outlook Mail ID being suitable here (though I could well be wrong). Is there a means of doing this?
If you need to avoid reprocessing previously processed emails, you have no recourse but to either record the email's unique identifiers externally for later checking, or flagging emails with a specific value (such as adding a Category, setting a flag, adding a custom field, setting an internal field, etc.) to avoid processing them later.
I am using redis via booksleeve on c#/.NET
Lets say i want to send an unimplemented command like SCRIPT FLUSH or EVAL. How do i do so with booksleeve? Also i'm not sure if echo is implemented so using that as an example would be cool.
Flush is there - try under .Server (from memory - not near a PC). For the others - there is no "run custom command" API, but I have contributions in my inbox (waiting to be merged) that add these. They might even be attachments in the issue list (I honestly can't remember). Or wait a few days.
I am using Peter Huber's POP3 client to connect to gmail and download messages.
The inboxes being accessed are transactional inboxes used only for code-access. That is, a message comes in with a order file attached, code will process it and then delete the message. One stipulation of the code though was a DEBUG flag, which if set would prevent the code from deleting the message so that you can run the program again later without the debug flag and reprocess the message. So, in my code I have
If Not Arguments.Debug Then pop.DeleteEmail(eid)
This works fine. Problem is, even when not deleting the message, running the program a second time will not re-retrieve the message, even though if I login to gmail and look at the inbox, it is still there. The only way I can get the program to see the message again is to forward the message back to the same inbox. But in Peter's code I do not see anywhere where he is keeping track of seen messages between sessions.
Is this something that is done on gmail's end? Refusing to deliver a message to the same client a second time? If so, is there any way I can change my gmail account so that it will always show all messages in the inbox to a client when retrieving the list of messages, even ones already "seen"? I don't see anything in the gmail settings screen.
UPDATE: I tried adding a method to send a RSET command to the server, as per this comment on the codeproject page. I then call my new Reset() method after retrieving my messages but before disconnecting, ... but I still have the same problem.
Okay... found a "sort of" answer after reading through pages of the comments on the codeproject project.
According to this comment, the RSET command does not actually do anything when you are dealing with gmail's servers.
The "answer" is to prepend your username with the string "recent:", so instead of logging in with [myaccount#gmail.com] you log in with [recent:myaccount#gmail.com]. Rather hackish, ... but it works.
What does the POP3 CAPA UIDL command do?
It checks if the pop3 server understands (has the CAPAbility) the UIDL command.
The response should be "+OK" or "-ERR" depending on wether the server supports the UIDL command.
The UIDL command returns (if supported) an uniqe identify for each message, so a client can identify messages reliably.
See also: rfc2449(CAPA) and rfc1939(POP3).
CAPA is one command. UIDL is another command. You can try them out using telnet to port 110 of the POP server ( telnet:pop.example.com:110 ). After telnet establishes the TCP connection the POP server should send something like "+OK The Microsoft Exchange POP3 service is ready." You can type "CAPA" and hit return, then the POP server should respond with a list of capabilities that it supports (in that state of the session, that is prior to logging in). You can log in by sending "user #name# and hit return, where #name# would be changed to your POP account name. You then type "pass #pw#" and hit return, where #pw# is your password. This sends you password over the network in the clear so someone sniffing the link could easily see your password. Your POP server may require other more secure login. (Don't type in the double quotes in the example above).
Assuming that went well, you can try "CAPA" again now that the session has been established and is in a different state. The list of capabilities may be the same or different depending on the server configuration. At this point you can type "STAT" and hit return. The POP server should return "+OK #x# #y#" where #x# is the number of messages and #y# is the length in bytes of all the messages. Now you can try typing "UIDL" and hit return. the POP server will return a list with "#n# #uid#" where #n# is the message number and #uid# is a unique identifier assigned by the POP server.
Type QUIT and hit return to end the session and close the TCP connection.
The UIDL capability indicates that the optional UIDL command is supported.
POP3 servers may assign a unique number to each incoming mail message. This allows mail to be left on the server after it has been downloaded to the user. Both the mail client and the POP server must support this feature.
According to the POP3 RFC the UIDL command will give you a Unique ID for a message.
The RFC goes on to say:
The unique-id of a message is an arbitrary server-determined string, consisting of one to 70 characters in the range 0x21 to 0x7E, which uniquely identifies a message within a maildrop and which persists across sessions.
The POP3 Exensions RFC says that the CAPA command allows you to query the capabilities of the server.
So the CAPA UIDL command is used to see if a server supports unique IDs.
UIDL is the Unique ID listing capability described in RFC 1939. It means the server supports generating unique IDs for each message to make it easier for the client to handle messages left on the server.
Gives the unique identifier for a message on the POP3 server.
Possible responses: +OK or -ERR