What does the POP3 CAPA UIDL command do?
It checks if the pop3 server understands (has the CAPAbility) the UIDL command.
The response should be "+OK" or "-ERR" depending on wether the server supports the UIDL command.
The UIDL command returns (if supported) an uniqe identify for each message, so a client can identify messages reliably.
See also: rfc2449(CAPA) and rfc1939(POP3).
CAPA is one command. UIDL is another command. You can try them out using telnet to port 110 of the POP server ( telnet:pop.example.com:110 ). After telnet establishes the TCP connection the POP server should send something like "+OK The Microsoft Exchange POP3 service is ready." You can type "CAPA" and hit return, then the POP server should respond with a list of capabilities that it supports (in that state of the session, that is prior to logging in). You can log in by sending "user #name# and hit return, where #name# would be changed to your POP account name. You then type "pass #pw#" and hit return, where #pw# is your password. This sends you password over the network in the clear so someone sniffing the link could easily see your password. Your POP server may require other more secure login. (Don't type in the double quotes in the example above).
Assuming that went well, you can try "CAPA" again now that the session has been established and is in a different state. The list of capabilities may be the same or different depending on the server configuration. At this point you can type "STAT" and hit return. The POP server should return "+OK #x# #y#" where #x# is the number of messages and #y# is the length in bytes of all the messages. Now you can try typing "UIDL" and hit return. the POP server will return a list with "#n# #uid#" where #n# is the message number and #uid# is a unique identifier assigned by the POP server.
Type QUIT and hit return to end the session and close the TCP connection.
The UIDL capability indicates that the optional UIDL command is supported.
POP3 servers may assign a unique number to each incoming mail message. This allows mail to be left on the server after it has been downloaded to the user. Both the mail client and the POP server must support this feature.
According to the POP3 RFC the UIDL command will give you a Unique ID for a message.
The RFC goes on to say:
The unique-id of a message is an arbitrary server-determined string, consisting of one to 70 characters in the range 0x21 to 0x7E, which uniquely identifies a message within a maildrop and which persists across sessions.
The POP3 Exensions RFC says that the CAPA command allows you to query the capabilities of the server.
So the CAPA UIDL command is used to see if a server supports unique IDs.
UIDL is the Unique ID listing capability described in RFC 1939. It means the server supports generating unique IDs for each message to make it easier for the client to handle messages left on the server.
Gives the unique identifier for a message on the POP3 server.
Possible responses: +OK or -ERR
Related
I'm trying to implement Telnet Client using C++ and QT as GUI.
I have no idea to handling the telnet negotiations.
Every telnet command is preceded by IAC, e.g.
IAC WILL SUPPRESS_GO_AHEAD
The following is how I handling the negotiation.
Search for IAC character in received buffer
According to the command and option, response to the request
My questions are described as follows:
It seems that the telnet server won't wait for a client response after a negotiation command is sent.
e.g. (send two or more commands without waiting for client reponse)
IAC WILL SUPPRESS_GO_AHEAD
IAC WILL ECHO
How should I handle such situation? Handle two requests or just the last one?
What the option values would be if I don't response the request? Are they set as default?
Why IAC character(255) won't be treated as data instead of command?
Yes, it is allowed to send out several negotiations for different options without synchronously waiting for a response after each of them.
Actually it's important for each side to try to continue (possibly after some timeout if you did decide to wait for a response) even if it didn't receive a reply, as there are legitimate situations according to the RFC when there shouldn't or mustn't be a reply and also the other side might just ignore the request for whatever reason and you don't have control over that.
You need to consider both negotiation requests the server sent, as they are both valid requests (you may choose to deny one or both, of course).
I suggest you handle both of them (whatever "handling" means in your case) as soon as you notice them, so as not to risk getting the server stuck if it decides to wait for your replies.
One possible way to go about it is presented by Daniel J. Bernstein in RFC 1143. It uses a finite state machine (FSM) and is quite robust against negotiation loops.
A compliant server (the same goes for a compliant client) defaults all negotiable options to WON'T and DON'T (i.e. disabled) at the start of the connection and doesn't consider them enabled until a request for DO or WILL was acknowledged by a WILL or DO reply, respectively.
Not all servers (or clients for that matter) behave properly, of course, but you cannot anticipate all ways a peer might misbehave, so just assume that all options are disabled until enabling them was requested and the reply was positive.
I'll assume here that what you're actually asking is how the server is going to send you a byte of 255 as data without you misinterpreting it as an IAC control sequence (and vice versa, how you should send a byte of 255 as data to the server without it misinterpreting it as a telnet command).
The answer is simply that instead of a single byte of 255, the server (and your client in the opposite direction) sends IAC followed by another byte of 255, so in effect doubling all values of 255 that are part of the data stream.
Upon receiving an IAC followed by 255 over the network, your client (and the server in the opposite direction) must replace that with a single data byte of 255 in the data stream it returns.
This is also covered in RFC 854.
I have a project coming up where I need to send and receive messages through a specific mobile operator, which only provides an SMPP interface. The whole project will be a hosted website. I have already read quite a lot, but I do not yet quite understand what is actually needed from my side to use the protocol.
Should my application try to maintain a constant connection to the smpp?
Can I simply connect, send a message and then disconnect?
Are receiving messages based on push or pull?
Thanks for the help.
SMPP is a peer-to-peer protocol. That should mean that SMS Gateway (your side) and SMSC (your mobile operator) need to have a proper bind/connection established. Even when there are no SMS or DLRs to send/receive, there is a continous exchange of smpp PDU (enquire_link/enquire-link_resp) that ensure that the bind is established.
In detail, if you send an enquire_link PDU and you get no response (enquire_link_resp) the bind is broken. Your sms won't be delivered (will remain enqueued in your gateway store), and you won't receive MOs (incoming sms) or DLRs (delivery report). To re-establish the connection you should re-initiate the connection.
So, my answer would be that you need a constant connection to SMSC.
You are stating you want to receive messages, as a result at least a bind_receiver is needed. Because you don't know when messages are going to come in, you will have to be constantly connected, rather than disconnecting after each event.
With regards to your question about "push or pull" this depends on how you solve the first problem. If you can build a solution that is constantly connected, the result will be a push (the carrier will push it to you as soon as they receive the message). If (for some reason) you cannot maintain a constant connection, you'll end up building a pull mechanism. You'll connect to the carrier ever X seconds to see if they have a message waiting for you.
I do need to highlight 2 pitfalls though:
A number of carriers in the world, do not store or even accept messages if you are not connected, therefore, depending on which carrier you interact with, you might be forced to use a continuous connection.
Most carriers do not allow you to open and close connections in quick succession. Once you disconnect, you can not reconnect for a time frame of X seconds.
Therefore a constant connection is really the way to go. Alternatively, you can look into a company like Nexmo, which will provide you with a HTTP Call every time a message arrives.
I'm not sure which language your developing your application in, but if you use any of the popular languages (Java, PHP, Perl) there are modules out there that handle basic SMPP Connectivity for you. A quick google search for your language and "SMPP Client" will give you a list of references.
my task is to retrieve a set of messages from GMail via POP3 (no IMAP).
I can do RETR MSG #, and it is prohibited to delete.
Fetchmail and procmail are constantly trying to download a same set of new, unread messages (this part goes to ServerFault). Is there a header specially designed to distinguish previously read messages? Or I should do checksum of message body/subject/date?
The POP3 protocol doesn't support a read/seen sort of flag. Some servers support a non-standard header like X-Seen which acts like a read flag, you'll have to use the TOP to get a message's headers and see if it's been set (well more to determine if it's even there).
It's supposed to be up to the client to cover read flags in POP3, but the good news is you don't need to do a checksum, just use the UIDL which will give you a list of non-changing, unique IDs for the messages in the inbox, or if called with a message # will give you the unique ID for the message in that position in the mailbox (since you can't guarantee message position in the mailbox if other client's are accessing and may be deleting).
Try to control messages with message-id
Message message ;
....
String messageId = message.getHeader("message-id")[0];
Say I've got a client-side counter variable. When I send a request to the server, I send this variable's value, then increment it by one (for the next request to the server). The server keeps track of this counter independantly, and checks to make sure that the count sent by the client is 1 more than it's own (server-side) copy of the count. This is all fine and dandy, but consider the following situation:
The client sends, say, 23 to the server.
The server receives 23, validates it, and increments its own counter to 23.
The server returns the All-Okay code to the client
-BUT-
Along the way from the server to the client, the return code gets corrupted. The client thus thinks that the server did not update its counter, and so leaves the client-side counter at 23. From this point onwards, the client and server are out of sync.
Does anybody know any robust schemes that would work in the face of such possible corruption/errors?
Thanks,
Cameron
Instead of using a linearly increasing counter, you can use a random "nonce" value of a good 64 bits of entropy or more. When the server receives a request from the client, the server checks to see whether the nonce matches the last one it sent the client. If so, then the request is processed and the server generates a new random nonce value to send in the response.
The server can keep the last two nonce values to send to the client. If the client sends the old value, then the server assumes that the most recent message to the client might have been lost.
The above method assumes that your goal is to prevent two different clients from using the same credentials to communicate with the server. The advantage of using the nonce method is that the next value can't easily be predicted.
The easy answer is to make one of the client or server the owner of the resource, instead of making both own its own copy of the resource.
If you're using a reliable protocol like TCP though, you don't have to worry about the message not getting to the client.
A good thing to follow when doing client/server work though is to make all operations idempotent. That is to say each function could be called one or many times without any side effects. In this case you wouldn't have an 'increment' function at all. Instead you'd have a 'set' function.
How about just not having the client update its local copy until the server acknowledges the update of the server counter.
So:
Client calculates next value
Client sends next value to server
Server checks that next value is valid (not already seen)
Server updates counter to next value (if needed)
Server notifies client that next value has been received
Client updates local counter to next value
If the server does not receive the client update, the client merely resends the calculated next value. If the client does not receive the next value confirmation, it will resend the next value, but the server having already seen it does not update, but merely acknowledges. Eventually, the client sees the server's message and continues. This covers the case of lost messages.
If you are concerned about corruption, calculate a checksum on the message and send that as well. Recalculate the checksum on receipt and compare it to the one sent. Generally the network stack does this for you, though, so I wouldn't worry too much about it unless your are running your own protocol.
I use Thunderbird to receive email using POP3. I have Thurnderbird configured to leave email on the server. Lets say one day I uses POP3 to retrieve (RETR) 10 email messages, then I logout for the night. Overnight 10 more messages are sent to my mailbox. When I fire up Thunderbird the next morning, the STAT command should show 20 messages. However, Thunderbird should not download the first 10 messages; it should start at message 11 (or the unique identifier or UID for message 11). Thunderbird will send a POP3 UIDL command, then compare the UID's to the UID of the last message Thunderbird retrieved yesterday. It will find that the last UID matches the UIDL list for message 10, then Thunderbird will RETR 11, RETR 12, and so on.
In my case, the POP3 STAT command shows that I have 5379 messages on the POP server. I have already received about 5000 of them. For some reason Thunderbird wants to download all 5379 messages instead of starting at 5001. I am trying to debug this and was looking for the UID that Thunderbird thinks was the last message retrieved.
Does anyone know where Thunderbird (on Windows) stores the last UID, which it will use to compare to the UIDL (list)?
Is there a way to manually set it so I can force Thunderbird to start retrieving somewhere close to 5001?
Thunderbird has a file called popstate.dat that contains the UID, an timestamp (epoch), and a flag. The flag indicates the action that Thunderbird is to perform for the associated message.
Evidently, Thunderbird does not actually work like I described above. I think Thunderbird does the following. It sends a POP3 UIDL command to get a list of the UID's stored on the POP server. It then compares this list to the UID's stored in popstate.dat. Any UID's that are not already in popstate.dat are new messages to be retrieved. The UIDL command previously returned the message number and the associated UID. Thunderbird must then do an POP3 RETR command using the message number associated with the UID's that it has not yet retrieved. Thunderbird must also look at the flag in popstate.dat and take any actions for the associated message. For example, the flag d tells Thunderbird to delete the associated message. The f flag means that Thunderbird has only a truncated part of the message and should retrieve the full message.
At some point Thunderbird updates the popstate.dat with the new messages. I think this happens in a batch update to popstate.dat after all the actions have been completed. That is, if there are 10 new messages to retrieve, popstate.dat is not updated until all 10 messages have been retrieved.
I think my problem resides on the server. Apparently our infrastructure upgraded to a new version of the POP server and new UIDs were assigned on the new version. My popstate.dat had all the old UIDs. The UIDL to the new POP server send a list of 5000+ UIDs that Thunderbird did not have listed in popstate.dat. So, Thunderbird proceeded to download all 5000+ messages. If the new POP server had retained the old UID's then Thunderbird would have seen that I already had retrieved most of the 5000+ messages and would have just downloaded the ones that I did not have. I think most people in my organization use Outlook and do not use POP3, and however the version update was done to the POP server did not cause a problem for those users. Seems like some extra care was needed to ensure the new server had the same UIDs as the old server. Live and learn!
Which version of server software there is?
http://courier.sourcearchive.com/documentation/0.60.0-2/pop3dserver_8c-source.html
00718 ** The UIDL of the message is really just its filename, up to the first MDIRSEP character
May be you just need to replace first part of UIDL in popstate.dat to new ones?
There is an extension for Thunderbird to find duplicates and delete them. e.g. based on message-id (which is usually generated by the original sender of the mail and thus hasn't changed with your infrastructure update).
Perhaps not the answer the OP is exactly looking for, but it may probably help him.
I have had similar problems and to resolve them I have created a small php script to rebuild the popstate file.
You can find it here:
https://github.com/Magentron/rebuild_thunderbird_popstate
Simple PHP script to rebuild the Thunderbird's popstate.dat file. Primarily
useful when Thunderbird crashes and after starts retrieving all messages
that have been kept on the mail server. Default file that will be generated
is popstate.dat.GENERATED. If you change this to the popstate.dat file in
the appropriate Thunderbird directory, please make sure that Thunderbird is
closed, otherwise you might have more problems.
Please note that no attempt has been made to make this the best code I ever
wrote...
NB: Windows support is there, but never tested!
Usage
usage: rebuild_popstate.php [-d] [-i n] [-s] [-v] [-f file] server [ port ]
-c CRLF flag, use when talking to Windows servers
-d debug flag
-f output filename (if popstate.dat, Thunderbird needs to be closed!)
-i ignore the last n messages (for if you don't have them yet)
-s use for secure POP3 (SSL/TLS)
-v verbose flag