Arduino+WiFly shield failing to communicate to xively - api

I have a problem in delivering sensor data to the xively API via an Arduino Uno v3 and a Sparkfun WiFly shield. The problem is not in the hardware, or in the WiFly shield library since I can deliver the data to the Paraimpu server just fine.
The most fundamental problem is that the xively library will not work with the sparkfun WiFly library. The relevant declarations (suggested by xively in the documentation) are:
WiFlyClient client;
XivelyClient xivelyclient(client);
this will not work since the WiFlyClient declaration expects a server and port, hence I modified this to:
byte server[] = {173,203,98,29}; //api.xively.com IP address
WiFlyClient client(server,80);
XivelyClient xivelyclient(client);
This gives me an error on compilation of :
Xively_sketch2_aug20a:60: error: no matching function for call to 'XivelyClient::XivelyClient(WiFlyClient&)'
/Users/paultravers/Documents/Arduino/libraries/xively/XivelyClient.h:11: note: candidates are: XivelyClient::XivelyClient(Client&)
/Users/paultravers/Documents/Arduino/libraries/xively/XivelyClient.h:9: note: XivelyClient::XivelyClient(const XivelyClient&)
At this point I am stuck, and my attempts to modify the various libraries to try to reconcile this issue have come to no avail - mostly because it is above my skill level and I really don't know what I am doing.
To get round this, I have written the code to build the put request and send that to the API, using the template of the code that runs successfully to send data to Paraimpu.
I open a connection (either to api.xively.com or to 173.203.98.29; it makes no difference at this stage) and send the following :
PUT /v2/feeds/<feed ID redacted>.json
Host: api.xively.com
Content-Type: application/json
User-Agent : Xively-Arduino-Lib/1.0
X-ApiKey: < API key redacted >
Content-Length: 197
{"version":"1.0.0","datastreams" : [{"id":"TEMPERATURE_CHANNEL" , "current_value" : "29.00"},{"id":"LIGHT_SENSOR_CHANNEL","current_value":"541.00"},{"id":"ALARM_CHANNEL","current_value":"0.00"}]}
Terminating with a blank line.
Needless to say I have set up channels in xively with those names. But this does not work - I don't get the expected return, and the channels don't update. If I read the returning input on the connection it is just a long string of numbers. Can anyone see anything wrong in the format of this request?

i might take a look at WiFlyClient's documentation...maybe you need to set the port/etc in some other way...than extend WiFlyClient
class WiFlyClient2 : public WiFlyClient {
WiFlyClient2(int[] ip,int port) : WiFlyClient() {
//setup up your parent by calling it's functions here
}
}
then it should work like:
byte server[] = {173,203,98,29}; //api.xively.com IP address
WiFlyClient client(server,80);
XivelyClient xivelyclient(client);
note: i never use wifly/etc, i just use arduino...and i run into troubles like this too :)

I don't have a WiFly shield to test with. But as you said in your comment to Zoltan, I don't think that the WiFly client works in the same way that the Arduino Ethernet or Wifi do. This means that the Xively library likely does not work with the WiFly client.
This means that your approach of writing your own HTTP request is probably the way to go. I looked through your request and it looks exactly correct to me. Does anything show up in the Xively Workbench Request Log when you send the request?
One thing you could try is to use the CSV format instead. This can be better for testing since you avoid the possibility of a minor JSON error causing your entire request to not work. To do this change your .json to .csv and then simply make your body a comma separated list of datastream,value. Your body should look something like this:
TEMPERATURE_CHANNEL,29.00
LIGHT_SENSOR_CHANNEL,541.00
ALARM_CHANNEL,0.00
If this does not work it would be helpful if you could post what the return is, that would help us help you. Good luck!

One thing I noticed in your code is that you don't send the HTTP version in your request.
You should try sending:
PUT /v2/feeds/<feed ID redacted>.json HTTP/1.1
I was getting 405 errors when writing to Xively because I incorrectly had HTTP/1.0 as the version in my put requests.
There's also an example from SparkFun's WiFly library that shows writing to Thingspeak which is helpful to make sure you're using the WiFlyClient instance correctly.

Related

Not able to use string over 260 characters as a segment of URL in .NET Core

I'm making a request that works great and acts as supposed to. The actual authorization is provided using headers and working as expected too. This is the URL of it.
https://localhost:44385/api/security/check
By coincidence, I happened to replace the verbatim string check with the actual token, so the URL changed to
https://localhost:44385/api/security/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ...
All in all, the token happens to be 475 characters long. Then, when executing that call, I get the error message as follows.
Error: connect ECONNREFUSED 127.0.0.1:44300
I don't understand the issue and the status code 400 tells me only that the request is bad. Is it purely due ot the length of the URL? It seems like a bit too short (although there is a limitation for that but we're talking about a few thousands characters)...
The signature of the receiving method in the controller looks like this. It resides in the controller with path Security.
[HttpHead("{check}"), Authorize]
public IActionResult IsAuthorized(string check) { ... }
I also tried GET instead of HEAD with the same result. It's difficult to learn more about the error based on 400 Bad request only. It's a bit like something went wrong somewhere kind of error.
After some experimenting, I can confirm that it's not the length of the URL as such but rather the length of the segment between slashes. The first request works, the other does too but the third doesn't. The xxx part is precisely 260 characters and **yyy* part is precisely 261.
https://localhost:44385/api/test/xxx
https://localhost:44385/api/testtest/xxx
https://localhost:44385/api/test/yyy
What is this about?! It's like string in a method in my WebAPI can't be longer than 260 characters. Not 256, which at least would make some kind of sense...
Googling gave a veeery wide range of vastly spread hits and gave me nothing that I could relate to. Postman provides pretty much the same, limited information. The browser's network tab give even less.
A bit confused how to get to know more, how to diagnose it further and/or what to google for. Since it's a non-problem for the production environment, I can't bother my colleagues - the question is purely academic.
The limit you're hitting is UrlSegmentMaxLength (260).
This is all the way down in Http.Sys and only configurable in the
registry.
https://support.microsoft.com/en-us/help/820129/http-sys-registry-settings-for-windows
Workaround: break it up into multiple path segments, or move it to the
query or body. Or use Kestrel without IIS.
Resource: https://github.com/aspnet/AspNetCore/issues/2823#issuecomment-360921436
Here's a related post:
Setting UrlSegmentMaxLength from commadline

How to implement ALLO command on Apache FTP?

We have an embedded Apache FTP server running in a gateway for several years. It always worked without problems.
But now a customer is trying to connect with a device of a brand that we've never had before, and contrary to all other clients so far, that thing sends the ALLO command in advance to make sure the server has enough space.
But Apache FTP doesn't seem to know that command. the trace log states:
RECEIVED: ALLO 77482
SENT: 502 Command ALLO not implemented.
following which the client cuts the connection.
The command is also not present in the Apache documentation:
https://mina.apache.org/ftpserver-project/ftpserver_commands.html
So the question is, can I plug my own implementation into the server somehow?
Just to be clear, I'm not asking how to implement the functionality. Just how I can pass my own implementation to Apache FTP for use. If that is possible without touching the source code.
Since the application in question has been running very stable for a long time, I would really hate to tear the Apache FTP server out of there and embed another one...
Well, that was surprisingly simple once I dug myself through to the right code.
The implementation of a command is simple enough, in this case I've just started with a stub for testing:
class ALLO : AbstractCommand() {
override fun execute(session: FtpIoSession, context: FtpServerContext, request: FtpRequest) {
session.write(LocalizedFtpReply.translate(session, request, context,
FtpReply.REPLY_200_COMMAND_OKAY, "ALLO", "bring it!"))
}
}
Inherit AbstractCommand, override execute and write a response to the session.
The question is of course then how to make the server aware of the implementation, which also turns out to be really simple, although there sure as hell doesn't seem to be any documentation around. But you can just instantiate a CommandFactoryFactory, map your implementation, build the CommandFactory and set it in the FtpServerFactory:
val commandFactoryFactory = CommandFactoryFactory()
commandFactoryFactory.addCommand("ALLO", ALLO())
serverFactory.commandFactory = commandFactoryFactory.createCommandFactory()

HTTPS proxy with support for chunked-encoded requests

I'm developing a simple HTTPS proxy (written in Python) which receives POST/GET requests/responses, applies some transformation and finally forwards the result to the recipient.
I need to handle chunked-encoded requests/responses in a "streaming" fashion, meaning that as soon as a chunk is received the proxy transforms it and forwards it to the recipient.
Before deciding to support chunked-encoded requests, I've been using mitmproxy http://mitmproxy.org/ and it worked perfectly. Unfortunately, I noticed that it waits until the entire body is received before letting me handle the response/request.
How can I implement a proxy supporting chunked-encoded requests/responses? Has anyone of you ever done something like this?
Thanks
EDIT: MORE INFO ON MY USE CASE
I need to handle POST requests and GET responses.
In the POST request I receive a JSON object and I have to encrypt some of its values.
In the GET response I receive a JSON object and I have to decrypt some of its values.
Till now, the following code has worked perfectly:
def handle_request(self, r):
if(r.method=='POST'):
// encryption of r.get_form_urlencoded()
def handle_response(self, r):
if(r.request.method=='GET'):
// decryption of r.content
How can I do the same thing with single chunks?
EDIT: UPDATES
After evaluating different solutions, I decided to go for Squid (proxy) + ICAP (content adaptation).
I've successfully configured Squid and the performance are just great. Unfortunately, I can't find a suitable ICAP server (in Python, if possible) for doing content adaptation (modification). I thought this one https://github.com/netom/pyicap could do the job but looks like it doesn't read the body of myPOST requests.
Do you guys know a Python ICAP server that I can use together with Squid?
Thanks
The answer below is outdated. You can now pass --stream to mitmproxy, whose behaviour is explained in the mitmproxy documentation.
mitmproxy developer here. This is definitely a feature we want for mitmproxy as well, but it's not that trivial and probably not coming very soon. If you really want to implement that yourself, I can recommend two things:
If you have a very specific use case, you can employ libmproxy.protocol.http.HTTPRequest.from_stream for parsing the header and do the body processing yourself.
If you do not want to modify the request/response body, you may find it sufficient to modify mitmproxy itself. In a nutshell, you would need to read the request/response without content (see 1.), modify it to your needs, pass it to the server and then delegate control to the libmproxy.protocol.tcp (see https://github.com/mitmproxy/mitmproxy/blob/master/libmproxy/proxy/server.py#L169)
If you have further questions, don't hesistate to ask here or on mitmproxy's IRC channel.
Re Comment #1:
You can't take too much out of mitmproxy, but at least you get delegate the header parsing & processing.
# ...accept request, socket.makefile() etc...
req = HTTPRequest.from_stream(client_conn.rfile, include_content=False)
# manually forward to the server (req._assemble_head())
# manually receive response body chunk by chunk and forward it to the server, see
# https://github.com/mitmproxy/netlib/blob/master/netlib/http.py#L98
resp = HTTPResponse.from_stream(server_conn.rfile, include_content=False)
# manually forward headers
# manually process body and forward
That being said, this is a fairly complex topic. Eventually, you're better off hacking that directly into libmproxy.protocol.http.HTTPHandler.
Another option, depending on your use case again: Use mitmproxy, set the conntype to tcp and forward traffic as-is and use regex replacements on the content in libmproxy.protocol.tcp . Probably the easiest way, but the most hacky one.
If you can provide some context, I may guide you further in the right direction.
Re Comment #2:
Before we get to the main part: JSON is a really bad choice for streaming/chunking as long as you don't want to encrypt the complete JSON object and treat it as a single string. You should definitely consider something like tnetstrings if you only want to encrypt parts.
Apart from that, hooking into read_chunk works, but first you need to get to the point where you can actually receive chunks over the line. Then, it's as simple as reading the single chunks, encrypting them and forwarding them.

Sending a file from a java client to a server using wcf method?

I want to build a wcf web service so that the client and the server would be able to transfer files between each other. Do you know how I can achieve this? I think I should turn it into a byte array but I have no idea how to do that. The file is also quite big so I must turn on streamed response.
It sounds like you're on the right track. A quick search of the interwebz yielded this link: http://www.codeproject.com/Articles/166763/WCF-Streaming-Upload-Download-Files-Over-HTTP
Your question indicates that you want to send a file from a java client to a WCFd endpoint, but the contents of your question indicate that this should be a bidirectional capability. If this is the case, then you'll need to implement a service endpoint on your client as well. As far as that is concerned, I cannot be of much help, but there are resources out there like this SO question: In-process SOAP service server for Java
As far as practical implementation, I would think that using these two links you should be able to produce some code for your server and client.
As far as reading all bytes of a file, in C# you can use: File.ReadAllBytes It should work as in the following code:
//Read The contents of the file indicated
string fileName = "/path/to/some/file";
//store the binary in a byte array
byte[] buffer = File.ReadAllBytes(fileName);
//do something with those bytes!
Be sure to use the search function in the future:

xbuf_frurl does not work properly without server header of content length?

I try to get some info from other sites with xbuf_frurl.
I got some site OK but some Not OK.
By Now, I still can not make sure what is going wrong.
But some sites are missing the content length header.
Who can tell whether xbuf_frurl() relies on the (potentially missing) content length header, esp. when growing the buffer?
xbuf_frurl() is indeed reading a body IF an HTTP content-length header is present. It will not try to decode chunked responses.
To deal with those servers using chunked replies, use the G-WAN curl.c example provided with the distribution. With libcurl you have even the opportunity to use SSL/TLS.
If that's not resolving your problem, the only way to troubleshoot this kind of issues is to give a non-working example, with both the full request that you have sent and the full reply received from the server.
That's why the xbuf_xcat("%v") format has been added: to give hexdumps, even with binary replies.
Edit your question and add this information to let people help you with a well-defined problem.