This is a strange bug that we started seeing about a year ago. At first, I only occasionally noticed it on my dev machine, but now it's been starting to appear in production, which is problematic.
We use Ubuntu (11.04), and Mono 2.6.7 in production (I can also repro with Mono 2.10.x), inside apache, using mod_mono.
Basically, sometimes (very hard to reproduce), when apache starts the application, SolrNet decides to lower case the entire URL it transmits to the solr server. If the application is in this state, it stays this way until it is restarted (and occasionally requires a couple of restarts to clear up.) We might go for 20 - 50 or more restarts without seeing this problem come up. or sometimes it will happen every 2 or 3.
A good url looks like this:
INFO: [] webapp=/solr path=/select params={sort=Creative.PromotionalScore+desc&start=0&q=*:*&?=&qt=standard&fq={!tag%3DCreative.GalleryReviewStatus}Creative.GalleryReviewStatus:Approved&fq={!tag%3DCreative.SectionIncludedTarget}Creative.SectionIncludedTarget:220358+OR+(NOT+Creative.SectionIncludedTarget:[*+TO+*]+*:*)&fq={!tag%3DActive}Active:true&fq={!tag%3DCreative.ShowInGallery}Creative.ShowInGallery:true&fq={!tag%3DCreative.Size}Creative.Size:"Rectangle"&fq={!tag%3DRecordType}RecordType:FiveToOne.Gallery.Rmx.Creative&rows=12}
A bad url looks like this:
http://solrServer:8080/solr/select?qt=standard&fq=%7b!tag%3dcreative.galleryreviewstatus%7dcreative.galleryreviewstatus%3aapproved&fq=%7b!tag%3dcreative.sectionincludedtarget%7dcreative.sectionincludedtarget%3a306433+or+(not+creative.sectionincludedtarget%3a%5b*+to+*%5d+*%3a*)&fq=%7b!tag%3dactive%7dactive%3atrue&fq=%7b!tag%3dcreative.showingallery%7dcreative.showingallery%3atrue&fq=%7b!tag%3dcreative.size%7dcreative.size%3a%22rectangle%22&fq=%7b!tag%3drecordtype%7drecordtype%3afivetoone.gallery.rmx.creative&sort=creative.promotionalscore+desc&rows=18&start=0&q=*%3a*&?
(first, I apologize, these two URLs are extracted from different stages of the pipe, all I have access to at the moment.)
When the bad url is submitted, Solr throws a fatal exception, complaining of an unknown field:
HTTP Status 400 - can not sort on undefined field: creative.promotionalscore
type Status report
message can not sort on undefined field: creative.promotionalscore
description The request sent by the client was syntactically incorrect (can not sort on undefined field: creative.promotionalscore).
Having used Solr and SolrNet in production for four years now, all I can say is that I've never seen this on .NET, so I'm guessing it's a bug in Mono.
In order to pinpoint where exactly the URL is becoming lowercase, I'd put several logging points, for example here, here, and here.
After this logging is in place, once the bug occurs, the log should be analyzed, logging should be probably refined, etc, until the exact source of the bug is located.
Hopefully then the bug can be reproduced, reported, fixed, etc.
Related
I'm making a request that works great and acts as supposed to. The actual authorization is provided using headers and working as expected too. This is the URL of it.
https://localhost:44385/api/security/check
By coincidence, I happened to replace the verbatim string check with the actual token, so the URL changed to
https://localhost:44385/api/security/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ...
All in all, the token happens to be 475 characters long. Then, when executing that call, I get the error message as follows.
Error: connect ECONNREFUSED 127.0.0.1:44300
I don't understand the issue and the status code 400 tells me only that the request is bad. Is it purely due ot the length of the URL? It seems like a bit too short (although there is a limitation for that but we're talking about a few thousands characters)...
The signature of the receiving method in the controller looks like this. It resides in the controller with path Security.
[HttpHead("{check}"), Authorize]
public IActionResult IsAuthorized(string check) { ... }
I also tried GET instead of HEAD with the same result. It's difficult to learn more about the error based on 400 Bad request only. It's a bit like something went wrong somewhere kind of error.
After some experimenting, I can confirm that it's not the length of the URL as such but rather the length of the segment between slashes. The first request works, the other does too but the third doesn't. The xxx part is precisely 260 characters and **yyy* part is precisely 261.
https://localhost:44385/api/test/xxx
https://localhost:44385/api/testtest/xxx
https://localhost:44385/api/test/yyy
What is this about?! It's like string in a method in my WebAPI can't be longer than 260 characters. Not 256, which at least would make some kind of sense...
Googling gave a veeery wide range of vastly spread hits and gave me nothing that I could relate to. Postman provides pretty much the same, limited information. The browser's network tab give even less.
A bit confused how to get to know more, how to diagnose it further and/or what to google for. Since it's a non-problem for the production environment, I can't bother my colleagues - the question is purely academic.
The limit you're hitting is UrlSegmentMaxLength (260).
This is all the way down in Http.Sys and only configurable in the
registry.
https://support.microsoft.com/en-us/help/820129/http-sys-registry-settings-for-windows
Workaround: break it up into multiple path segments, or move it to the
query or body. Or use Kestrel without IIS.
Resource: https://github.com/aspnet/AspNetCore/issues/2823#issuecomment-360921436
Here's a related post:
Setting UrlSegmentMaxLength from commadline
One of my clients has a PHP script that kept crashing inexplicably. After hours of research, I determined if you send any PHP script a variable (either through GET or POST) that contains " having t", or escaped for the URL "+having+t", it crashes the script and returns a "403 forbidden error". To test it, I made a sample script with the entire contents:
<?php echo "works";
I put it live (temporarily) here: http://primecarerefer.com/test/test.php
Now if you try sending it some data like: http://primecarerefer.com/test/test.php?x=+having+x
It fails. The last letter can be any letter and it will still crash, but changing any other letter makes the script load fine. What would cause this and how can it be fixed? The link is live for now if anyone wants to try out different combinations.
PS - I found that if I get the 403 error a bunch of times in a row, the sever blocks me for 15 minutes.
I had this type of issue on a webserver that ran apache mod_security, but it was very poorly configured, actually mod_security has very bad default regex rules, which are very easy to trip with valid POST or GET data.
To be clear, this has nothing to do with PHP or HTML, it's about POST and GET data passing through mod_security, almost certainly, and mod_security rejecting the request because it believes it is an sql injection attempt.
You can edit the rules yourself depending on the server access, but I don't believe you can do anything, well, if it's mod_security, I know you can't do anything via PHP to get around this.
/etc/httpd/conf.d/mod_security.conf (old path, it's changed, but it gives the idea)
Examples of the default rules:
SecFilter "delete[[:space:]]+from"
SecFilter "insert[[:space:]]+into"
SecFilter "select.+from"
These are samples of the rules
https://www.howtoforge.com/apache_mod_security
here they trip the filter:
http://primecarerefer.com/test/test.php?x=%20%22%20%20select%20from%22
Note that the article is very old and the rules actually are quite differently structured now, but the bad regex remains, ie: select[any number of characters, no matter how far removed, or close]from will trip it, any sql that matches these loose rules will trip it.
But since editing those default files requires access to them, and also assumes they won't be altered in an upgrade of apache mod_security at some point, it's not a good way to fix the problem I found, moving to a better, more professionally setup, hoster, fixed those issues for us. But it does help if you talk to the hosting support to know what the cause of the issue is.
In fact 'having' is not irrelevant at all, it's part of sql injection filters in the regex rules in the security filters run on POST/GET. We used to hit this all the time when admins would edit CMS pages, which would trigger invariably some sql filter, since any string of human words would invariably contain something like 'select.*from' or 'insert.*into' etc.
This mod_security issue used to drive me bonkers trying to debug why backend edit form updates would just hang, until I finally realized it was badly done generic regex patterns in the mod_security file itself.
In a sense, this isn't an answer, because the only fix is going into the server and either editing the rules file, which is pretty easy, or disabling mod_security, or moving to a web hoster that doesn't use those bad generic defaults.
We are seeing this warning messages in our logs
javax.naming.PartialResultException: Unprocessed Continuation Reference(s); remaining name 'dc=global,dc=com'
It appears whenever users log-in to our application.
As per this SO post, it can be resolved, by setting Context.REFERRAL to follow. But it increases the search time from 1 second to 4 seconds.
In fact you can refer this SO post, it says using follow slows down the search.
So my question is, what is the best way to get rid of this exception from our logs without affecting performance?.
Another possible solution that may work is to change the port number (assuming this is a GC server):
If you were using the port 389 change it to 3268
If you were using the port 636 change it to 3269
This may work because (and I quote):
A GC (global catalog) server returns referrals on 389 to refer to the
greater AD "forest", but acts like a regular LDAP server on 3268 (and
3269 for LDAPS)
It worked for me.
I found this solution in the Shibboleth Users list, answered by Paul Caskey (all the credit to him).
You can check the conversation on this link:
https://lists.internet2.edu/sympa/arc/shibboleth-users/2008-06/msg00039.html
OK. You will be seeing this exception, when your search returns referral and you set to ignore the referral.
[Referral: When you search in AD, if AD thinks there are more information available in another place, it returns a referral [place to find more info] along with your search results.]
You could avoid this exception by setting Context.REFERRAL to follow. Then it would search in the referral also [That's why it takes more time to return result].
But in my case the referral is invalid and returned an another exception.
I fixed this issue by changing the baseDN (search base) to be more specific. E.g. ou=users,dc=mydomain,dc=com. Now I'm not seeing this exception, because it doesn't return any referral.
Both of the other two answers work by following any referral that is returned in the LDAP query. That's probably the recommended way in most situations.
However, if the requirement is if fact to use "ignore" so not to follow referrals, then a javax.naming.PartialResultException: Unprocessed Continuation Reference(s) will always be returned whenever a referral is found.
The solution is to also ignore the javax.naming.PartialResultException (by catching and swallowing it) when Context.REFERRAL is set to ignore. I recommend to at least log a warning though.
I assume this somehow awkward implementation of "ignore" was made to make sure that the caller is aware of the fact that partial results are really accepted.
I am struggling (in some sense) to determine which HTTP method is more appropriate for rebooting a remote resource: GET or PUT?
On one hand, it seems more semantic to call http://tools.serviceprovider.net/canopies/d34db33fc4f3?reboot=true because one might want to GET a representation of a freshly rebooted canopy.
On the other hand, a reboot is not 'safe' (nor is it necessarily idempotent, but then a canopy or modem is not just a row in a database) so it might seem more semantic to PUT the canopy into a state of rebooting, then have the server return a 202 to indicate that the reboot was initiated and is processing.
I have been reading up on HTTP/1.1, REST, HATEOAS, and other related concepts over the last week, so I am still putting the pieces together. Could a more seasoned developer please weigh in and confirm or dispel my hunch?
A GET doesn't seem appropriate because a GET is expected, like you said, to be "safe". i.e. no action other than retrieval.
A PUT doesn't seem appropriate because a PUT is expected to be idempotent. i.e. multiple identical operations cause same side-effects as as a single operation. Moreover, a PUT is usually used to replace the content at the request URI with the request body.
A POST appears most appropriate here. Because:
A POST need not be safe
A POST need not be idempotent
It also appears meaningful in that you are POSTing a request for a reboot (much like submitting a form, which also happens via POST), which can then be processed, possibly leading to a new URI containing reboot logs/results returned along with a 303 See Other status code.
Interestingly, Tim Bray wrote a blog post on this exact topic (which method to use to tell a resource representing a virtual machine to reboot itself), in which he also argued for POST. At the bottom of that post there are links to follow-ups on that topic, including one from none other than Roy Fielding himself, who concurs.
Rest is definitely not HTTP. But HTTP definitely does not have only four (or eight) methods. Any method is technically valid (even if as an extension method) and any method is RESTful when it is self describing — such as ‘LOCK’, ‘REBOOT’, ‘DELETE’, etc. Something like ‘MUSHROOM’, while valid as an HTTP extension, has no clear meaning or easily anticipated behavior, thus it would not be RESTful.
Fielding has stated that “The REST style doesn’t suggest that limiting the set of methods is a desirable goal. [..] In particular, REST encourages the creation of new methods for obscure operations” and that “it is more efficient in a true REST-based architecture for there to be a hundred different methods with distinct (non-duplicating), universal semantics.”
Sources:
http://xent.com/pipermail/fork/2001-August/003191.html
http://tech.groups.yahoo.com/group/rest-discuss/message/4732
With this all in mind I am going to be 'self descriptive' and use the REBOOT method.
Yes, you could effectively create a new command, REBOOT, using POST. But there is a perfectly idempotent way to do reboots using PUT.
Have a last_reboot field that contains the time at which the server was last rebooted. Make a PUT to that field with the current time cause a reboot if the incoming time is newer than the current time. If an intermediate server resends the PUT, no problem -- it has the same value as the first command, so it's a no-op.
You might want to get the current time from the server you're rebooting, unless you know that everyone is reasonably time-synced.
Or you could just use a times_rebooted count, eliminating the need for a clock. A PUT times_rebooted: 4 request will cause a reboot if times_rebooted is currently 3, but not if it's 4 or 5. If the current value is 2 and you PUT a 4, that's an error.
The only advantage to using time, if you have a clock, is that sometimes you care about when it happened. You could of course have BOTH a times_rebooted and a last_reboot_time, letting times_rebooted be the trigger.
I'm working on a site and to help catch errors that we may not hear about, I've created a custom 500 error page.
This page basically records information about the current situation and logs it including the following:
Request.Servervariables("URL")
But, the log seem to actually be providing information about the location of the error.asp file instead of the actual file causing the error. And it doesn't seem to pick up Server.GetLastError().
Any ideas on how to ensure these scripts pick up the errors and deatils about the page causing the error and not the page that is used for 500 errors?
NOTE: When there's an error, the url in the address bar is always the address fo the page causing the error, but the log shows the error handler page 'error.asp'.
I would follow what Dee said, but also be aware that there was something finicky with IIS7 (or 7.5). I can't remember exactly, but you have to do something special to make sure it works on IIS7. Check out this article. IIS7 breaks the server.getlasterror and there is a workaround provided.
Also a cool thing to do is to email yourself those errors. So in your custom 500 asp script just fire off an email with the details. Depends on how critical errors are to your program, but it's good to be in the loop rather than have another log to worry about.