Maximum threads issue - restlet

To begin with, I checked the discussions regarding this issue and couldn't find an answer to my problem and that's why I'm opening this question.
I've set up a web service using restlet 2.0.15.The implementation is only for the server. The connections to the server are made through a webpage, and therefore I didn't use ClientResource.
Most of the answers to the exhaustion of the thread pool problem suggested the inclusion of
#exhaust + #release
The process of web service can be described as a single function.Receive GET requests from the webpage, query the database, frame the results in XML and return the final representation. I used a Filter to override the beforeHandle and afterHandle.
The code for component creation code:
Component component = new Component();
component.getServers().add(Protocol.HTTP, 8188);
component.getContext().getParameters().add("maxThreads", "512");
component.getContext().getParameters().add("minThreads", "100");
component.getContext().getParameters().add("lowThreads", "145");
component.getContext().getParameters().add("maxQueued", "100");
component.getContext().getParameters().add("maxTotalConnections", "100");
component.getContext().getParameters().add("maxIoIdleTimeMs", "100");
component.getDefaultHost().attach("/orcamento2013", new ServerApp());
component.start();
The parameters are the result of a discussion present in this forum and modification by my part in an attempt to maximize efficiency.
Coming to the Application, the code is as follows:
#Override
public synchronized Restlet createInboundRoot() {
// Create a router Restlet that routes each call to a
// new instance of HelloWorldResource.
Router router = new Router(getContext());
// Defines only one route
router.attach("/{taxes}", ServerImpl.class);
//router.attach("/acores/{taxes}", ServerImplAcores.class);
System.out.println(router.getRoutes().size());
OriginFilter originFilter = new OriginFilter(getContext());
originFilter.setNext(router);
return originFilter;
}
I used an example Filter found in a discussion here, too. The implementation is as follows:
public OriginFilter(Context context) {
super(context);
}
#Override
protected int beforeHandle(Request request, Response response) {
if (Method.OPTIONS.equals(request.getMethod())) {
Form requestHeaders = (Form) request.getAttributes().get("org.restlet.http.headers");
String origin = requestHeaders.getFirstValue("Origin", true);
Form responseHeaders = (Form) response.getAttributes().get("org.restlet.http.headers");
if (responseHeaders == null) {
responseHeaders = new Form();
response.getAttributes().put("org.restlet.http.headers", responseHeaders);
responseHeaders.add("Access-Control-Allow-Origin", origin);
responseHeaders.add("Access-Control-Allow-Methods", "GET,POST,DELETE");
responseHeaders.add("Access-Control-Allow-Headers", "Content-Type");
responseHeaders.add("Access-Control-Allow-Credentials", "true");
response.setEntity(new EmptyRepresentation());
return SKIP;
}
}
return super.beforeHandle(request, response);
}
#Override
protected void afterHandle(Request request, Response response) {
if (!Method.OPTIONS.equals(request.getMethod())) {
Form requestHeaders = (Form) request.getAttributes().get("org.restlet.http.headers");
String origin = requestHeaders.getFirstValue("Origin", true);
Form responseHeaders = (Form) response.getAttributes().get("org.restlet.http.headers");
if (responseHeaders == null) {
responseHeaders = new Form();
response.getAttributes().put("org.restlet.http.headers", responseHeaders);
responseHeaders.add("Access-Control-Allow-Origin", origin);
responseHeaders.add("Access-Control-Allow-Methods", "GET,POST,DELETE"); //
responseHeaders.add("Access-Control-Allow-Headers", "Content-Type");
responseHeaders.add("Access-Control-Allow-Credentials", "true");
}
}
super.afterHandle(request, response);
Representation requestRepresentation = request.getEntity();
if (requestRepresentation != null) {
try {
requestRepresentation.exhaust();
} catch (IOException e) {
// handle exception
}
requestRepresentation.release();
}
Representation responseRepresentation = response.getEntity();
if(responseRepresentation != null) {
try {
responseRepresentation.exhaust();
} catch (IOException ex) {
Logger.getLogger(OriginFilter.class.getName()).log(Level.SEVERE, null, ex);
} finally {
}
}
}
The responseRepresentation does not have a #release method because it crashes the processes giving the warning WARNING: A response with a 200 (Ok) status should have an entity (...)
The code of the ServerResource implementation is the following:
public class ServerImpl extends ServerResource {
String itemName;
#Override
protected void doInit() throws ResourceException {
this.itemName = (String) getRequest().getAttributes().get("taxes");
}
#Get("xml")
public Representation makeItWork() throws SAXException, IOException {
DomRepresentation representation = new DomRepresentation(MediaType.TEXT_XML);
DAL dal = new DAL();
String ip = getRequest().getCurrent().getClientInfo().getAddress();
System.out.println(itemName);
double tax = Double.parseDouble(itemName);
Document myXML = Auxiliar.getMyXML(tax, dal, ip);
myXML.normalizeDocument();
representation.setDocument(myXML);
return representation;
}
#Override
protected void doRelease() throws ResourceException {
super.doRelease();
}
}
I've tried the solutions provided in other threads but none of them seem to work. Firstly, it does not seem that the thread pool is augmented with the parameters set as the warnings state that the thread pool available is 10. As mentioned before, the increase of the maxThreads value only seems to postpone the result.
Example: INFO: Worker service tasks: 0 queued, 10 active, 17 completed, 27 scheduled.
There could be some error concerning the Restlet version, but I downloaded the stable version to verify this was not the issue.The Web Service is having around 5000 requests per day, which is not much.Note: the insertion of the #release method either in the ServerResource or OriginFilter returns error and the referred warning ("WARNING: A response with a 200 (Ok) status should have an entity (...)")
Please guide.
Thanks!

By reading this site the problem residing in the server-side that I described was resolved by upgrading the Restlet distribution to the 2.1 version.
You will need to alter some code. You should consult the respective migration guide.

Related

Accessing HttpContext.Session from static method

I am getting following error when accessing HttpContext.Session from static method placed in separate task:
Session has not been configured for this application or request.
I used this article to implement access to HttpContext outside the controller
From controller I invoke this static method that used to retrieve image data:
public static void CreateDummyGallery(Gallery gallery)
{
Logger.LogDebug(LogModule.Dummy, $"Starting gallery creation.");
Task.Factory.StartNew(() =>
{
try
{
List<DummyPicture> pictures;
using (var context = new MyzeumContext())
{
int top = 10;
pictures = context.DummyPictures.FromSql($"SELECT * FROM dummypictures ORDER BY RAND() LIMIT {top}").ToList();
}
Logger.LogDebug(LogModule.Dummy, $"Starting retrieving images.");
Parallel.ForEach(pictures, picture => {
using (WebClient client = new WebClient())
{
}
});
Logger.LogDebug(LogModule.Dummy, $"Done retrieving images.");
}
catch(Exception e)
{
Logger.LogError(LogModule.Server, e.Message, e);
}
});
}
The problem occurs in Logger.LogDebug() because this is where I access HttpContext:
public void LogDebug(LogModule module, string message, Exception stackTrace = null)
{
Log record = new Log();
record.Module = module;
record.ThreadId = Environment.CurrentManagedThreadId;
record.SessionId = HttpContextHelper.Current?.Session?.Id;
record.Message = message;
record.Logged = DateTime.UtcNow;
if(stackTrace != null)
{
record.Message += $" :{stackTrace.StackTrace}";
}
queue.Enqueue(record);
}
The problem 99% occurs in the first call inside task:
Logger.LogDebug(LogModule.Dummy, $"Starting retrieving images.");
BUT, right after application starts this whole task block works fine and does not throw any exception. Problem starts after following requests.

Access Raw Request Body

I'm trying to access a request's raw input body/stream in ASP.net 5. In the past, I was able to reset the position of the input stream to 0 and read it into a memory stream but when I attempt to do this from the context the input stream is either null or throws an error (System.NotSupportedException => "Specified method is not supported.").
In the first example below I can access the raw request in a controller if I declare the controller method's parameter object type as dynamic. For various reasons, this is not a solution and I need to access the raw request body in an authentication filter anyways.
This Example Works, But Is Not a Reasonable Solution:
[HttpPost("requestme")]
public string GetRequestBody([FromBody] dynamic body)
{
return body.ToString();
}
Throws Error:
[HttpPost("requestme")]
public string GetRequestBody()
{
var m = new MemoryStream();
Request.Body.CopyTo(m);
var contentLength = m.Length;
var b = System.Text.Encoding.UTF8.GetString(m.ToArray());
return b;
}
Throws Error:
[HttpPost("requestme")]
public string GetRequestBody()
{
Request.Body.Position = 0;
var input = new StreamReader(Request.Body).ReadToEnd();
return input;
}
Throws Error:
[HttpPost("requestme")]
public string GetRequestBody()
{
Request.Body.Position = 0;
var input = new MemoryStream();
Request.Body.CopyTo(input);
var inputString = System.Text.Encoding.UTF8.GetString(input.ToArray());
return inputString;
}
I need to access the raw request body of every request that comes in for an API that I am building.
Any help or direction would be greatly appreciated!
EDIT:
Here is the code that I would like to read the request body in.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Microsoft.AspNet.Mvc;
using Microsoft.AspNet.Http;
namespace API.Filters
{
public class CustomAuthorizationAttribute : Attribute, IAuthorizationFilter
{
public CustomAuthorizationAttribute()
{ }
public void OnAuthorization(AuthorizationContext context)
{
if (context == null)
throw new ArgumentNullException("OnAuthorization AuthorizationContext context can not be null.");
else
{
if (this.AuthorizeCore(context.HttpContext) == false)
{
// Do Other Stuff To Check Auth
}
else
{
context.Result = new HttpUnauthorizedResult();
}
}
}
protected virtual bool AuthorizeCore(HttpContext httpContext)
{
var result = false;
using (System.IO.MemoryStream m = new System.IO.MemoryStream())
{
try
{
if (httpContext.Request.Body.CanSeek == true)
httpContext.Request.Body.Position = 0;
httpContext.Request.Body.CopyTo(m);
var bodyString = System.Text.Encoding.UTF8.GetString(m.ToArray());
return CheckBody(bodyString); // Initial Auth Check returns true/false <-- Not Shown In Code Here on Stack Overflow
}
catch (Exception ex)
{
Logger.WriteLine(ex.Message);
}
}
return false;
}
}
}
This code would be accessed when a call is made to a controller method marked with the CustomAuthorization attribute like so.
[Filters.CustomAuthorizationAuthorization]
[HttpPost]
public ActionResult Post([FromBody]UserModel Profile)
{
// Process Profile
}
Update
The information below is pretty outdated by now. Due to performance reasons this is not possible by default, but fortunately can be changed. The latest solution should be to enable request buffering with EnableBuffering:
Request.EnableBuffering();
See also this blog post for more information: https://devblogs.microsoft.com/aspnet/re-reading-asp-net-core-request-bodies-with-enablebuffering/.
Old, outdated answer for reference
The implementation of Request.Body depends on the controller action.
If the action contains parameters it's implemented by Microsoft.AspNet.WebUtilities.FileBufferingReadStream, which supports seeking (Request.Body.CanSeek == true). This type also supports setting the Request.Body.Position.
However, if your action contains no parameters it's implemented by Microsoft.AspNet.Loader.IIS.FeatureModel.RequestBody, which does not support seeking (Request.Body.CanSeek == false). This means you can not adjust the Position property and you can just start reading the stream.
This difference probably has to do with the fact that MVC needs to extract the parameters values from the request body, therefore it needs to read the request.
In your case, your action does not have any parameters. So the Microsoft.AspNet.Loader.IIS.FeatureModel.RequestBody is used, which throws an exception if you try to set the Position property.
**Solution**: either do not set the position or check if you actually _can_ set the position first:
if (Request.Body.CanSeek)
{
// Reset the position to zero to read from the beginning.
Request.Body.Position = 0;
}
var input = new StreamReader(Request.Body).ReadToEnd();
The exceptions you see in your three last snippets are the direct consequence of trying to read the request body multiple times - once by MVC 6 and once in your custom code - when using a streamed host like IIS or WebListener. You can see this SO question for more information: Read body twice in Asp.Net 5.
That said, I'd only expect this to happen when using application/x-www-form-urlencoded, since it wouldn't be safe for MVC to start reading the request stream with lengthy requests like file uploads. If that's not the case, then it's probably a MVC bug you should report on https://github.com/aspnet/Mvc.
For workarounds, you should take a look at this SO answer, that explains how you can use context.Request.ReadFormAsync or add manual buffering: Read body twice in Asp.Net 5
app.Use(next => async context => {
// Keep the original stream in a separate
// variable to restore it later if necessary.
var stream = context.Request.Body;
// Optimization: don't buffer the request if
// there was no stream or if it is rewindable.
if (stream == Stream.Null || stream.CanSeek) {
await next(context);
return;
}
try {
using (var buffer = new MemoryStream()) {
// Copy the request stream to the memory stream.
await stream.CopyToAsync(buffer);
// Rewind the memory stream.
buffer.Position = 0L;
// Replace the request stream by the memory stream.
context.Request.Body = buffer;
// Invoke the rest of the pipeline.
await next(context);
}
}
finally {
// Restore the original stream.
context.Request.Body = stream;
}
});
I just had this same issue. Remove the parameters from the method signature, and then read the Request.Body Stream how you want to.
You need to call Request.EnableRewind() to allow the stream to be rewound so you can read it.
string bodyAsString;
Request.EnableRewind();
using (var streamReader = new StreamReader(Request.Body, Encoding.UTF8))
{
bodyAsString = streamReader.ReadToEnd();
}
I Know this my be late but in my case its Just I had a problem in routing as bellow
At startup.cs file I was beginning the routing with /api
app.MapWhen(context => context.Request.Path.StartsWithSegments(new PathString("/api")),
a =>
{
//if (environment.IsDevelopment())
//{
// a.UseDeveloperExceptionPage();
//}
a.Use(async (context, next) =>
{
// API Call
context.Request.EnableBuffering();
await next();
});
}
//and I was putting in controller
[HttpPost]
[Route("/Register", Name = "Register")]
//Just Changed the route to start with /api like my startup.cs file
[HttpPost]
[Route("/api/Register", Name = "Register")]
//and now the params are not null and I can ready the body request multiple

Dropwizard / JerseyClient ignored JsonProperty when sending http request

I have two REST services implemented with Dropwizard-0.8.
Both share an API dependency with following POJO:
public class Report{
private String text;
#JsonProperty("t")
public String getText()
{
return text;
}
public void setText(String tx)
{
text = tx;
}
}
My Server has a rest recourse:
#POST
#Consumes(MediaType.APPLICATION_JSON + ";charset=UTF-8")
#Produces(MediaType.TEXT_PLAIN + ";charset=UTF-8")
#Timed
public Response receive(Report dto) {
//do some stuff with dto
}
My Client has a method :
sendReport(report);
with:
private void sendReport(Report report) {
final String uri = "http://localhost:8080/.....";
Response response = null;
try {
response = client.target(uri).request().post(Entity.entity(report, MediaType.APPLICATION_JSON), Response.class);
final int status = response.getStatus();
if (status != Status.ACCEPTED.getStatusCode()) {
final StatusType statusInfo = response.getStatusInfo();
throw new SomeException();
}
}
catch (Exception e) {
LOGGER.error(e.getMessage());
}
finally {
if (response != null) {
response.close();
}
}
}
The Client is made in the Dropwizard application class with:
service.client = new JerseyClientBuilder(env).using(conf.getJerseyClient()).withProvider(JacksonJaxbJsonProvider.class).build(getName());
env.jersey().register(service);
Where 'service' is my rest class calling the 'sendReport' method.
Problem
When I call the rest service of my server from a browser or with curl etc it works perfectly as expected with following messagebody:
{"t":"some text for the server"}
But when I run my application to call the rest service I get a 400 "unable to process JSON".
Debugging and the log messages showed me that the application sends the following JSON to my server:
{"text":"some text for the server"}
Which leads to the error that Jackson cant find a property "text".
Why is the JerseyClient ignoring the JsonProperty annotation?
From what I understand you using Entity.entity from jersey which has no idea about the #JsonProperty annotation(which is from jackson library) . What you need to do is do serialisation using a jackson library and give it to post call .

org.apache.commons.io.FileCleaningTracker does not delete temp files unless explicitly calling System.gc()?

I am working on a upload image feature for my web app, and am having a strange issue with the "FileCleaningTracker" from apache commons fileupload. I have a ImageUploadService with a instance variable FileCleaningTracker, then I have a upload method that creates an instance of DiskFileItemFactory and then references the FileCleaningTracker, after the upload method completes successfully, I set the FileCleaningTracker of DiskFileItemFactory to null, so i would expect the DiskFileItemFactory to be garbage collected and then the underlying subclass of PhantomReference in FileCleaningTracker will be notified hence delete the temp file the DiskFileItemFactory created.
But that does not happen until I null the DiskFileItemFactory and call System.gc() (only nulling the DiskFileItemFactory does not help) at the end of the upload method. THis seems very strange to me. Here is my code :
#Override
public void upload(final HttpServletRequest request) {
ValidateUtils.checkNotNull(request, "upload request");
final File tmp = new File(this.tempFolder);
if (!tmp.exists()) {
tmp.mkdir();
}
DiskFileItemFactory fileItemFactory = new DiskFileItemFactory(this.sizeThreshold, tmp);
fileItemFactory.setFileCleaningTracker(this.fileCleaningTracker);
ServletFileUpload uploadHandler = new ServletFileUpload(fileItemFactory);
List items;
try {
items = uploadHandler.parseRequest(request);
} catch (final FileUploadException e) {
throw new ImageUploadServiceException("Error parsing the http servlet request for image upload.", e);
}
final Iterator it = items.iterator();
while (it.hasNext()) {
final DiskFileItem item = (DiskFileItem) it.next();
if (item.isFormField()) {
// log message
} else {
final String fileName = item.getName();
final File destination = this.createFileForUpload(fileName, this.uploadFolder);
FileChannel outChannel;
try {
outChannel = new FileOutputStream(destination).getChannel();
} catch (final FileNotFoundException e) {
throw new ImageUploadServiceException(e);
}
FileChannel inChannel = null;
try {
inChannel = new FileInputStream(item.getStoreLocation()).getChannel();
outChannel.transferFrom(inChannel, 0, item.getSize());
} catch (final IOException e) {
throw new ImageUploadServiceException(String.format("Error uploading image to '%s/%s'.", this.uploadFolder, destination.getName()), e);
} finally {
IOUtils.closeChannel(inChannel);
IOUtils.closeChannel(outChannel);
}
}
}
fileItemFactory.setFileCleaningTracker(null);
}
The above code causes every upload creates a file in the temp folder but does not remove it at the end by the "fileCleaningTracker", possibly because the DiskFileItemFactory instance is not garbage collected(I've failed to see why it shouldn't have) or it has been GCed but not notified by the PhantomReference in fileCleaningTracker(how reliable is PhantomReference?)
I waited 10 minutes and the files are still there, so it should't be because the GC has not run. and there are no exceptions.
Now if I add the following code, the temp files are removed every time after the upload:
fileItemFactory = null;
System.gc();
This looks very strange to me as I would expect the fileItemFactory be GCed without an explict call to System.gc().
Any input will be appreciated.
Thank you.
I have the same problem. The temporary files are never removed even after the server shutdown: GC process had not been started so FileCleaningTracker had no chance to get tracked files to delete from ReferenceQueue and all the files remain on the hard drive.
Due to specific behavior of my application I have to clean up after each upload (files might be very big). Instead of using standard org.apache.commons.io.FileCleaningTracker I am feeling lucky to override this class with my own implementation:
/**
* Cleaning tracker to clean files after each upload with special method invocation.
* Not thread safe and must be used with 1 factory = 1 thread policy.
*/
public class DeleteFilesOnEndUploadCleaningTracker extends FileCleaningTracker {
private List<String> filesToDelete = new ArrayList();
public void deleteTemporaryFiles() {
for (String file : filesToDelete) {
new File(file).delete();
}
filesToDelete.clear();
}
#Override
public synchronized void exitWhenFinished() {
deleteTemporaryFiles();
}
#Override
public int getTrackCount() {
return filesToDelete.size();
}
#Override
public void track(File file, Object marker) {
filesToDelete.add(file.getAbsolutePath());
}
#Override
public void track(File file, Object marker, FileDeleteStrategy deleteStrategy) {
filesToDelete.add(file.getAbsolutePath());
}
#Override
public void track(String path, Object marker) {
filesToDelete.add(path);
}
#Override
public void track(String path, Object marker, FileDeleteStrategy deleteStrategy) {
filesToDelete.add(path);
}
}
If this the right case for you just inject the instance of the class above into your DiskFileItemFactory:
DeleteFilesOnEndUploadCleaningTracker tracker = new DeleteFilesOnEndUploadCleaningTracker();
fileItemFactory.setFileCleaningTracker(tracker);
And don't forget to invoke the cleaning method after your work with uploaded items is done:
tracker.deleteTemporaryFiles();
Forgot to mention: I use commons-fileupload version 1.2.2 and commons-io version 1.3.2.

Duplex WCF + Static Collection of COM objects

I am trying to build a WCF service that exposes the functionality of a particular COM object that I do not have the original source for. I am using duplex binding so that each client has their own instance as there are events tied to each particular instance which are delivered through a callback (IAgent). It appears there is a deadlock or something because after the first action, my service blocks at my second action's lock. I have tried implementing these custom STA attribute and operation behaviors (http://devlicio.us/blogs/scott_seely/archive/2009/07/17/calling-an-sta-com-object-from-a-wcf-operation.aspx) but my OperationContext.Current is always null. Any advice is much appreciated.
Service
Collection:
private static Dictionary<IAgent, COMAgent> agents = new Dictionary<IAgent, COMAgent>();
First action:
public void Login(LoginRequest request)
{
IAgent agent = OperationContext.Current.GetCallbackChannel<IAgent>();
lock (agents)
{
if (agents.ContainsKey(agent))
throw new FaultException("You are already logged in.");
else
{
ICOMClass startup = new ICOMClass();
string server = ConfigurationManager.AppSettings["Server"];
int port = Convert.ToInt32(ConfigurationManager.AppSettings["Port"]);
bool success = startup.Logon(server, port, request.Username, request.Password);
if (!success)
throw new FaultException<COMFault>(new COMFault { ErrorText = "Could not log in." });
COMAgent comAgent = new COMAgent { Connection = startup };
comAgent.SomeEvent += new EventHandler<COMEventArgs>(comAgent_COMEvent);
agents.Add(agent, comAgent);
}
}
}
Second Action:
public void Logoff()
{
IAgent agent = OperationContext.Current.GetCallbackChannel<IAgent>();
lock (agents)
{
COMAgent comAgent = agents[agent];
try
{
bool success = comAgent.Connection.Logoff();
if (!success)
throw new FaultException<COMFault>(new COMFault { ErrorText = "Could not log off." });
agents.Remove(agent);
}
catch (Exception exc)
{
throw new FaultException(exc.Message);
}
}
}
Take a look at this very similar post: http://www.netfxharmonics.com/2009/07/Accessing-WPF-Generated-Images-Via-WCF
You have to use an OperationContextScope to have access to the current OperationContext from the newly generated thread:
System.Threading.Thread thread = new System.Threading.Thread(new System.Threading.ThreadStart(delegate
{
using (System.ServiceModel.OperationContextScope scope = new System.ServiceModel.OperationContextScope(context))
{
result = InnerOperationInvoker.Invoke(instance, inputs, out staOutputs);
}
}));