I've been looking all over the place and haven't been able to find a suitable answer to this question. I've created a NEST client using this code:
var myIndex = "myTestIndex";
var myType = "myTestType";
var myClusterUri= "http://localhost:9200";
var uri = new Uri(myClusterUri);
var settings = new ConnectionSettings(uri);
var client = new ElasticClient(settings);
and then later, using this to make a call to the bulk api.
var myJson = PopulateJsonForBulkAPI();
var rawBulkResult = client.Raw.Bulk(myIndex, myType, myJson);
The problem I'm having is that I'm getting an OutOfMemoryException when making the bulk api call. The method that populates myJson creates a HUGE block of JSON but not big enough to throw the exception (but big enough to throw it, if it were duplicated). Then when I make the call to the bulk api it throws the OutOfMemoryException because NEST holds onto the original request (in essence, duplicating the JSON and not having enough memory to hold onto everything). Is there a way to make the call to the Bulk API but tell NEST to NOT hold onto the original request so the huge block of JSON isn't duplicated in memory?
Edit
I'm using NEST version 1.7.2 and ElasticSearch version 1.7.2
In NEST 1.x, the request bytes are always made available on the response but you could write a HttpConnection implementation that doesn't do this, overriding DoSynchronousRequest and DoAsyncRequest.
If you're getting OutOfMemoryExceptions though, this sounds like you're trying to send too much data in one bulk request. Consider splitting up the data into batches of bulk requests.
Related
I’m currently using DataTables.js with server-site data source written on PHP.
The server-side script gives out the data exactly as required by datatables:
{“iTotalDisplayRecords”:”777”,”sEcho”:0,”aaData”:[[row1],[row2],[row3]]}
Now I would like to add an additional security layer with encrypting the response from the server and decrypting it after it is received by datatables.
I need this as I noticed some of the clients work through HTTPS proxy and the content of some rows get mistakenly blocked.
I’m using this solution for server-side PHP script to give out encrypted content using openssl_encrypt.
Then at client side I have:
function datatable_init (source) {
$.getJSON(source, function(data) {
decryptedContent = JSON.parse(CryptoJSAesDecrypt(“password”, data));
oTable = $(‘dtable’).dataTable({
“bProccesing”: false,
“bServerSide: true,
//“sAjaxSource”: source,
“data”: decryptedContent
...
});
I had to replace ”sAjaxSource” to ”data” as it is different datasource type now which requires different type of datatable JSON format:
{data:[[row1],[row2],[row3]}
and I can’t to pass iTotalDisplayRecords anymore.
Is there a way I can keep feeding server-side format of JSON to datatable but feed it as a local JS object/array?
P.S. Another idea I had is to encrypt/decrypt each individual row of the table but that’s probably going to be more complicated and slower
The ajax.dataSrc option seems to be helpful, because it provides the possibility to modify the data received via ajax and thus allows you to define a function which takes care of decrypting the received data again. Especially the last example given on the reference page looks promising in my opinion.
.net Core.
Using RestSharp 106.5.4, this was valid. The Priorities endpoint supports a List<string> priorities as a query parameter.
string server = "http://localhost:11111";
var client = new RestClient(server);
string filter = "?priority=Low&priority=Medium"
var request = new RestRequest($"priorities{filter}", Method.GET);
var response = client.Execute<Priorities>(request);
When I view the response objects ResponseUri property, it has
http://localhost:11111/priorities?priority=Low&priority=Medium
Exactly like I would expect. I upgraded my project to use RestSharp 106.6.2, and the request URI completely changed, and breaks existing code.
With RestSharp 106.6.2, it changes my URI and I get this ResponseUri back
http://localhost:11111/priorities?priority=Low,Medium
This means I now only get one element in my array of string in the request, comma seperated, instead of 2 elements like it did before.
Why was this unpassive change made? This broke our clients because now it only sees one element passed into the list of priorities, and that is "Low,Medium", instead of the two elements in the array as it should. This changes what was passed into the call, and makes the call invalid as Low,Medium is not a value, Low is a value, and Medium is a value.
Is this a defect in restsharps newest version, or was this done on purpose?
I know I can turn and use the parameters objects to do this, but up until the newest restsharp version, I didn't have too and it worked perfectly.
For reasons outlined here I need to review a set values from they querystring or formdata before each request (so I can perform some authentication). The keys are the same each time and should be present in each request, however they will be located in the querystring for GET requests, and in the formdata for POST and others
As this is for authentication purposes, this needs to run before the request; At the moment I am using a MessageHandler.
I can work out whether I should be reading the querystring or formdata based on the method, and when it's a GET I can read the querystring OK using Request.GetQueryNameValuePairs(); however the problem is reading the formdata when it's a POST.
I can get the formdata using Request.Content.ReadAsFormDataAsync(), however formdata can only be read once, and when I read it here it is no longer available for the request (i.e. my controller actions get null models)
What is the most appropriate way to consistently and non-intrusively read querystring and/or formdata from a request before it gets to the request logic?
Regarding your question of which place would be better, in this case i believe the AuthorizationFilters to be better than a message handler, but either way i see that the problem is related to reading the body multiple times.
After doing "Request.Content.ReadAsFormDataAsync()" in your message handler, Can you try doing the following?
Stream requestBufferedStream = Request.Content.ReadAsStreamAsync().Result;
requestBufferedStream.Position = 0; //resetting to 0 as ReadAsFormDataAsync might have read the entire stream and position would be at the end of the stream causing no bytes to be read during parameter binding and you are seeing null values.
note: The ability of a request's content to be read single time only or multiple times depends on the host's buffer policy. By default, the host's buffer policy is set as always Buffered. In this case, you will be able to reset the position back to 0. However, if you explicitly make the policy to be Streamed, then you cannot reset back to 0.
What about using ActionFilterAtrributes?
this code worked well for me
public HttpResponseMessage AddEditCheck(Check check)
{
var request= ((System.Web.HttpContextWrapper)Request.Properties.ToList<KeyValuePair<string, object>>().First().Value).Request;
var i = request.Form["txtCheckDate"];
return Request.CreateResponse(HttpStatusCode.Ok);
}
I have a Grid, a Store and Model for its data and AJAX proxy for the Store that is pointing to my self-written PHP back-end. The PHP backend writes to log each time it is called.
The system works OK for Read, Update and Delete calls. However now I need to add new field to Store, which I do in such a way:
(here, some new data were generated...)
var newEntry=Ext.ModelManager.create({
id:id,
title: title,
url: '/php/'+fname,
minithumb: '/php/'+small,
thumb:'/php/'+thumb
}, 'MyApp.model.fileListModel');
var store=Ext.getCmp('currGallery').getStore();
store.add(newEntry);
store.sync();
I have the new line appearing in the Grid.
But with or withour sync() call, I have no calls going to my PHP back end. It however reads one more time. Store has parameter autoSync :true and does great updating data automatically when I edit existing line in the Grid.
What am I missing?
Try not to set id when creating new record.
In fact I was missing a
newEntry.phantom = true;
flag. After I set it before adding to store, Store and its Proxy started to send data to server.
Maybe ID solution also works, dunno.
Can Flash be used together with SQL? I have a Flash form and I need to connect it to SQL. If there is any example on the net about this topic. I can't find it.
You don't use ActionScript directly with an SQL database. Instead you make http requests from ActionScript to a server, specifying the correct parameters. A typical opensource setup, is a PHP script communicating with a MySQL DB, but you can use Java with Oracle, Ruby with CouchDB, .NET with SQL or any other possible configuration. The important point is that you must be able to call a server script and pass variables... typically a Restful setup.
Once your PHP script has been properly configured, you can use http POST or http GET to send values from ActionScript.
PHP:
<?php
$updateValue = $_POST["updateValue"];
$dbResult = updateDB( $updateValue ); //This should return the db response
echo( $dbResult );
?>
To call this script from ActionScript, you need to create a variables object.
var variables:URLVariables = new URLVariables();
variables.updateValue = "someResult";
The variable name .updateValue, must match the php variable exactly.
now create a URLRequest Object, specifying the location of your script. For this example the method must be set to POST. You add the variable above to the data setter of the request.
var request:URLRequest = new URLRequest( "yourScript.php" );
request.method = URLRequestMethod.POST;
request.data = variables;
Now create a URLLoader and add an event listener. Do not pass the request created above to the constructor, but to the load method.
var loader:URLLoader = new URLLoader();
loader.addEventListener(Event.COMPLETE, onComplete );
loader.load( request );
The handler would look something like this.
private function onComplete( e:Event ) : void
{
trace( URLLoader( e.target ).data.toString() );
}
This example shows how to update and receive a response from a server / db combo. However, you can also query a DB through the script and parse the result. So in the PHP example above, you can output JSON, XML or even a piped string, and this can be consumed by ActionScript.
XML is a popular choice, as ActionScript's e4x support treats XML like a native object.
To treat the response above like an XML response, use the following in the onComplete handler.
private function onComplete( e:Event ) : void
{
var result:XML = XML( URLLoader( e.target ).data );
}
This will throw an error if your xml is poorly formed, so ensure the server script always prints out valid XML, even if there is a DB error.
The problem with this is giving someone a flash file that directly accesses SQL server is very insecure. Even if it's possible, which I have seen SOCKET classes out there to do so for MySQL (though never used it), allowing users to remotely connect to your DB is insecure as the user can sniff the login information.
In my opinion, the best way to do this is to create a Client/Server script. You can easily do this with PHP or ASP.net by using SendAndLoad to send the data you need to pass to SQL via POST fields. You can then send back the values in PHP with:
echo 'success='.+urlencode(data);
With this, flash can access the data via the success field.
I don't personally code flash but I work with a company who develops KIOSK applications for dozens of tradeshow companies, and my job is to store the data, return it to them. This is the method we use. You can make it even cleaner by using actual web services such as SOAP, but this method gets the job done if its just you using it.
You should look into Zend Amf or even the Zend Framework for server side communication with Flash. As far as I know Zend Amf is the fastest way to communicate with PHP ( therefore your database ) also you can pass & return complex Objects from the client to the server and vice versa.
Consider this , for instance. You have a bunch of data in your database , you implement functions in ZF whereas this data is formatted and set as a group of Value Objects. From Flash , you query ZF , Zf queries the database , retrieve & formats your data, return your Value Objects as a JSON string ( for instance ). In Flash, you retrieve you JSON string , decode it and assign your Value Objects to whatever relevant classes you have.
There are plenty of tutorials out there regarding Flash communication with the Zend Framework.
Here's an example:
http://gotoandlearn.com/play.php?id=90