In cytoscape.js, I'm trying to change the values of two node attributes, for each node in a 329-node network.
This works, but takes 86 seconds. I was hoping it would be much faster. Here is my code:
var data = // data of the form {lfc: {key1: 0.01, key2: 0.02, ...}, pval: {key1: 0.03, key2: 0.04, ...}};
var nodes = cy.nodes();
console.log('starting to update node data...');
var start = new Date().getTime();
for (var i = 0; i < nodes.length; i++)
{
var node = nodes[i];
var id = node.data("id");
var lfc = data['lfc'][id];
var pval = data['pval'][id];
node.data({lfc: lfc, pval: pval});
}
var end = new Date().getTime();
console.log("finished in " + (end - start) + "ms");
I know if I am setting the attributes of all nodes in the network to the same value, it's much faster. This takes about half a second:
var start = new Date().getTime();
cy.nodes().data({lfc: 1.0, pval: 1.0});
var end = new Date().getTime();
console.log("finished in " + (end - start) + "ms");
But since I'm trying to give each node individual values, I can't do this.
Any ideas for improving performance?
The problem with calling functions like that in a loop is that each iteration causes events and notifications to the renderer -- because visual updates may be needed. As you noted, this is not an issue when adding/loading in bulk, because this only generates one notification.
The ideal architecture would be to load in all the elements you need before intialising cytoscape.js, and then updating individual nodes as they change on the server. Polling the server and updating all the nodes will likely cause you other issues other than loading performance.
Related
Given a node in cytoscape.js, I'm trying to remove the node and all directed (in and out) edges of that node. I've wrote some quick and dirty filters to do so, but calling cy.remove(edges); prints a 'no such method exists' error message to the console. Removing a node through the same approach works fine, but removes all nodes connected to the removed node, which in my case, is the whole graph (most of the times). So I've tried removing the edges and the node from my data source and then re-drawing the graph, but this is not the correct approach since it re-draws the graph, and changes the layout( i.e. refreshes the canvas).
How can I solve this?
//data is the id of the node I want to remove
var filteredEdges = this.elements.edges.filter((x, idx, arr) =>
{
return x.source != data && x.target != data ;
});
var removedEdges = this.elements.edges.filter((x, idx, arr) =>
{
return x.source == data || x.target == data ;
});
//this.cy.remove(removedEdges); this fails
this.elements.edges = filteredEdges;
var filteredNodes = this.elements.nodes.filter((x, idx, arr) =>
{
return x.id != data;
});
this.cy.remove(data);
this.elements.nodes = filteredNodes;
this.render(); // this re-draws the whole graph, not a useable approach
Solved it using selectors. The documentation for cytoscape.js is very lacking. For anyone encountering this issue sometime in the future, here's how I did it:
this.cy.remove('edge[source=\'' + nodeId + '\']');
this.cy.remove('edge[target=\'' + nodeId + '\']');
this.cy.remove('#' + nodeId);
I am using the following script to add rows of files from a student loop in the Google spreadsheet if credits are less than x. The script was working good but as the data in the spreadsheet is being added daily, now the script is throwing "Exceeded maximum execution time" error (we have more than 2000 files). As I am new to scripting I don't know how to optimize the code.
Could someone help me to optimize the code or any solution so that the execution time take less than 5 min. Every time you compare to an email, it has to be compared to many emails. Please Help!
function updated() {
//Final file data (Combined)
var filecombined = SpreadsheetApp.openById("XXXXXXXXXX");
var sheet2 = filecombined.getSheets();
//Folder with all the files
var parentFolder = DriveApp.getFolderById("YYYYYYYYYYYY");
var files = parentFolder.getFiles();
//Current Date
var fecha = new Date();
//Path for each file in the folder
while (files.hasNext()) {
var idarchivo = files.next().getId();
var sps = SpreadsheetApp.openById(idarchivo);
var sheet = sps.getSheetByName('STUDENT PROFILE');
var data = sheet.getDataRange().getValues();
var credits = data[5][1];
//Flat; bandera:1 (new row), bandera:2 (update row)
var bandera = 1;
//Take data from final file (Combined)
var data2 = sheet2[0].getDataRange().getValues();
//If credits are less than X: write
if (credits < 120) {
var email = data[2][1];
var lastrow = filecombined.getLastRow();
var u = 0;
//comparison loop by email, if found it, update and exit the loop
while (u < lastrow) {
u = u + 1;
if (email == data2[u - 1][1]) {
sheet2[0].getRange(u, 3).setValue(credits);
sheet2[0].getRange(u, 4).setValue(fecha);
u = lastrow;
bandera = 2;
}
}
//if that email does not exist, write a new row
if (bandera == 1) {
var nombre = data[0][1];
sheet2[0].getRange(lastrow + 1, 1).setValue(nombre);
sheet2[0].getRange(lastrow + 1, 2).setValue(email);
sheet2[0].getRange(lastrow + 1, 3).setValue(credits);
sheet2[0].getRange(lastrow + 1, 4).setValue(fecha);
}
}
}
SpreadsheetApp.flush();
}
The questioner's code is taking taking more than 4-6 minutes to run and is getting an error Exceeded maximum execution time.
The following answer is based solely on the code provided by the questioner. We don't have any information about the 'filecombined' spreadsheet, its size and triggers. We are also in the dark about the various student spreadsheets, their size, etc, except that we know that there are 2,000 of these files. We don't know how often this routine is run, nor how many students have credits <120.
getvalues and setvalues statements are very costly; typically 0.2 seconds each. The questioners code includes a variety of such statements - some are unavoidable but others are not.
In looking at optimising this code, I made two major changes.
1 - I moved line 27 var data2 = sheet2[0].getDataRange().getValues();
This line need only be executed once and I relocated it at the top of the code just after the various "filecombined" commands. As it stood, this line was being executed once for every student spreadsheet; this along may have contributed to several minutes of execution time.
2) I converted certain setvalue commands to an array, and then updated the "filecombined" spreadsheet from the array once only, at the end of the processing. Depending on the number of students with low credits and who are not already on the "filecombined" sheet, this may represent a substantial saving.
The code affected was lines 47 to 50.
line47: sheet2[0].getRange(lastrow+1, 1).setValue(nombre);
line48: sheet2[0].getRange(lastrow+1, 2).setValue(email);
line49: sheet2[0].getRange(lastrow+1, 3).setValue(credits);
line50: sheet2[0].getRange(lastrow+1, 4).setValue(fecha);
There are setvalue commands also executed at lines 38 and 39 (if the student is already on the "filecombined" spreadsheet), but I chose to leave these as-is. As noted above, we don't know how many such students there might be, and the cost of these setvalue commands may be minor or not. Until this is clear, and in the light of other time savings, I chose to leave them as-is.
function updated() {
//Final file data (Combined)
var filecombined = SpreadsheetApp.openById("XXXXXXXXXX");
var sheet2 = filecombined.getSheets();
//Take data from final file (Combined)
var data2 = sheet2[0].getDataRange().getValues();
// create some arrays
var Newdataarray = [];
var Masterarray = [];
//Folder with all the files
var parentFolder = DriveApp.getFolderById("YYYYYYYYYYYY");
var files = parentFolder.getFiles();
//Current Date
var fecha = new Date();
//Path for each file in the folder
while (files.hasNext()) {
var idarchivo = files.next().getId();
var sps = SpreadsheetApp.openById(idarchivo);
var sheet = sps.getSheetByName('STUDENT PROFILE');
var data = sheet.getDataRange().getValues();
var credits = data[5][1];
//Flat; bandera:1 (new row), bandera:2 (update row)
var bandera = 1;
//If credits are less than X: write
if (credits < 120){
var email = data[2][1];
var lastrow = filecombined.getLastRow();
var u = 0;
//comparison loop by email, if found it, update and exit the loop
while (u < lastrow) {
u = u + 1;
if (email == data2[u-1][1]){
sheet2[0].getRange(u, 3).setValue(credits);
sheet2[0].getRange(u, 4).setValue(fecha);
u = lastrow;
bandera = 2;
}
}
//if that email does not exist, write a new row
if(bandera == 1){
var nombre = data[0][1];
Newdataarray = [];
Newdataarray.push(nombre);
Newdataarray.push(email);
Newdataarray.push(credits);
Newdataarray.push(fecha);
Masterarray.push(Newdataarray);
}
}
}
// update the target sheet with the contents of the array
// these are all adding new rows
lastrow = filecombined.getLastRow();
sheet2[0].getRange(lastrow+1, 1, Masterarray.length, 4);
sheet2[0].setValues(Masterarray);
SpreadsheetApp.flush();
}
As I mentioned in my comment, the biggest issue you have is that you repeatedly search an array for a value, when you could use a much faster lookup function.
// Create an object that maps an email address to the (last) array
// index of that email in the `data2` array.
const knownEmails = data2.reduce(function (acc, row, index) {
var email = row[1]; // email is the 2nd element of the inner array (Column B on a spreadsheet)
acc[email] = index;
return acc;
}, {});
Then you can determine if an email existed in data2 by trying to obtain the value for it:
// Get this email's index in `data2`:
var index = knownEmails[email];
if (index === undefined) {
// This is a new email we didn't know about before
...
} else {
// This is an email we knew about already.
var u = ++index; // Convert the array index into a worksheet row (assumes `data2` is from a range that started at Row 1)
...
}
To understand how we are constructing knownEmails from data2, you may find the documentation on Array#reduce helpful.
I am a Photoshop beginner and currently use version Photoshop CS3. I use keyboard shortcut all the time to speed up the design process such as creation of new layers etc.
However, one command I feel Photoshop must have is to create a new layer below the current working layer and therefore I cannot assign it via a shortcut.
I have to create a new layer above the current layer and then manually drag it below the current layer which I feel can be automated using action or scripting, both of which are difficult for me being a beginner.
Can anybody help me in this regard.
Thanks
dkj
It can be scripted with the following:
I've simplified my answer - you don't need to find the index, you can use the active layer instead.
create_new_layer("Gwen!");
// function CREATE NEW LAYER (layername)
// --------------------------------------------------------
function create_new_layer(layername)
{
if (layername == undefined) layername = "Layer";
// create new layer at top of layers
var originalLayer = app.activeDocument.activeLayer;
var layerRef = app.activeDocument.artLayers.add();
// name it & set blend mode to normal
layerRef.name = layername;
layerRef.blendMode = BlendMode.NORMAL;
// Move the layer below
layerRef.moveAfter(originalLayer);
// Move the layer above if you desire
// layerRef.moveBefore(originalLayer);
}
You can then record this script as an action and put on a keyboard short cut.
Few years ago i thought that native PS API working with DOM is cool and should work faster, but actually under the hood it's callstack often even bigger than same commands performed via actions. + Also sometimes DOM functions consist of multiple underlying calls, like artLayers.add() for example which is actually make layer + move it to top of the document. So here's action version of that functionality from my PS scripting library:
// get current layer number
function curLyrN(){
if(app.activeDocument.artLayers.length<2) return 1;
var idLyr = charIDToTypeID("Lyr ");
var idItmI = charIDToTypeID("ItmI");
var aref = new ActionReference();
aref.putEnumerated(idLyr, charIDToTypeID("Ordn"), charIDToTypeID("Trgt"));
var id = executeActionGet(aref).getInteger(charIDToTypeID("LyrI"));
aref = new ActionReference();
aref.putProperty(charIDToTypeID("Prpr"), idItmI);
aref.putIdentifier(idLyr, id);
id = executeActionGet(aref).getInteger(idItmI);
if(id) return id;
return 0;
}
// select [LayerNum], optionally [add] to selection (if add=2: with inclusion)
function selLyr(LyrN,add){
var adesc = new ActionDescriptor();
var aref = new ActionReference();
aref.putIndex(charIDToTypeID("Lyr "), LyrN);
adesc.putReference(charIDToTypeID("null"), aref);
if(add){
add = (add==2) ? stringIDToTypeID("addToSelectionContinuous") : stringIDToTypeID("addToSelection");
adesc.putEnumerated(stringIDToTypeID("selectionModifier"),stringIDToTypeID("selectionModifierType"),add);
}
adesc.putBoolean(charIDToTypeID("MkVs"), false);
return executeAction(charIDToTypeID("slct"), adesc, DialogModes.NO);
}
// move current layer to [LayerNum]
function movLyr(LyrN){
var idLyr = charIDToTypeID("Lyr ");
var adesc = new ActionDescriptor();
var aref = new ActionReference();
aref.putEnumerated(idLyr, charIDToTypeID("Ordn"), charIDToTypeID("Trgt"));
adesc.putReference(charIDToTypeID("null"), aref);
aref = new ActionReference();
aref.putIndex(idLyr, LyrN);
adesc.putReference(charIDToTypeID("T "), aref);
adesc.putBoolean(charIDToTypeID("Adjs"), false);
return executeAction(charIDToTypeID("move"), adesc, DialogModes.NO);
}
// create new empty layer
function mkLyr(){
var aref = new ActionReference();
aref.putClass(charIDToTypeID("Lyr "));
var adesc = new ActionDescriptor();
adesc.putReference(charIDToTypeID("null"), aref);
return executeAction(charIDToTypeID("Mk "), adesc, DialogModes.NO);
}
// count all inner layers from layer-set (group)
function cntLyrS(lyrs,c){
if(!c){
if(lyrs.typename!='LayerSet') return 0;
c = 0;
}
var ls = lyrs.layers.length;
var i = 0;
while(i<ls){
c++;
if(lyrs.layers[i].typename=='LayerSet') c=cntLyrS(lyrs.layers[i],c);
i++;
}
return c+1;
}
// make new layer below current or [LayerNum], optionally [ignoreGroups]
function mkLyrBelow(LyrN,noGr){
var doc = app.activeDocument;
if(!doc) return false;
if(LyrN){
selLyr(LyrN);
}else{
LyrN = curLyrN();
if(!LyrN) return false;
}
var actv = doc.activeLayer;
if(actv.isBackgroundLayer) actv.isBackgroundLayer=false;
mkLyr();
if(curLyrN()==LyrN) return true;
if(!noGr){
var lc = cntLyrS(actv);
if(lc && lc<LyrN-1) LyrN -= lc;
}
return movLyr(LyrN-1);
}
And even tho it looks pretty cumbersome and scary - i doubt that it will perform much slower. And as a bonus it will create minimal amount of actions in the history (no unnecessary layer moves) + it will correctly work with background layer + it will work properly with the groups (layer-sets): if group is opened - it will create new layer inside of it, and if group is closed it will correctly move layer under the whole group-structure including other possible groups inside the selected one.
Use it like that: mkLyrBelow(); to create new layer under selected one, or mkLyrBelow(LayerNumber); to create layer under another one via it's number, also u can optionally add 2d parameter to ignore groups (it will move new layer inside the group even if it's closed): mkLyrBelow(LayerNumber,true); or mkLyrBelow(0,1);...
P.S. don't get me wrong about ActionRefs - they're not the silver bullet, just oftenly have some more convenience in the end, but ofc best results obtained when u combine ARef's with native API. Just believe me on that, i've coded my first PS script like 8 years ago, so i've tried pretty much everything =)
If I understand your question correctly, Photoshop already has these shortcuts
Ctrl+Shift+N (Creating New Layer)
Ctrl+] (To move the layer up)
Ctrl+[ (To move the layer down)
I'm trying to draw a line using ray casting. Basically I want to set up some lines coming from my "player" object out in all directions.
(like so: https://gamedev.stackexchange.com/questions/35013/how-to-handle-3d-collisions-using-raycasting-with-a-reflection-vector)
I want this so I can then use I can see my collision detection visually.
I know I can use different ways to do collision detection, but i'm using this way as a learning detection.
My issue is the code below draws a line but it seems to randomly change length and not always point at the same angle.
var ray = new THREE.Ray( player.model.objects.position, new THREE.Vector3(1, 1, 1));
var geometry = new THREE.Geometry();
// my issue is here. I don't think this is the right way use a ray to workout the second vector?
// EDIT: Realized this should be set at player position and outwards.
//var newx = 300 * ray.direction.x;
//var newz = 300 * ray.direction.z;
// EDIT CODE UPDATE
var newx = (player.model.objects.position.x) + (60 * ray.direction.x);
var newz = (player.model.objects.position.z) + (60 * ray.direction.z);
// THREE.Vector3 {x: 1310.1526178356803, y: 0, z: 1290.8237947033065}
console.log(player.model.objects.position);
geometry.vertices.push( player.model.objects.position);
geometry.vertices.push( new THREE.Vector3(newx, player.model.objects.position.y, newz));
var line = new THREE.Line(geometry, material);
scene.add(line);
Any help appreciated.
I was trying to do the same thing after seeing that model.. Since I tried to do it the same way and couldn't figure it out, I'll offer an alternative.
var line;
function update() {
// Z- DIRECTION
raycaster.ray.direction.set(0, 0, -1);
var geometry = new THREE.Geometry();
intersections = raycaster.intersectObjects( objects );
if ( intersections.length > 0 ) {
var geometry = new THREE.Geometry();
// POSITION OF MESH TO SHOOT RAYS OUT OF
geometry.vertices.push( obj.position );
geometry.vertices.push( intersections[0].point );
scene.remove(line);
line = new THREE.Line(geometry, new THREE.LineBasicMaterial({color: 0x990000}));
scene.add(line);
}
}
So now you have a line shooting out of your mesh into whatever the closest intersect is.
https://dl.dropbox.com/u/42766757/guy.png
I have written the following tests to compare performance of Linq2SQL and NHibernate and I find results to be somewhat strange. Mappings are straight forward and identical for both. Both are running against a live DB. Although I'm not deleting Campaigns in case of Linq, but that shouldn't affect performance by more than 10 ms.
Linq:
[Test]
public void Test1000ReadsWritesToAgentStateLinqPrecompiled()
{
Stopwatch sw = new Stopwatch();
Stopwatch swIn = new Stopwatch();
sw.Start();
for (int i = 0; i < 1000; i++)
{
swIn.Reset();
swIn.Start();
ReadWriteAndDeleteAgentStateWithLinqPrecompiled();
swIn.Stop();
Console.WriteLine("Run ReadWriteAndDeleteAgentState: " + swIn.ElapsedMilliseconds + " ms");
}
sw.Stop();
Console.WriteLine("Total Time: " + sw.ElapsedMilliseconds + " ms");
Console.WriteLine("Average time to execute queries: " + sw.ElapsedMilliseconds / 1000 + " ms");
}
private static readonly Func<AgentDesktop3DataContext, int, EntityModel.CampaignDetail>
GetCampaignById =
CompiledQuery.Compile<AgentDesktop3DataContext, int, EntityModel.CampaignDetail>(
(ctx, sessionId) => (from cd in ctx.CampaignDetails
join a in ctx.AgentCampaigns on cd.CampaignDetailId equals a.CampaignDetailId
where a.AgentStateId == sessionId
select cd).FirstOrDefault());
private void ReadWriteAndDeleteAgentStateWithLinqPrecompiled()
{
int id = 0;
using (var ctx = new AgentDesktop3DataContext())
{
EntityModel.AgentState agentState = new EntityModel.AgentState();
var campaign = new EntityModel.CampaignDetail { CampaignName = "Test" };
var campaignDisposition = new EntityModel.CampaignDisposition { Code = "123" };
campaignDisposition.Description = "abc";
campaign.CampaignDispositions.Add(campaignDisposition);
agentState.CallState = 3;
campaign.AgentCampaigns.Add(new AgentCampaign
{
AgentState = agentState
});
ctx.CampaignDetails.InsertOnSubmit(campaign);
ctx.AgentStates.InsertOnSubmit(agentState);
ctx.SubmitChanges();
id = agentState.AgentStateId;
}
using (var ctx = new AgentDesktop3DataContext())
{
var dbAgentState = ctx.GetAgentStateById(id);
Assert.IsNotNull(dbAgentState);
Assert.AreEqual(dbAgentState.CallState, 3);
var campaignDetails = GetCampaignById(ctx, id);
Assert.AreEqual(campaignDetails.CampaignDispositions[0].Description, "abc");
}
using (var ctx = new AgentDesktop3DataContext())
{
ctx.DeleteSessionById(id);
}
}
NHibernate (the loop is the same):
private void ReadWriteAndDeleteAgentState()
{
var id = WriteAgentState().Id;
StartNewTransaction();
var dbAgentState = agentStateRepository.Get(id);
Assert.IsNotNull(dbAgentState);
Assert.AreEqual(dbAgentState.CallState, 3);
Assert.AreEqual(dbAgentState.Campaigns[0].Dispositions[0].Description, "abc");
var campaignId = dbAgentState.Campaigns[0].Id;
agentStateRepository.Delete(dbAgentState);
NHibernateSession.Current.Transaction.Commit();
Cleanup(campaignId);
NHibernateSession.Current.BeginTransaction();
}
Results:
NHibernate:
Total Time: 9469 ms
Average time to execute 13 queries: 9 ms
Linq:
Total Time: 127200 ms
Average time to execute 13 queries: 127 ms
Linq lost by 13.5 times! Event with precompiled queries (both read queries are precompiled).
This can't be right, although I expected NHibernate to be faster, this is just too big of a difference, considering mappings are identical and NHibernate actually executes more queries against the DB.
Update. I have refactored a project to use NHibernate instead of Linq2Sql and the performance gain seems to be a lot less (about 20-30%) compared to test working on the same mappings. Does anyone have some real world examples of their own?
Run a profiler, both on the .NET code and on the SQL Server database. Also, identify what SQL statements are being run under the covers for both scenarios. Where is the time being lost for LinqToSql? If the underlying SQL statements are different, why? It's very likely you can tweak both ORMs to be faster. They should likely be in the same ballpark performance wise for simple tests. This feels like a configuration problem.