I have a road network with some dangled nodes and some looped roads. I have removed the dangled roads and now I also want to remove the looped roads. Can anyone tell me hoe can I do that. Thanks in advance (:
1) Finding Loops in Road Network:
For the roads that form 3/4 loops use the sinuosity formula (distance between line ends/ length). The Python Formula is:
!Shp_lngth! / math.pow(math.pow( !X_Start! - !X_End! , 2 ) + math.pow( !Y_Start! - !Y_End!, 2 ), 0.5) > 2
For the loops that are just at the end of the road (like cul de sac bulbs) use the Create Route tool and find roads with MMonotonicity values that are 4 or above.
2) Finding Dangling Nodes in Road Network:
Use this code
using System;
using System.Collections.Generic;
using ESRI.ArcGIS.Geodatabase;
using ESRI.ArcGIS.Carto;
using System.Windows.Forms;
using ESRI.ArcGIS.esriSystem;
using ESRI.ArcGIS.EditorExt;
namespace KalkulatorAddin
{
public class TestButton : ESRI.ArcGIS.Desktop.AddIns.Button
{
public TestButton()
{
}
protected override void OnClick()
{
try
{
Test();
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
}
protected override void OnUpdate()
{
}
public void Test()
{
var fSel = ArcMap.Document.FocusMap.get_Layer(0) as IFeatureSelection;
fSel.Clear();
var dangleOids = GetDangleOids();
if (dangleOids.Count > 0)
{
var oidarray = dangleOids.ToArray();
fSel.SelectionSet.AddList(dangleOids.Count, ref oidarray[0]);
}
((IActiveView)ArcMap.Document.FocusMap).Refresh();
}
private List<int> GetDangleOids()
{
UID topoUiD = new UID();
topoUiD.Value = "esriEditorExt.TopologyExtension";
var topoExt = ArcMap.Application.FindExtensionByCLSID(topoUiD) as ITopologyExtension;
var mapTopology = topoExt.CurrentTopology as IMapTopology;
if (mapTopology == null)
throw new Exception("map topology not found");
//assume just one class in the map topology
var extent = ((IGeoDataset)mapTopology.get_Class(0)).Extent;
mapTopology.Cache.Build(extent, false);
var dangleOids = new List<int>();
var nodes = mapTopology.Cache.Nodes;
nodes.Reset();
ITopologyNode node;
while ((node = nodes.Next()) != null)
{
// sometimes degree is referred to as valence
if (node.Degree == 1)
{
var parents = node.Parents;
parents.Reset();
int oid = parents.Next().m_FID;
if (!dangleOids.Contains(oid))
dangleOids.Add(oid);
}
}
return dangleOids;
}
}
}
Related
I'm trying to aplicate Flyweight method pattern in a simple .net core Api to see how much memory is saved compared to not using the pattern.
I have two methods, the first one creates 5000 objects without uses the pattern and the another creates 5000 object using the pattern. After each of them create the objects, then they call a method that returns the current memory used by the App.
public class MemoryService : IMemoryService
{
private readonly TreeFactory _treeFactory;
public MemoryService()
{
_treeFactory = new TreeFactory();
}
//create without pattern
public long SetObjectsMemory()
{
List<Tree> trees = new List<Tree>();
for (int i = 0; i < 5000; i++)
{
var tree = new Tree()
{
Id = new Random().Next(1, 9999999),
Part = new PartTree()
{
Name = "Nameany",
Bark = "Barkany",
Color = "Colorany"
}
};
trees.Add(tree);
};
return Utilities.GetCurrentMemoryUsed();
}
//crete with flyweight pattern
public long SetObjectsMemoryFactory()
{
List<Tree> trees = new List<Tree>();
for (int i = 0; i < 5000; i++)
{
var tree = new Tree()
{
Id = new Random().Next(1, 9999999),
Part = _treeFactory.GetPartTree("Nameany", "Barkany", "Colorany")
};
trees.Add(tree);
}
return Utilities.GetCurrentMemoryUsed();
}
}
I use the pattern like a class that uses a list of Parts and return a part object if exists.
public class TreeFactory
{
private static List<PartTree> _parts;
public TreeFactory() {
_parts = new List<PartTree>();
}
public PartTree GetPartTree(string name, string bark, string color)
{
if (_parts.Any(x => x.Name == name && x.Bark == bark && x.Color == color))
{
return _parts.Where(x => x.Name == name && x.Bark == bark && x.Color == color).FirstOrDefault();
}
else {
var newpart = new PartTree()
{
Name = name,
Bark = bark,
Color = color
};
_parts.Add(newpart);
return newpart;
}
}
}
The way to get the current memory used by the App is using Process of this way (in Utilities class):
public static long GetCurrentMemoryUsed() {
Int64 memory;
using (Process proc = Process.GetCurrentProcess())
{
memory = proc.PrivateMemorySize64 / (1024 * 1024);
}
return memory;
}
And in my Startup i inject the MemoryService like a Singleton. In the controller i use 3 methods for call the functions:
[HttpGet, Route(nameof(WeatherForecastController.GenerateMemory))]
public IActionResult GenerateMemory()
{
var total=_memoryService.SetObjectsMemory();
return Ok(total);
}
[HttpGet, Route(nameof(WeatherForecastController.GenerateLiftMemory))]
public IActionResult GenerateLiftMemory()
{
var total = _memoryService.SetObjectsMemoryFactory();
return Ok(total);
}
[HttpGet, Route(nameof(WeatherForecastController.GetMemory))]
public IActionResult GetMemory()
{
var total = Utilities.GetCurrentMemoryUsed();
return Ok(total);
}
The problem is: When i call in the navigator the method in controller without pattern (/weatherforecast/GenerateMemory), then this returns (current)+2mb, but when i call the method
with pattern (/weatherforecast/GenerateLiftMemory) this returns (current)+3mb.
Why the method with pattern flyweight returns more used MB (growing) than the methods without the pattern ??
The repository with the code for test it. Gitlab repository memory api
The code which uses TreeFactory consumes more memory because its GetPartTree method called many times in a loop so as Linq methods Any and Where inside it. Both of these methods create additional Iterator objects under the hood in order to iterate through the collection and it causes additional memory consumption.
I wrote simple benchmark using BenchmarkDotNet with more options to demonstrate the issue
Extended MemoryService
public class MemoryService : IMemoryService
{
private const int TreeCount = 50000;
private readonly TreeFactory _treeFactory;
public MemoryService()
{
_treeFactory = new TreeFactory();
}
//crea objetos en memoria sin patrones
public decimal SetObjectsMemory()
{
List<Tree> trees = new List<Tree>();
for (int i = 0; i < TreeCount; i++)
{
var tree = new Tree()
{
Id = 1,
Part = new PartTree()
{
Name = "Nameany",
Bark = "Barkany",
Color = "Colorany"
}
};
trees.Add(tree);
};
return Utilities.GetCurrentMemoryUsed();
}
//crea objetos en memoria usando patron flyweight
public decimal SetObjectsMemoryFactory()
{
List<Tree> trees = new List<Tree>();
for (int i = 0; i < TreeCount; i++)
{
var tree = new Tree()
{
Id = 1,
Part = _treeFactory.GetPartTree("Nameany", "Barkany", "Colorany")
};
trees.Add(tree);
}
return Utilities.GetCurrentMemoryUsed();
}
public decimal SetObjectsMemoryFactoryImproved()
{
List<Tree> trees = new List<Tree>();
for (int i = 0; i < TreeCount; i++)
{
var tree = new Tree()
{
Id = 1,
Part = _treeFactory.GetPartTreeImproved("Nameany", "Barkany", "Colorany")
};
trees.Add(tree);
}
return Utilities.GetCurrentMemoryUsed();
}
//crea objetos en memoria usando patron flyweight
public decimal SetObjectsMemoryFactoryWithoutLambda()
{
List<Tree> trees = new List<Tree>();
for (int i = 0; i < TreeCount; i++)
{
var tree = new Tree()
{
Id = 1,
Part = _treeFactory.GetPartTreeWithoutLambda("Nameany", "Barkany", "Colorany")
};
trees.Add(tree);
}
return Utilities.GetCurrentMemoryUsed();
}
}
Extended TreeFactory
public class TreeFactory
{
private static List<PartTree> _parts;
public TreeFactory()
{
_parts = new List<PartTree>();
}
public PartTree GetPartTree(string name, string bark, string color)
{
if (_parts.Any(x => x.Name == name && x.Bark == bark && x.Color == color))
{
return _parts.Where(x => x.Name == name && x.Bark == bark && x.Color == color).FirstOrDefault();
}
var newpart = new PartTree()
{
Name = name,
Bark = bark,
Color = color
};
_parts.Add(newpart);
return newpart;
}
public PartTree GetPartTreeImproved(string name, string bark, string color)
{
var existingPart = _parts.Where(x => x.Name == name && x.Bark == bark && x.Color == color).FirstOrDefault();
if (existingPart != null)
return existingPart;
var newpart = new PartTree()
{
Name = name,
Bark = bark,
Color = color
};
_parts.Add(newpart);
return newpart;
}
public PartTree GetPartTreeWithoutLambda(string name, string bark, string color)
{
for (int i = 0; i < _parts.Count; i++)
{
var x = _parts[i];
if (x.Name == name && x.Bark == bark && x.Color == color)
return x;
}
var newpart = new PartTree()
{
Name = name,
Bark = bark,
Color = color
};
_parts.Add(newpart);
return newpart;
}
}
Benchmark in a separate console project
class Program
{
static void Main(string[] args)
{
var result = BenchmarkRunner.Run<MemoryBenchmark>();
}
}
[MemoryDiagnoser]
public class MemoryBenchmark
{
private IMemoryService memoryService;
[GlobalSetup]
public void Setup()
{
memoryService = new MemoryService();
}
[Benchmark]
public object SimpleTrees()
{
var trees = memoryService.SetObjectsMemory();
return trees;
}
[Benchmark]
public object FlyTrees()
{
var trees = memoryService.SetObjectsMemoryFactory();
return trees;
}
[Benchmark]
public object FlyTreesImproved()
{
var trees = memoryService.SetObjectsMemoryFactoryImproved();
return trees;
}
[Benchmark]
public object FlyTreesWithoutLambda()
{
var trees = memoryService.SetObjectsMemoryFactoryWithoutLambda();
return trees;
}
}
And its results
Method
Mean
Error
StdDev
Gen 0
Gen 1
Gen 2
Allocated
SimpleTrees
9.040 ms
0.1804 ms
0.2346 ms
718.7500
453.1250
265.6250
4.44 MB
FlyTrees
19.701 ms
0.1716 ms
0.1521 ms
2500.0000
906.2500
437.5000
15.88 MB
FlyTreesImproved
18.075 ms
0.2869 ms
0.2684 ms
1781.2500
625.0000
312.5000
10.92 MB
FlyTreesWithoutLambda
4.919 ms
0.0273 ms
0.0242 ms
421.8750
281.2500
281.2500
2.53 MB
my assignment involves using recursive method. Write a program that reverses a LinkedList. This is the code i have done below, can anyone see what i am doing wrong, thank you very much!
PS: this is done in jGRASP
// Java program for reversing the linked list
class MyLinkedList {
static Node head;
static class Node {
int data;
Node next;
Node(int d) {
data = d;
next = null;
}
}
/* Function to reverse the linked list */
Node reverse(Node node) {
Node prev = null;
Node current = node;
Node next = null;
while (current != null) {
next = current.next;
current.next = prev;
prev = current;
current = next;
}
node = prev;
return node;
}
// prints content of double linked list
void printList(Node node) {
while (node != null) {
System.out.print(node.data + " ");
node = node.next;
}
}
public static void main(String[] args) {
MyLinkedList list = new MyLinkedList();
list.head = new Node(85);
list.head.next = new Node(15);
list.head.next.next = new Node(4);
list.head.next.next.next = new Node(20);
System.out.println("Given Linked list");
list.printList(head);
head = list.reverse(head);
System.out.println("");
System.out.println("Reversed linked list ");
list.printList(head);
}
}
I am using script component in ssis with C# code using Microsoft.Analysisservices namespace to fetch the cube metadata. The code looks somewhat like this
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
using Microsoft.AnalysisServices;
using System.Windows.Forms;
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
//IDTSConnectionManager100 connMgr;
Server OLAPServer = new Server();
public override void AcquireConnections(object Transaction)
{
OLAPServer.Connect(this.Connections.OLAPConnection.ConnectionString);
}
public override void PreExecute()
{
base.PreExecute();
/*
Add your code here for preprocessing or remove if not needed
*/
}
public override void PostExecute()
{
base.PostExecute();
/*
Add your code here for postprocessing or remove if not needed
You can set read/write variables here, for example:
Variables.MyIntVar = 100
*/
}
public override void CreateNewOutputRows()
{
IDTSVariables100 vars = null;
string OLAPDBName;
VariableDispenser.LockOneForRead("OLAPDBName", ref vars);
Database OLAPDB;
OLAPDBName = vars[0].Value.ToString();
try
{
OLAPDB = OLAPServer.Databases.GetByName(OLAPDBName);
}
catch
{
return;
}
// loop through cubes
CubeCollection Cubes = OLAPDB.Cubes;
MeasureGroupCollection Mgroups;
CubeDimensionCollection Dimensions;
MeasureGroupDimensionCollection MgroupDims;
DimensionAttributeCollection Attributes;
foreach (Cube cb in Cubes)
{
//Test for one Measure Group
//MeasureGroup mgroup = Mgroups.GetByName("Inward Exposure");
Mgroups = cb.MeasureGroups;
// all dimensions associated with that Measure Group
// loop through Measure Groups
foreach (MeasureGroup mg in Mgroups)
{
// loop though all cube dimensions
Dimensions = cb.Dimensions;**strong text**
foreach (CubeDimension dim in Dimensions)
{
bool CanBeAnalysed = false;**strong text**
// loop through dimensions and see if dimension exists in mgroupDims (ie check if it can be analysed)
MgroupDims = mg.Dimensions;
foreach (MeasureGroupDimension mgd in MgroupDims)
{
if (mgd.CubeDimension == dim)
{
CanBeAnalysed = true;
break;
}
}
// loop through each Measure and Attribute a
String DimName = dim.Name;
bool DimVisible = dim.Visible;
String MgroupName = mg.Name;
String CubeName = cb.Name;
String MeasureExpression;
String Description;
// for every attribute in dimension
Attributes = dim.Dimension.Attributes;
foreach (DimensionAttribute Attr in Attributes)
{
String AttrName = Attr.Name;
bool AttrVisible = Attr.AttributeHierarchyVisible;
String AttrNameColumn = Attr.NameColumn.ToString();
String AttributeRelationship = Attr.AttributeRelationships.ToString();
// get every measure in measuregroup
foreach (Measure m in mg.Measures)
{
String MeasureName = m.Name.ToString();
bool MeasureVisible = m.Visible;
String MeasureNameColumn = m.Source.ToString();
if (m.MeasureExpression != null)
{
// MessageBox.Show(m.MeasureExpression.ToString());
MeasureExpression = m.MeasureExpression.ToString();
}
else
{
// MessageBox.Show(m.MeasureExpression.ToString());
MeasureExpression = " " ;
}
if (m.Description != null)
{
// MessageBox.Show(m.MeasureExpression.ToString());
Description = m.Description.ToString();
}
else
{
// MessageBox.Show(m.MeasureExpression.ToString());
Description = " ";
}
Output0Buffer.AddRow();
Output0Buffer.OLAPDBName = OLAPDBName;
Output0Buffer.CubeName = CubeName;
Output0Buffer.DimensionName = DimName;
Output0Buffer.DimensionVisible = DimVisible;
Output0Buffer.AttrDDSColumn = AttrNameColumn;
Output0Buffer.AttrName = AttrName;
Output0Buffer.AttrVisible = AttrVisible;
Output0Buffer.MeasureGroupName = MgroupName;
Output0Buffer.MeasureName = MeasureName;
Output0Buffer.MeasureVisible = MeasureVisible;
Output0Buffer.MeasureDDSColumn = MeasureNameColumn;
Output0Buffer.IsAnalysable = CanBeAnalysed;
Output0Buffer.MeasureExpression = MeasureExpression;
Output0Buffer.Description = Description;
Output0Buffer.AttributeRelationship = AttributeRelationship;
}
}
} // end of Cube Dim Loop
} // end of Measure Group loop
} // end of cube loop
}
}
I was successful in getting the cube metadata with the above code.However, i am stuck at getting the metadata of the perspective cube and the Relationships of the measure groups i.e whether the measure groups are many-many. Any help is very much appreciated.
Here is some code for detecting dimension relationships including many-to-many. See the GetDimensionUsage function:
https://raw.githubusercontent.com/BIDeveloperExtensions/bideveloperextensions/master/SSAS/PrinterFriendlyDimensionUsage.cs
Here is some code around navigating perspectives:
https://raw.githubusercontent.com/BIDeveloperExtensions/bideveloperextensions/master/SSAS/TriStatePerspectivesPlugin.cs
Start reading around the following line:
if (perspective.MeasureGroups.Contains(mg.Name))
I work with the kinect. My goal is to store the values gives me the kinect for the location of the body (head,hand etc). I have written some code but I can not understand what values should save and how.I want to store in the db or in a txt file the position of the head,hands and foots.I want the data to understand the movements of the person who stand in front of kinect.For example if someone move her hand the kinect will sent a value.I must store it and understand and to understand the move taken.Sorry for the few info
here is my code:
using System;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Shapes;
using Microsoft.Kinect;
using System.Linq;
using System.IO;
namespace KinectSkeletonApplication1
{
public partial class MainWindow : Window
{
//Instantiate the Kinect runtime. Required to initialize the device.
//IMPORTANT NOTE: You can pass the device ID here, in case more than one Kinect device is connected.
KinectSensor sensor = KinectSensor.KinectSensors[0];
byte[] pixelData;
Skeleton[] skeletons;
public MainWindow()
{
InitializeComponent();
///////////////////////////////////
///////////////////////////////
//Runtime initialization is handled when the window is opened. When the window
//is closed, the runtime MUST be unitialized.
this.Loaded += new RoutedEventHandler(MainWindow_Loaded);
this.Unloaded += new RoutedEventHandler(MainWindow_Unloaded);
sensor.ColorStream.Enable();
sensor.SkeletonStream.Enable();
}
void runtime_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{
bool receivedData = false;
using (SkeletonFrame SFrame = e.OpenSkeletonFrame())
{
if (SFrame == null)
{
// The image processing took too long. More than 2 frames behind.
}
else
{
skeletons = new Skeleton[SFrame.SkeletonArrayLength];
SFrame.CopySkeletonDataTo(skeletons);
receivedData = true;
}
}
if (receivedData)
{
Skeleton currentSkeleton = (from s in skeletons
where s.TrackingState == SkeletonTrackingState.Tracked
select s).FirstOrDefault();
if (currentSkeleton != null)
{
SetEllipsePosition(head, currentSkeleton.Joints[JointType.Head]);
SetEllipsePosition(leftHand, currentSkeleton.Joints[JointType.HandLeft]);
SetEllipsePosition(rightHand, currentSkeleton.Joints[JointType.HandRight]);
SetEllipsePosition(shoulder_center, currentSkeleton.Joints[JointType.ShoulderCenter]);
}
}
}
//This method is used to position the ellipses on the canvas
//according to correct movements of the tracked joints.
//IMPORTANT NOTE: Code for vector scaling was imported from the Coding4Fun Kinect Toolkit
//available here: http://c4fkinect.codeplex.com/
//I only used this part to avoid adding an extra reference.
private void SetEllipsePosition(Ellipse ellipse, Joint joint)
{
Microsoft.Kinect.SkeletonPoint vector = new Microsoft.Kinect.SkeletonPoint();
vector.X = ScaleVector(640, joint.Position.X);
vector.Y = ScaleVector(480, -joint.Position.Y);
vector.Z = joint.Position.Z;
Joint updatedJoint = new Joint();
updatedJoint = joint;
updatedJoint.TrackingState = JointTrackingState.Tracked;
updatedJoint.Position = vector;
Canvas.SetLeft(ellipse, updatedJoint.Position.X);
Canvas.SetTop(ellipse, updatedJoint.Position.Y);
}
private float ScaleVector(int length, float position)
{
float value = (((((float)length) / 1f) / 2f) * position) + (length / 2);
if (value > length)
{
return (float)length;
}
if (value < 0f)
{
return 0f;
}
string r = Convert.ToString(value);
string path = #"C:\Test\MyTest.txt";
// This text is added only once to the file.
if (!File.Exists(path))
{
// Create a file to write to.
string createText = "Hello and Welcome" + Environment.NewLine;
File.WriteAllText(path, createText);
}
// This text is always added, making the file longer over time
// if it is not deleted.
//string appendText = "This is extra text" + Environment.NewLine;
File.AppendAllText(path, r);
// Open the file to read from.
string readText = File.ReadAllText(path);
return value;
}
void MainWindow_Unloaded(object sender, RoutedEventArgs e)
{
sensor.Stop();
}
void MainWindow_Loaded(object sender, RoutedEventArgs e)
{
sensor.SkeletonFrameReady += runtime_SkeletonFrameReady;
sensor.ColorFrameReady += runtime_VideoFrameReady;
sensor.Start();
}
void runtime_VideoFrameReady(object sender, ColorImageFrameReadyEventArgs e)
{
bool receivedData = false;
using (ColorImageFrame CFrame = e.OpenColorImageFrame())
{
if (CFrame == null)
{
// The image processing took too long. More than 2 frames behind.
}
else
{
pixelData = new byte[CFrame.PixelDataLength];
CFrame.CopyPixelDataTo(pixelData);
receivedData = true;
}
}
if (receivedData)
{
BitmapSource source = BitmapSource.Create(640, 480, 96, 96,
PixelFormats.Bgr32, null, pixelData, 640 * 4);
videoImage.Source = source;
}
}
}
}
I think what you are looking for is getting the X, Y, Z coordiantes for certain Joints?
In this case add this code:
Vector3D ShoulderCenter = new Vector3D(skeleton.Joints[JointType.ShoulderCenter].Position.X, skeleton.Joints[JointType.ShoulderCenter].Position.Y, skeleton.Joints[JointType.ShoulderCenter].Position.Z);
Vector3D RightShoulder = new Vector3D(skeleton.Joints[JointType.ShoulderRight].Position.X, skeleton.Joints[JointType.ShoulderRight].Position.Y, skeleton.Joints[JointType.ShoulderRight].Position.Z);
Vector3D LeftShoulder = new Vector3D(skeleton.Joints[JointType.ShoulderLeft].Position.X, skeleton.Joints[JointType.ShoulderLeft].Position.Y, skeleton.Joints[JointType.ShoulderLeft].Position.Z);
Vector3D RightElbow = new Vector3D(skeleton.Joints[JointType.ElbowRight].Position.X, skeleton.Joints[JointType.ElbowRight].Position.Y, skeleton.Joints[JointType.ElbowRight].Position.Z);
Vector3D LeftElbow = new Vector3D(skeleton.Joints[JointType.ElbowLeft].Position.X, skeleton.Joints[JointType.ElbowLeft].Position.Y, skeleton.Joints[JointType.ElbowLeft].Position.Z);
Vector3D RightWrist = new Vector3D(skeleton.Joints[JointType.WristRight].Position.X, skeleton.Joints[JointType.WristRight].Position.Y, skeleton.Joints[JointType.WristRight].Position.Z);
Vector3D LeftWrist = new Vector3D(skeleton.Joints[JointType.WristLeft].Position.X, skeleton.Joints[JointType.WristLeft].Position.Y, skeleton.Joints[JointType.WristLeft].Position.Z);
This only defines the Vectors for the upper Body part as you can see. Just add the missing joints the way I did it.
You need following assemblies:
using System.Windows.Media;
using Microsoft.Kinect.Toolkit.Fusion;
using System.Windows.Media.Media3D;
I am interfacing with a PostgreSQL database with NHibernate.
Background
I made some simple tests...it seems it's taking 2 seconds to persist 300 records.
I have a Perl program with identical functionality, but issue direct SQL instead, takes only 70% of the time.
I am not sure if this is expected. I thought C#/NHibernate would be faster or at least on par.
Questions
One of my observation is that (with show_sql turned on), the NHibernate is issuing INSERTs a few hundreds times, instead of doing bulk INSERT that take cares of multiple rows. And note I am assigning the primary key myself, not using the "native" generator.
Is that expected? Is there anyway I could make it issue bulk INSERT statement instead? It seems to me that this could be one of the area I could speed up the performance.
As stachu found out correctly: NHibernate does not have *BatchingBatcher(Factory) for PostgreSQL(Npgsql)
As stachu askes: Did anybody managed to force Nhibarnate to do batch inserts to PostgreSQL
I wrote a Batcher that doesn't use any Npgsql batching stuff, but does manipulate the SQL String "oldschool style" (INSERT INTO [..] VALUES (...),(...), ...)
using System;
using System.Collections;
using System.Data;
using System.Diagnostics;
using System.Text;
using Npgsql;
namespace NHibernate.AdoNet
{
public class PostgresClientBatchingBatcherFactory : IBatcherFactory
{
public virtual IBatcher CreateBatcher(ConnectionManager connectionManager, IInterceptor interceptor)
{
return new PostgresClientBatchingBatcher(connectionManager, interceptor);
}
}
/// <summary>
/// Summary description for PostgresClientBatchingBatcher.
/// </summary>
public class PostgresClientBatchingBatcher : AbstractBatcher
{
private int batchSize;
private int countOfCommands = 0;
private int totalExpectedRowsAffected;
private StringBuilder sbBatchCommand;
private int m_ParameterCounter;
private IDbCommand currentBatch;
public PostgresClientBatchingBatcher(ConnectionManager connectionManager, IInterceptor interceptor)
: base(connectionManager, interceptor)
{
batchSize = Factory.Settings.AdoBatchSize;
}
private string NextParam()
{
return ":p" + m_ParameterCounter++;
}
public override void AddToBatch(IExpectation expectation)
{
if(expectation.CanBeBatched && !(CurrentCommand.CommandText.StartsWith("INSERT INTO") && CurrentCommand.CommandText.Contains("VALUES")))
{
//NonBatching behavior
IDbCommand cmd = CurrentCommand;
LogCommand(CurrentCommand);
int rowCount = ExecuteNonQuery(cmd);
expectation.VerifyOutcomeNonBatched(rowCount, cmd);
currentBatch = null;
return;
}
totalExpectedRowsAffected += expectation.ExpectedRowCount;
log.Info("Adding to batch");
int len = CurrentCommand.CommandText.Length;
int idx = CurrentCommand.CommandText.IndexOf("VALUES");
int endidx = idx + "VALUES".Length + 2;
if (currentBatch == null)
{
// begin new batch.
currentBatch = new NpgsqlCommand();
sbBatchCommand = new StringBuilder();
m_ParameterCounter = 0;
string preCommand = CurrentCommand.CommandText.Substring(0, endidx);
sbBatchCommand.Append(preCommand);
}
else
{
//only append Values
sbBatchCommand.Append(", (");
}
//append values from CurrentCommand to sbBatchCommand
string values = CurrentCommand.CommandText.Substring(endidx, len - endidx - 1);
//get all values
string[] split = values.Split(',');
ArrayList paramName = new ArrayList(split.Length);
for (int i = 0; i < split.Length; i++ )
{
if (i != 0)
sbBatchCommand.Append(", ");
string param = null;
if (split[i].StartsWith(":")) //first named parameter
{
param = NextParam();
paramName.Add(param);
}
else if(split[i].StartsWith(" :")) //other named parameter
{
param = NextParam();
paramName.Add(param);
}
else if (split[i].StartsWith(" ")) //other fix parameter
{
param = split[i].Substring(1, split[i].Length-1);
}
else
{
param = split[i]; //first fix parameter
}
sbBatchCommand.Append(param);
}
sbBatchCommand.Append(")");
//rename & copy parameters from CurrentCommand to currentBatch
int iParam = 0;
foreach (NpgsqlParameter param in CurrentCommand.Parameters)
{
param.ParameterName = (string)paramName[iParam++];
NpgsqlParameter newParam = /*Clone()*/new NpgsqlParameter(param.ParameterName, param.NpgsqlDbType, param.Size, param.SourceColumn, param.Direction, param.IsNullable, param.Precision, param.Scale, param.SourceVersion, param.Value);
currentBatch.Parameters.Add(newParam);
}
countOfCommands++;
//check for flush
if (countOfCommands >= batchSize)
{
DoExecuteBatch(currentBatch);
}
}
protected override void DoExecuteBatch(IDbCommand ps)
{
if (currentBatch != null)
{
//Batch command now needs its terminator
sbBatchCommand.Append(";");
countOfCommands = 0;
log.Info("Executing batch");
CheckReaders();
//set prepared batchCommandText
string commandText = sbBatchCommand.ToString();
currentBatch.CommandText = commandText;
LogCommand(currentBatch);
Prepare(currentBatch);
int rowsAffected = 0;
try
{
rowsAffected = currentBatch.ExecuteNonQuery();
}
catch (Exception e)
{
if(Debugger.IsAttached)
Debugger.Break();
throw;
}
Expectations.VerifyOutcomeBatched(totalExpectedRowsAffected, rowsAffected);
totalExpectedRowsAffected = 0;
currentBatch = null;
sbBatchCommand = null;
m_ParameterCounter = 0;
}
}
protected override int CountOfStatementsInCurrentBatch
{
get { return countOfCommands; }
}
public override int BatchSize
{
get { return batchSize; }
set { batchSize = value; }
}
}
}
I also found that NHibernate is not doing batch inserts into PostgreSQL.
I identified two possible reasons:
1) Npgsql driver does not support batch inserts/updates (see forum)
2) NHibernate does not have *BatchingBatcher(Factory) for PostgreSQL(Npgsql). I tried using Devart dotConnect driver with NHibernate (I wrote custom driver for NHibernate) but it still did not worked.
I suppose this driver should also implement IEmbeddedBatcherFactoryProvider interface, but it seems not trivial for me (using one for Oracle did not worked ;) )
Did anybody managed to force Nhibarnate to do batch inserts to PostgreSQL or can confirm my conclusion?