Npgsql - BulkCopy from ConcurrentQueue<T> - bulkinsert

Let's say I have a method like this:
public async Task BulkCopy(ConcurrentQueue<Model> modelQueue, string connectionString)
{
while(modelQueue.IsEmpty == false)
{
try
{
using(NpgsqlConnection connection = new NpgsqlConnection(connectionString))
{
await connection.OpenAsync();
using(var writer = connection.BeginBinaryImport("COPY myTable (Id,Name,Something,SomethingElse)"))
{
// Is this what I'm supposed to do?
foreach(Model item in modelQueue)
{
writer.WriteRow(item);
}
}
}
}
}
}
Model has properties Guid Id, string Name, string Something, string SomethingElse(just like the table).
Can I use WriteRow() and pass in an entire object? Is this implementation way to go or I am doing it wrong?

The answer to this is pretty simple, since ConcurrentQueue implements IEnumerable all I had to do is run through the queue and write the data.
public async Task BulkCopy(ConcurrentQueue<Model> modelQueue, string connectionString)
{
while(modelQueue.IsEmpty == false)
{
try
{
using(NpgsqlConnection connection = new NpgsqlConnection(connectionString))
{
await connection.OpenAsync();
using(var writer = connection.BeginBinaryImport("COPY myTable (Id,Name,Something,SomethingElse) FROM STDIN (FORMAT BINARY)"))
{
foreach(Model item in modelQueue)
{
writer.StartRow();
writer.Write(item.Id, NpgsqlTypes.NpgsqlDbType.Uuid);
writer.Write(item.Name);
writer.Write(item.Something);
writer.Write(item.SomethingElse);
}
}
}
}
}
}
This seems to do the job. The time needed for 30000 records is average 540ms.

Related

Is there a way to improve the code? .Net Core 6.0 and Cosmos Database with SQL API

It's my first time using .Net Core 6.0 WebAPI and Cosmos Database with SQL API, so I'm concerned that I've done things correctly.
I have used /address/zipcode as a partition key in the below sample
public class FamilyService : IFamilyService
{
public FamilyService(CosmosClient dbClient, string databaseName, string containerName)
{
this._container = dbClient.GetContainer(databaseName, containerName);
}
public Task<Family> GetFamilyDataAsync(string id, string zipcode)
{
return Task.FromResult(_container.GetItemLinqQueryable<Family>(allowSynchronousQueryExecution: true)
.Where(p => p.Id == id && p.Address.ZipCode == zipcode)
.ToList().First());
}
public async Task AddFamilyDataAsync(Family family)
{
await this._container.CreateItemAsync<Family>(family, new PartitionKey(family.Address.ZipCode));
}
public async Task UpdateFamilyDataAsync(Family family)
{
await this._container.ReplaceItemAsync<Family>(family, family.Id, new PartitionKey(family.Address.ZipCode));
}
public async Task DeleteFamilyDataAsync(string id, string zipcode)
{
await this._container.DeleteItemAsync<Family>(id, new PartitionKey(zipcode));
}
private async Task<bool> GetFamilyDataFromId(string id, string zipcode)
{
string query = $"select * from c where c.id=#familyId";
QueryDefinition queryDefinition = new QueryDefinition(query).WithParameter("#familyId", id);
List<Family> familyResults = new List<Family>();
FeedIterator streamResultSet = _container.GetItemQueryStreamIterator(
queryDefinition,
requestOptions: new QueryRequestOptions()
{
PartitionKey = new PartitionKey(zipcode),
MaxItemCount = 10,
MaxConcurrency = 1
});
while (streamResultSet.HasMoreResults)
{
using (ResponseMessage responseMessage = await streamResultSet.ReadNextAsync())
{
if (responseMessage.IsSuccessStatusCode)
{
dynamic streamResponse = FromStream<dynamic>(responseMessage.Content);
List<Family> familyResult = streamResponse.Documents.ToObject<List<Family>>();
familyResults.AddRange(familyResult);
}
else
{
return false;
}
}
}
if (familyResults != null && familyResults.Count > 0)
{
return true;
}
return familyResults;
}
Especially I want to focus more on those methods that retrieves the data from the database.
Update#1 : I have updated the code as suggested by #Mark Brown.
public class FamilyService : IFamilyService
{
private Container _container;
private static readonly JsonSerializer Serializer = new JsonSerializer();
public FamilyService(CosmosClient dbClient, string databaseName, string containerName)
{
this._container = dbClient.GetContainer(databaseName, containerName);
}
public async Task<Family> GetFamilyDataAsync(string id, string zipCode)
{
return await _container.ReadItemAsync<Family>(id, new PartitionKey(zipCode));
}
public async Task<List<Family>> GetAllFamilyDataAsync(string zipCode)
{
string query = $"SELECT * FROM Family";
QueryDefinition queryDefinition = new QueryDefinition(query);
List<Family> familyResults = new List<Family>();
FeedIterator<Family> feed = _container.GetItemQueryIterator<Family>(
queryDefinition,
requestOptions: new QueryRequestOptions()
{
PartitionKey = new PartitionKey(zipCode),
MaxItemCount = 10,
MaxConcurrency = 1
});
while (feed.HasMoreResults)
{
FeedResponse<Family> response = await feed.ReadNextAsync();
foreach (Family item in response)
{
familyResults.Add(item);
}
}
return familyResults;
}
public async Task<List<Family>> GetAllFamilyDataByLastNameAsync(string lastName, string zipCode)
{
string query = $"SELECT * FROM Family where Family.lastName = #lastName";
QueryDefinition queryDefinition = new QueryDefinition(query).WithParameter("#lastName", lastName);
List<Family> familyResults = new List<Family>();
FeedIterator<Family> feed = _container.GetItemQueryIterator<Family>(
queryDefinition,
requestOptions: new QueryRequestOptions()
{
PartitionKey = new PartitionKey(zipCode),
MaxItemCount = 10,
MaxConcurrency = 1
});
while (feed.HasMoreResults)
{
FeedResponse<Family> response = await feed.ReadNextAsync();
foreach (Family item in response)
{
familyResults.Add(item);
}
}
return familyResults;
}
public async Task<List<Family>> GetAllFamilyDataPaginatedAsync(int pageNumber, string zipCode)
{
int pageSize = 2;
int itemsToSkip = (pageSize * pageNumber) - pageSize;
QueryRequestOptions options = new QueryRequestOptions { MaxItemCount = pageSize };
string continuationToken = null;
List<Family> familyResults = new List<Family>();
IQueryable<Family> query = this._container
.GetItemLinqQueryable<Family>(false, continuationToken, options)
.Where(i => i.Address.ZipCode == zipCode)
.Skip(itemsToSkip);
using (FeedIterator<Family> iterator = query.ToFeedIterator<Family>())
{
FeedResponse<Family> feedResponse = await iterator.ReadNextAsync();
foreach (Family item in feedResponse)
{
familyResults.Add(item);
}
}
return familyResults;
}
public async Task AddFamilyDataAsync(Family family)
{
await this._container.CreateItemAsync<Family>(family, new PartitionKey(family.Address.ZipCode));
}
public async Task UpdateFamilyDataAsync(Family family)
{
await this._container.ReplaceItemAsync<Family>(family, family.Id, new PartitionKey(family.Address.ZipCode));
}
public async Task DeleteFamilyDataAsync(string id, string zipCode)
{
await this._container.DeleteItemAsync<Family>(id, new PartitionKey(zipCode));
}
}
If you are searching for something with the partition key and id then that can only return a single item as id must be unique within a pk. So GetFamilyDataAsync() can be implemented using ReadItemAsync() rather than a query. PS: Not sure why you would ever do this, allowSynchronousQueryExecution: true
GetFamilyDataFromId() also can be implemented using ReadItemAsync(). Even if this did require a query, it doesn't make sense to implement to return a stream, then deserialize into an object. Just use the variant that can deserialize directly into a POCO.
Can't say if this is completely optimal for everything, but you can go here to find more things as a place to start, https://learn.microsoft.com/en-us/azure/cosmos-db/sql/best-practice-dotnet
The other thing to call out is zip code as a partition key. This might work at small scale, but it could run into issues at larger scale depending on how many families and how many records for each family are stored within a single partition. Max size for a single partition key value is 20 GB. Fine for small workloads but possibly an issue at larger scale.

Blazor Server - Return Count from SQL - Returns System.Collections.Generic.List`1[System.Object]

I'm trying to return a count of row's from a stored procedure in SQL. I've been working at this for hours now. I'm at a complete loss.
SQL Code: Returns result of 49 when run in SQL.
begin
select Count (EmployeeID)
from dbo.Employees
Where (Employees.Active = 'True')
end
Component Code:
#attribute [Authorize(Roles = "Admin")]
#using System.Collections;
#using WahlenDataAccessLibrary;
#using WahlenDataAccessLibrary.Models;
#using BlazorApp1.Models;
#inject IEmployeesData employeeData;
<div class="widget">
<h5 class="widget--title">Active Employees</h5>
<div class="widget--body">
<p class="widget--number">
<span>#employeeCount</span>
Active Employees
</p>
</div>
</div>
#code {
private string employeeCount { get; set; }
//private IEmployeeModel employeeCount = new EmployeeModel();
protected override async Task OnInitializedAsync()
{
var count = await employeeData.EmployeeCount();
//string employeeCount = await employeeData.EmployeeCount();
string employeeCount = count.ToString();
Console.WriteLine(employeeCount);
if (employeeCount != null)
{
Console.WriteLine("generic value");
}
else
{
Console.WriteLine("no value");
}
}
}
DataFile Code: To get the value from stored procedure.
public async Task<string> EmployeeCount()
{
var employeeCount = await _db.LoadValue("dbo.spWidget_EmployeeCount", "DefaultConnection");
return employeeCount.ToString();
}
}
The DataFile where 'LoadValue' is used. This is linked back to my SqlDataAccess File which uses this code.
public async Task<string> LoadValue(string storedProcedure, string connectionStringName)
{
string connectionString = _config.GetConnectionString(connectionStringName);
using (IDbConnection connection = new SqlConnection(connectionString))
{
var data = await connection.QueryAsync(storedProcedure,
commandType: CommandType.StoredProcedure);
return data.ToString();
}
}
When the application is running the console writes.
System.Collections.Generic.List`1[System.Object]
no value
The
System.Collections.Generic.List`1[System.Object]
comes from
var data = await connection.QueryAsync(storedProcedure,
commandType: CommandType.StoredProcedure);
return data.ToString();
the var data is actually an object. Callint .ToString() on a object will print out the type of the object (if the method is not over written).
Please check if the QueryAsync has a generic version. Usually it does and will make a type cast for you in the background. Try something like QueryAsync<int>. Or try to find the rigth method (with a single return item, instead of a list) on your ORM/db provider.
From the current context it is not possible to tell what db provider you are using.
I'm going to update the code here, as I've got it working.
Task<IEnumerable> was more so the answer here. If you have an sql query that is simply counting rows, this is how you access that data on Blazor on your front end. You can return the value as a string.
My sqlaccess file code now looks like this.
public async Task<IEnumerable<int>> LoadValue(string storedProcedure, string connectionStringName)
{
string connectionString = _config.GetConnectionString(connectionStringName);
using (IDbConnection connection = new SqlConnection(connectionString))
{
var data = await connection.QueryAsync<int>(storedProcedure,
commandType: CommandType.StoredProcedure);
return data;
}
}
And this is on the front end component.
private string employeeCount { get; set; }
protected override async Task OnInitializedAsync()
{
employeeCount = await employeeData.EmployeeCount();
Console.WriteLine(employeeCount);
if (employeeCount != null)
{
Console.WriteLine("generic value");
}
else
{
Console.WriteLine("no value");
}

Update existing table row in the same method that uses HttpPost to add a new entry

I have a web game that uses .NetCore Entity Framework.
I have one method that uses HttpPost to create a new Monster in the database.
This method also needs to add a foreign key, the new MonsterId, to an existing Dungeon in the table called DungeonList.
I got the part where the method creates a new Monster correctly.
However I'm not sure how to insert the new MonsterId into the appropriate Dungeon of DungeonList.
I'm not exactly sure how to get the Id of the Dungeon.
Should I pass in the DungeonId from the frontend?
I'm really confused.
Here is what I have so far but I am not sure where to go from here.
I'd love some advice...thank you!
[HttpPost]
public async Task<ActionResult<MonsterList>> PostMonsterList(MonsterList monsterList)
{
monsterList.MonsterId = Guid.NewGuid();
_context.MonsterList.Add(monsterList);
var dungeonListRef = new DungeonList();
if(dungeonListRef.MonsterId == null)
{
// ????
}
try
{
await _context.SaveChangesAsync();
}
catch (DbUpdateException)
{
if (MonsterListExists(monsterList.MonsterId))
{
return Conflict();
}
else
{
throw;
}
}
_context.DungeonList.Add(dungeonListRef);
return CreatedAtAction("GetMonsterList", new { id = monsterList.MonsterId }, monsterList);
}
Add Dungeon drop down list in your "Add new Monster" page. Send drop down list's dungeonID to PostMonsterList function.
[HttpPost]
public async Task<ActionResult<MonsterList>> PostMonsterList(Guid dungeonId, MonsterList monsterList)
{
Guid newMonsterId = Guid.NewGuid();
monsterList.MonsterId = newMonsterId;
_context.MonsterList.Add(monsterList);
try
{
await _context.SaveChangesAsync();
}
catch (DbUpdateException)
{
if (MonsterListExists(monsterList.MonsterId))
{
return Conflict();
}
else
{
throw;
}
}
var dungeonList = _context.DungeonList.Where(x => x.DungeonId == dungeonId).FirstOrDefault();
dungeonList.MonsterId = newMonsterId;
_context.DungeonList.Update(dungeonList);
await _context.SaveChangesAsync();
return CreatedAtAction("GetMonsterList", new { id = monsterList.MonsterId }, monsterList);
}

autodesk design automation

FATAL ERROR: Unhandled Access Violation Reading 0x0008 Exception at 1d8257a5h
Failed missing output
I finally made it work with HostApplicationServices.getRemoteFile in local AutoCAD, then migrated it to Design Automation. It is also working now. The below is the command of .NET plugin.
To have a simple test, I hard-coded the URL in the plugin. you could replace the URL with the workflow at your side (either by an json file, or input argument of Design Automation)
My demo ReadDWG the entities from the remote URL file, then wblock the entities to current drawing (HostDWG), finally save current drawing.
Hope it helps to address the problem at your side.
.NET command
namespace PackageNetPlugin
{
class DumpDwgHostApp: HostApplicationServices
{
public override string FindFile(string fileName,
Database database,
FindFileHint hint)
{
throw new NotImplementedException();
}
public override string GetRemoteFile(Uri url,
bool ignoreCache)
{
//return base.GetRemoteFile(url, ignoreCache);
Database db =
Autodesk.AutoCAD.ApplicationServices.Application.
DocumentManager.MdiActiveDocument.Database;
string localPath = string.Empty;
if (ignoreCache)
{
localPath =
Autodesk.AutoCAD.ApplicationServices.Application.
GetSystemVariable("STARTINFOLDER") as string;
string filename =
System.IO.Path.GetFileName(url.LocalPath);
localPath += filename;
using (var client = new WebClient())
{
client.DownloadFile(url, localPath);
}
}
return localPath;
}
public override bool IsUrl(string filePath)
{
Uri uriResult;
bool result = Uri.TryCreate(filePath,
UriKind.Absolute, out uriResult)
&& (uriResult.Scheme == Uri.UriSchemeHttp ||
uriResult.Scheme == Uri.UriSchemeHttps);
return result;
}
}
public class Class1
{
[CommandMethod("MyPluginCommand")]
public void MyPluginCommand()
{
try {
string drawingPath =
#"https://s3-us-west-2.amazonaws.com/xiaodong-test-da/remoteurl.dwg";
DumpDwgHostApp oDDA = new DumpDwgHostApp();
string localFileStr = "";
if (oDDA.IsUrl(drawingPath)){
localFileStr = oDDA.GetRemoteFile(
new Uri(drawingPath), true);
}
if(!string.IsNullOrEmpty(localFileStr))
{
//source drawing from drawingPath
Database source_db = new Database(false, true);
source_db.ReadDwgFile(localFileStr,
FileOpenMode.OpenTryForReadShare, false, null);
ObjectIdCollection sourceIds =
new ObjectIdCollection();
using (Transaction tr =
source_db.TransactionManager.StartTransaction())
{
BlockTableRecord btr =
(BlockTableRecord)tr.GetObject(
SymbolUtilityServices.GetBlockModelSpaceId(source_db),
OpenMode.ForRead);
foreach (ObjectId id in btr)
{
sourceIds.Add(id);
}
tr.Commit();
}
//current drawing (main drawing working with workitem)
Document current_doc =
Autodesk.AutoCAD.ApplicationServices.Application.
DocumentManager.MdiActiveDocument;
Database current_db = current_doc.Database;
Editor ed = current_doc.Editor;
//copy the objects in source db to current db
using (Transaction tr =
current_doc.TransactionManager.StartTransaction())
{
IdMapping mapping = new IdMapping();
source_db.WblockCloneObjects(sourceIds,
SymbolUtilityServices.GetBlockModelSpaceId(current_db),
mapping, DuplicateRecordCloning.Replace, false);
tr.Commit();
}
}
}
catch(Autodesk.AutoCAD.Runtime.Exception ex)
{
Autodesk.AutoCAD.ApplicationServices.Application.
DocumentManager.MdiActiveDocument.Editor.WriteMessage(ex.ToString());
}
}
}
}

service.close() vs. service.abort() - WCF example

In one of the WCF tutorials, I saw the following sample code:
Dim service as ...(a WCF service )
try
..
service.close()
catch ex as Exception()
...
service.abort()
end try
Is this the correct way to ensure that resources (i.e. connections) are released even under error conditions?
See Indisposable: WCF Gotcha #1*, where he comes up with a convenient wrapper method:
public delegate void UseServiceDelegate<T>(T proxy);
public static class Service<T>
{
public static ChannelFactory<T> _channelFactory = new ChannelFactory<T>("");
public static void Use(UseServiceDelegate<T> codeBlock)
{
var proxy = (IClientChannel)_channelFactory.CreateChannel();
var success = false;
try
{
codeBlock((T)proxy);
proxy.Close();
success = true;
}
finally
{
if (!success)
{
proxy.Abort();
}
}
}
}
Usage:
Service<IOrderService>.Use(
orderService =>
{
orderService.PlaceOrder(request);
});
* Link removed as it appears to be malicious.
I've had good luck with this model:
Dim service As New MyService()
Dim closed As Boolean = False
Try
service.Open()
If Not service.State = ServiceModel.CommunicationState.Opened Then
''Handle a not-opened state here
End If
service.MyMethod()
service.Close()
closed = true
Catch ex As Exception
''Handle errors here
Finally
If Not closed Then
service.Abort()
End If
End Try
service = Nothing
You've got the general idea correct. I've used the following extension method to keep the lines of repetitive code to a minimum.
public static class ICommunicationObjectExtensions
{
public static void SafelyCloseConnection(this ICommunicationObject objectToClose)
{
bool success = false;
try
{
objectToClose.Close();
success = true;
}
finally
{
if (!success)
{
objectToClose.Abort();
}
}
}
}
Example of code using this extension method:
HelloWorldServiceClient client = new HelloWorldServiceClient();
HelloWorldDataContract dc = new HelloWorldDataContract();
try
{
client.Open();
dc = client.SayHello();
} // Add catch blocks here for anything you want to handle.
finally
{
client.SafelyCloseConnection();
}
Of course this is C#, but I think that should still be of help.
If you use a client side cache, you might consider using Expression Trees (see http://thegrenade.blogspot.com/2009/07/using-expression-trees-for-more-elegant.html):
private static TEntity GetItem<TProxy, TEntity, TIdentity>(Expression<Func<TProxy, TIdentity, TEntity>> expression, TProxy proxy, TIdentity id)
where TEntity : class
where TProxy : ICommunicationObject
{
TEntity item = Cache.GetItem<TEntity, TIdentity>(id);
if (item == null)
{
try
{
var originalDelegate = expression.Compile();
item = originalDelegate.Invoke(proxy, id);
}
finally
{
try{ proxy.Close(); }
finally { proxy.Abort(); }
}
Cache.AddItem<TEntity, TIdentity>(item);
}
return item;
}
Usage:
Product p = GetItem((client, identifier) => client.GetProduct(identifier), new CatalogServiceClient(), 123);