Upload CSV data into SQL database using ASP.NET Core MVC - asp.net-core

I am trying to insert data from a .csv file into my database, but anytime I upload data, the record is empty.
This is my code so far:
[HttpPost]
public async Task<IActionResult> ImportFromExcel(IFormFile formFile)
{
var data = new MemoryStream();
await formFile.CopyToAsync(data);
data.Position = 0;
TextReader reader = new StreamReader(data);
var csvReader = new CsvReader(reader, new CsvConfiguration(System.Globalization.CultureInfo.CurrentCulture)
{
HasHeaderRecord = true,
HeaderValidated = null,
MissingFieldFound = null
});
var Name = csvReader.GetField(0).ToString();
var dep = "cccccccccc";
var pos = "bbbbbbbbbbb";
await dcx.Participants.AddAsync(new Participant
{
Name = Name,
Position = pos,
Department = dep,
});
dcx.SaveChanges();
return ViewComponent("ViewParticipants");
}
This is the sample data in my database table:

As long as the headers of your CSV match up to the names of the columns in your database, you should be able to do something like this. If the names don't match, you can use .Name("CsvColumnName") in ParticipantMap to add the name of the column in the CSV file. Example: Map(r => r.Description).Name("MyCsvDescription");.
[HttpPost]
public async Task<IActionResult> ImportFromExcel(IFormFile formFile)
{
var data = new MemoryStream();
await formFile.CopyToAsync(data);
data.Position = 0;
var conf = new CsvConfiguration(CultureInfo.InvariantCulture)
{
HasHeaderRecord = true,
HeaderValidated = null,
MissingFieldFound = null,
BadDataFound = context =>
{
bad.Add(context.RawRecord);
}
};
using (TextReader reader = new StreamReader(data))
using (var csvReader = new CsvReader(reader, config)) {
csvReader.Context.RegisterClassMap<ParticipantMap>();
var records = csvReader.GetRecords<Participant>().ToList();
var dep = "cccccccccc";
var pos = "bbbbbbbbbbb";
records.ForEach(r =>
{
r.Department = dep;
r.Position = pos;
});
await dcx.Participants.AddRangeAsync(records);
dcx.SaveChanges();
}
return ViewComponent("ViewParticipants");
}
public class ParticipantMap : ClassMap<Participant>
{
public ParticipantMap()
{
AutoMap(CultureInfo.InvariantCulture);
Map(r => r.Department).Ignore();
Map(r => r.Position).Ignore();
}
}

In my opinion, you should call csvReader.Read() to read the file row first.
You can refer to the following test code, it works fine.
[HttpPost]
public async Task<IActionResult> ImportFromExcel(IFormFile formFile)
{
var data = new MemoryStream();
await formFile.CopyToAsync(data);
data.Position = 0;
using (var reader = new StreamReader(data))
{
var bad = new List<string>();
var conf = new CsvConfiguration(CultureInfo.InvariantCulture)
{
HasHeaderRecord = true,
HeaderValidated = null,
MissingFieldFound = null,
BadDataFound = context =>
{
bad.Add(context.RawRecord);
}
};
using (var csvReader = new CsvReader(reader, conf))
{
while (csvReader.Read())
{
var Name = csvReader.GetField(0).ToString();
var pos = csvReader.GetField(1).ToString();
var dep = csvReader.GetField(2).ToString();
await dcx.Participants.AddAsync(new Participant
{
Name = Name,
Position = pos,
Department = dep,
});
dcx.SaveChanges();
}
}
}
return ViewComponent("ViewParticipants");
}
Test Result:

Related

Get incremental changes for a group in Microsoft Graph in C#

I have the following code to get users from an AAD group:
public async Task<IGroupTransitiveMembersCollectionWithReferencesPage> GetGroupMembersPageByIdAsync(string groupId)
{
return await graphServiceClient
.Groups[groupId]
.TransitiveMembers
.Request()
.Top(999)
.GetAsync();
}
public async Task<IGroupTransitiveMembersCollectionWithReferencesPage> GetGroupMembersNextPageAsnyc(
IGroupTransitiveMembersCollectionWithReferencesPage groupMembersRef,
string nextPageUrl)
{
groupMembersRef.InitializeNextPageRequest(_graphServiceClient, nextPageUrl);
return await groupMembersRef
.NextPageRequest
.GetAsync();
}
public async Task<(List<AzureADUser> users,
string nextPageUrl,
IGroupTransitiveMembersCollectionWithReferencesPage usersFromGroup)> GetFirstUsersPageAsync(Guid objectId)
{
var users = new List<AzureADUser>();
var usersFromGroup = await GetGroupMembersPageByIdAsync(objectId.ToString());
usersFromGroup.AdditionalData.TryGetValue("#odata.nextLink", out object nextLink1);
var nextPageUrl = (nextLink1 == null) ? string.Empty : nextLink1.ToString();
users.AddRange((IEnumerable<AzureADUser>)(usersFromGroup));
return (users, nextPageUrl, usersFromGroup);
}
public async Task<(List<AzureADUser> users,
string nextPageUrl,
IGroupTransitiveMembersCollectionWithReferencesPage usersFromGroup)> GetNextUsersPageAsync(string nextPageUrl, IGroupTransitiveMembersCollectionWithReferencesPage usersFromGroup)
{
var users = new List<AzureADUser>();
usersFromGroup = await GetGroupMembersNextPageAsnyc(usersFromGroup, nextPageUrl);
usersFromGroup.AdditionalData.TryGetValue("#odata.nextLink", out object nextLink2);
nextPageUrl = (nextLink2 == null) ? string.Empty : nextLink2.ToString();
users.AddRange((IEnumerable<AzureADUser>)(usersFromGroup));
return (users, nextPageUrl, usersFromGroup);
}
I'm trying to learn about how I can use delta query functionality: https://learn.microsoft.com/en-us/graph/delta-query-groups so that next time when I run this, I can get the difference (new users/removed users/updated users) and return that list. Is that possible via delta query functionality?
I had a test in my asp.net core mvc project and you can get delta information by code below.
using Azure.Identity;
using Microsoft.Graph;
public async Task<IActionResult> Index()
{
var scopes = new[] { "https://graph.microsoft.com/.default" };
var tenantId = "your_tenant_name.onmicrosoft.com";
var clientId = "azure_ad_app_id";
var clientSecret = "client_secret";
var clientSecretCredential = new ClientSecretCredential(
tenantId, clientId, clientSecret);
var graphClient = new GraphServiceClient(clientSecretCredential, scopes);
//get group members
var users = await graphClient.Groups["groupId"].TransitiveMembers.Request().Top(999).GetAsync();
//get group member delta info
var delta = await graphClient.Groups.Delta().Request().Filter("id eq 'group_id'").GetAsync();
return View();
}

Net core api to upload 1GB size csv file

I have following code segment it works for small file. But if the file is larger then application is loading for long and recieves No 'Access-Control-Allow-Origin' header is present on the requested resource.
[HttpPost]
[ScopeAuthorize(Constants.ClaimScopeSGCanManageAll, Constants.ClaimScopeUserCanManage)]
[DisableRequestSizeLimit, RequestFormLimits(MultipartBodyLengthLimit = int.MaxValue, ValueLengthLimit = int.MaxValue)]
public async Task<IActionResult> UploadFile()
{
if (!Request.Form.Files.Any())
{
throw new Common.Exceptions.ValidationException("Empty file");
}
IFormFile formFile = Request.Form.Files[0];
var csvDatas = new List<PatientsCSVItem>();
using (var reader = new StreamReader(formFile.OpenReadStream()))
{
while (!reader.EndOfStream)
{
var line = reader.ReadLine();
var values = line.Split(';');
//process csv rows
}
}
PatientCsvLog executionLog = _patientCsvManager.AddOrUpdatePatientsByCsvData(csvDatas, _userManager.GetLoggedUserId(User));
if (executionLog == null)
{
throw new ArgumentNullException(nameof(executionLog));
}
var response = new
{
NumberRecordImported = executionLog.NumberRecordImported,
NumberRecordUpdated = executionLog.NumberRecordUpdated,
NumberRecordDiscarded = executionLog.NumberRecordDiscarded,
DiscardedRecordList = executionLog.DiscardedRecordList
};
return Ok(response);
}

Azure Logic Apps internal server error 500

Am trying to create a an azure function that is triggered in a Logic Apps,
The functions purpose is to web crawl certain web sites, take the desired information, compare that with a a SQL Server database in Azure, compare if we already have that information if not add it.
My issue is that when ever i run it I get the Server 500 error, I assume its accessing the database that cause. Help?
public static async Task<IActionResult> Run(
[HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)] HttpRequest req, ILogger log
)
{
log.LogInformation("C# HTTP trigger function processed a request.");
string RequestBody = await new StreamReader(req.Body).ReadToEndAsync();
{
return await CrawlBlog(0, RequestBody);
}
}
private static async Task<IActionResult> CrawlBlog(int Picker, string req)
{
int BlogPicker = Picker;
string TheResult = req;
//Get the url we want to test
var Url = "";
if (BlogPicker == 0)
{
Url = "*********";
}
else if (BlogPicker == 1)
{
Url = "*********";
}
/*
else if (BlogPicker == 2)
{
Url = "https://azure.microsoft.com/en-in/blog/?utm_source=devglan";
}
*/
else
{
TheResult = "False we got a wrong pick";
return (ActionResult)new OkObjectResult
( new {TheResult });
}
var httpClient = new HttpClient();
var html = await httpClient.GetStringAsync(Url);
var htmlDocument = new HtmlDocument();
htmlDocument.LoadHtml(html);
//a list to add all availabel blogs we found
var Blog = new List<BlogStats>();
switch (BlogPicker)
{
case 0:
{
var divs =
htmlDocument.DocumentNode.Descendants("div")
.Where(node => node.GetAttributeValue("class", "").Equals("home_blog_sec_text")).ToList();
foreach (var divo in divs)
{
var Blogo = new BlogStats
{
Summary = divo.Descendants("p").FirstOrDefault().InnerText,
Link = divo.Descendants("a").FirstOrDefault().ChildAttributes("href").FirstOrDefault().Value,
Title = divo.Descendants("a").FirstOrDefault().InnerText
};
Blog.Add(Blogo);
}
break;
}
case 1:
{
var divs =
htmlDocument.DocumentNode.Descendants("div")
.Where(node => node.GetAttributeValue("class", "").Equals("post_header_title two_third last")).ToList();
foreach (var divo in divs)
{
//string TheSummary = "we goofed";
var ThePs = divo.Descendants("p").ToList();
var Blogo = new BlogStats
{
Summary = ThePs[1].GetDirectInnerText(),
Link = divo.Descendants("a").LastOrDefault().ChildAttributes("href").FirstOrDefault().Value,
Title = divo.Descendants("a").FirstOrDefault().InnerText
};
Blog.Add(Blogo);
}
break;
}
}
TheResult = await SqlCheck(Blog[0].Title, Blog[0].Summary, Blog[0].Link); //error 500
return (ActionResult)new OkObjectResult
(
new
{
TheResult
}
);
}
public static async Task<string> SqlCheck(string Tit, string Sumy, string Lin)
{
SqlConnectionStringBuilder builder = new SqlConnectionStringBuilder();
builder.DataSource = "flygon.database.windows.net";
builder.UserID = "*****";
builder.Password = "********";
builder.InitialCatalog = "torkoal";
System.Data.DataSet ds = new System.Data.DataSet();
SqlConnection connection = new SqlConnection(builder.ConnectionString);
connection.Open();
SqlCommand CheckCommand = new SqlCommand("SELECT * FROM TableBoto WHERE Link = #id3 ", connection);
CheckCommand.Parameters.AddWithValue("#id3", Lin);
SqlDataAdapter dataAdapter = new SqlDataAdapter(CheckCommand);
dataAdapter.Fill(ds);
int i = ds.Tables[0].Rows.Count;
if (i > 0)
{
return $" We got a Duplicates in title : {Tit}";
}
try
{
{
string query = $"insert into TableBoto(Title,Summary,Link) values('{Tit}','{Sumy}','{Lin}');";
SqlCommand command = new SqlCommand(query, connection);
SqlDataReader reader = await command.ExecuteReaderAsync();
reader.Close();
}
}
catch (SqlException)
{
// Console.WriteLine(e.ToString());
}
connection.Close();
return $" Success Ign +{Tit} + Ign {Sumy}+ Ign {Lin} Ign Success SQL ";
}
}
500 HTTP status code is a generic code which means that the server was not able to process the request due to some issues, First step would be to add some exception handling to your function and see if the failure occurs and where it occurs.
On Side note, you should not use HTTP client in the way used in the code, you should not new it up every time your function executes, this client should be static in nature. Refer Manage connections in Azure Functions

How to use Scroll while passing raw json Query to ElasticSearch using NEST

var query = #"
{
""query"": {
""match_all"": { }
}
}";
Func<SearchRequestParameters, SearchRequestParameters> requestParameters = a =>
a.SearchType(SearchType.Scan).Scroll(TimeSpan.FromSeconds(60));
var searchResult = await client.LowLevel.SearchAsync<SearchResponse<T>>(indexName, mappingName, query , requestParameters)
if (searchResult.Body.IsValid)
{
var scrollNo = 0;
var results = await client.ScrollAsync<T>("10s", searchResult.Body.ScrollId);
while (results.Documents.Any())
{
documents.AddRange(results.Documents);
scrollNo++;
results = await client.ScrollAsync<T>("10s", results.ScrollId);
return new Customresponse<T>
{
Documents = documents,
total = result.Body.Total
};
}
Would like to pull all data using scroll while passing raw json query. but scroll is not working properly while passing json raw query. Can anyone help on this ?.
Your example is nearly there but not quite; you're missing a closing brace for the while loop to collect all documents before returning the custom response.
Here's an example I just ran on the Stackoverflow data set, to return all questions tagged with nest
private IElasticClient _client;
void Main()
{
var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
var defaultIndex = "default-index";
var connectionSettings = new ConnectionSettings(pool);
_client = new ElasticClient(connectionSettings);
var query = #"
{
""query"": {
""term"": {
""tags"": {
""value"": ""nest""
}
}
}
}";
var result = RunScrollAsync(query).Result.Dump();
}
private async Task<Customresponse<Question>> RunScrollAsync(string query)
{
var scrollTime = "10s";
// omit the .SearchType(Scan) which is deprecated. Not
// specifying means the first response contains the first set
// of documents
var esResponse = await _client.LowLevel.SearchAsync<SearchResponse<Question>>(
"posts",
"question",
query, r => r.Scroll(TimeSpan.FromSeconds(10))).ConfigureAwait(false);
if (esResponse.Body.IsValid && esResponse.Body.Documents.Any())
{
// assume you have less than 2,147,483,647 documents to return?
var documents = new List<Question>((int)esResponse.Body.Total);
documents.AddRange(esResponse.Body.Documents);
var scrollNo = 0;
var response = await _client.ScrollAsync<Question>(scrollTime, esResponse.Body.ScrollId).ConfigureAwait(false);;
// keep scrolling until no more documents are returned
while (response.Documents.Any())
{
documents.AddRange(response.Documents);
scrollNo++;
response = await _client.ScrollAsync<Question>(scrollTime, response.ScrollId).ConfigureAwait(false);;
}
return new Customresponse<Question>
{
Documents = documents,
total = response.Total
};
}
// return an empty result.
// Or throw an exception, or log - whatever you need to do
return new Customresponse<Question>
{
Documents = Enumerable.Empty<Question>(),
total = 0
};
}
public class Customresponse<T>
{
public IEnumerable<T> Documents { get; set; }
public long total { get; set; }
}
This returns all 342 questions, with a total of 342 (Data set is from June 2016).

Passing list of object to Web API using RestSharp Client

I'm trying to send list of objects from MVC to WEBAPI using below methods. API is able to able receive the list from controller but, value of each item in the list is either empty/null on API side.
Can anyone please help me to fix this?
Controller Method:
private List<FCM.Models.Facility> GetFacilityDetails()
{
var url = "http://localhost:64664/";
var facilies = new List<Facility>();
facilies.Add( new Facility{ FCLT_ID = 100, FCLT_NM = "Facility 100" });
facilies.Add( new Facility{ FCLT_ID = 200, FCLT_NM = "Facility 200" });
facilies.Add( new Facility{ FCLT_ID = 300, FCLT_NM = "Facility 300" });
var json = JsonConvert.SerializeObject(facilies);
var _client = new RestClient(url);
var request = new RestRequest("api/facility/details", Method.GET) { RequestFormat = DataFormat.Json };
facilies.ForEach(fclt =>
request.AddParameter("facilites", fclt, ParameterType.GetOrPost));
var response = _client.Execute<List<FCM.Models.Facility>>(request);
if (response.Data == null)
{
throw new Exception(response.ErrorMessage);
}
return response.Data;
}
WebAPI method:
[Route("api/facility/details")]
public IEnumerable<Facility> GetFullAddress([FromUri] IEnumerable<Facility> facilities)
{
return null;
}
Like the comment suggested you maybe want to issue a POST request instead, but if you would like to send an array with a GETrequest you could do it like this (with System.Net.Http.HttpClient):
Add a Format method to you Facility class:
public class Facility
{
public int FCLT_ID { get; set; }
public string FCLT_NM { get; set; }
public string Format(int index)
{
return $"[{index}].FCLT_ID={FCLT_ID}&[{index}].FCLT_NM={FCLT_NM}";
}
}
Define a class which can format the array values:
public class FacilityList : List<Facility>
{
public string Format()
{
var builder = new StringBuilder();
for (var i = 0; i < Count; i++)
{
builder.Append(this[i].Format(i));
if(i != Count -1)
{
builder.Append("&");
}
}
return builder.ToString();
}
}
And then issue the request:
var client = new HttpClient()
{
BaseAddress = new Uri("http://localhost:64664/"),
DefaultRequestHeaders = {Accept = {new MediaTypeWithQualityHeaderValue("application/json")}}
};
var facilities = new FacilityList
{
new Facility {FCLT_ID = 100, FCLT_NM = "Facility 100"},
new Facility {FCLT_ID = 200, FCLT_NM = "Facility 200"},
new Facility {FCLT_ID = 300, FCLT_NM = "Facility 300"}
};
var format = facilities.Format();
var response = client.GetAsync("api/facility/details?" + format).GetAwaiter().GetResult();
var result = JsonConvert.DeserializeObject<IEnumerable<Facility>>(response.Content.ReadAsStringAsync().GetAwaiter().GetResult());
This will bind to your controller action:
[Route("api/facility/details")]
public IHttpActionResult Get([FromUri] IEnumerable<Facility> facilities)
{
// Do stuff..
return Ok(facilities);
}