Which is the most performant way to read a large csv file in .NET?
Using FileStream? or another class?
Thanks!
You can use the StreamReader returned by FileInfo.OpenText:
Dim file As New FileInfo("path\to\file")
Using reader As StreamReader = file.OpenText()
While Not reader.EndOfStream
Dim nextLine As String = reader.ReadLine()
ProcessCsvLine(nextLine)
End While
End Using
If you want to read it all into memory, a simple File.ReadAllText() will do just fine.
EDIT: If your file is indeed very large, then you can use the StreamReader class, see here for details. This approach is sometimes inevitable but should mostly be avoided for style reasons. See here for a more in-depth discussion.
The most efficient way of doing this is by taking advantage of deffered execution in LINQ. You can create a simple Linq-To-Text function that read one line at a time, work on it and then continue. This is really helpful since the file is really large.
I would desist from using the ReadBlock or ReadBlock or ReadToEnd methods of StreamReader class since they tend to read a number of lines at once or even the entire lines in the file. This ends up consuming more memory than if a line was read one at a time.
public static IEnumerable<string> Lines(this StreamReader source)
{
String line;
if (source == null)
throw new ArgumentNullException("source");
while ((line = source.ReadLine()) != null)
{
yield return line;
}
}
Note that the function is an extension method of the StreamReader class. This means it can be used as follows:
class Program
{
static void Main(string[] args)
{
using(StreamReader streamReader = new StreamReader("TextFile.txt"))
{
var tokens = from line in streamReader.Lines()
let items = line.Split(',')
select String.Format("{0}{1}{2}",
items[1].PadRight(16),
items[2].PadRight(16),
items[3].PadRight(16));
}
}
}
I had very good experience with this library:
http://www.codeproject.com/KB/database/CsvReader.aspx
I am using this library for Csv reading. This is really nice to use
http://www.codeproject.com/Articles/11698/A-Portable-and-Efficient-Generic-Parser-for-Flat-F
Related
We use non-manage DLL that has a funciton to replace text in PDF document (http://www.debenu.com/docs/pdf_library_reference/ReplaceTag.php).
We are trying to move to managed solution (ITextSharp or PdfSharp).
I know that this question has been asked before and that the answers are "you should not do it" or "it is not easily supported by PDF".
However there exists a solution that works for us and we just need to convert it to C#.
Any ideas how I should approach it?
According to your library reference link, you use the Debenu PDFLibrary function ReplaceTag. According to this Debenu knowledge base article
the ReplaceTag function simply replaces text in the page’s content stream, so for most documents it wouldn’t have any effect. For some simple documents it might be able to replace content, but it really depends on how the PDF was constructed. Essentially it’s the same as doing:
DPL.CombineContentStreams();
string content = DPL.GetContentStreamToString();
DPL.SetPageContentFromString(content.Replace("Moby", "Mary"));
That should be possible with any general purpose PDF library, it definitely is with iText(Sharp):
void VerySimpleReplaceText(string OrigFile, string ResultFile, string origText, string replaceText)
{
using (PdfReader reader = new PdfReader(OrigFile))
{
byte[] contentBytes = reader.GetPageContent(1);
string contentString = PdfEncodings.ConvertToString(contentBytes, PdfObject.TEXT_PDFDOCENCODING);
contentString = contentString.Replace(origText, replaceText);
reader.SetPageContent(1, PdfEncodings.ConvertToBytes(contentString, PdfObject.TEXT_PDFDOCENCODING));
new PdfStamper(reader, new FileStream(ResultFile, FileMode.Create, FileAccess.Write)).Close();
}
}
WARNING: Just like in case of the Debenu function, for most documents this code wouldn’t have any effect or would even be destructive. For some simple documents it might be able to replace content, but it really depends on how the PDF was constructed.
By the way, the Debenu knowledge base article continues:
If you created a PDF using Debenu Quick PDF Library and a standard font then the ReplaceTag function should work – however, for PDFs created with tools that do subsetted fonts or even kerning (where words will be split up) then the search text probably won’t be in the content in a simple format.
So in short, the ReplaceTag function will only work in some limited scenarios and isn’t a function that you can rely on for searching and replacing text.
Thus, if during your move to managed solution you also change the way the source documents are created, chances are that neither the Debenu PDFLibrary function ReplaceTag nor the code above will be able to change the content as desired.
for pdfsharp users heres a somewhat usable function, i copied from my project and it uses an utility method which is consumed by othere methods hence the unused result.
it ignores whitespaces created by Kerning, and therefore may mess up the result (all characters in the same space) depending on the source material
public static void ReplaceTextInPdfPage(PdfPage contentPage, string source, string target)
{
ModifyPdfContentStreams(contentPage, stream =>
{
if (!stream.TryUnfilter())
return false;
var search = string.Join("\\s*", source.Select(c => c.ToString()));
var stringStream = Encoding.Default.GetString(stream.Value, 0, stream.Length);
if (!Regex.IsMatch(stringStream, search))
return false;
stringStream = Regex.Replace(stringStream, search, target);
stream.Value = Encoding.Default.GetBytes(stringStream);
stream.Zip();
return false;
});
}
public static void ModifyPdfContentStreams(PdfPage contentPage,Func<PdfDictionary.PdfStream, bool> Modification)
{
for (var i = 0; i < contentPage.Contents.Elements.Count; i++)
if (Modification(contentPage.Contents.Elements.GetDictionary(i).Stream))
return;
var resources = contentPage.Elements?.GetDictionary("/Resources");
var xObjects = resources?.Elements.GetDictionary("/XObject");
if (xObjects == null)
return;
foreach (var item in xObjects.Elements.Values.OfType<PdfReference>())
{
var stream = (item.Value as PdfDictionary)?.Stream;
if (stream != null)
if (Modification(stream))
return;
}
}
Quick search here yielded nothing. So, I have started using some rather roundabout ways to use StreamWriter in my WinRT Application. Reading works well, writing works differently. What' I'm seeing is that when I select my file to write, if I choose a new file then no problem. The file is created as I expect. If I choose to overwrite a file, then the file is overwritten to a point, but the point where the stream stops writing, if the original file was large, then the old contents exist past where my new stream writes.
The code is as such:
public async void WriteFile(StorageFile selectedFileToSave)
{
// At this point, selectedFileToSave is from the Save File picker so can be a enw or existing file
StreamWriter writeStream;
Encoding enc = new UTF8Encoding();
Stream dotNetStream;
dotNetStream = await selectedFileToSave.OpenStreamForWriteAsync();
StreamWriter writeStream = new StreamWriter(dotNetStream, enc);
// Do writing here
// Close
writeStream.Write(Environment.NewLine);
await writeStream.FlushAsync();
await dotNetStream.FlushAsync();
}
Can anyone offer clues on what I could be missing? There are lots of functions missing in WinRT, so not really following ways to get around this
Alternatively you can set length of the stream to 0 with SetLength method before using StreamWriter:
var stream = await file.OpenStreamForWriteAsync();
stream.SetLength(0);
using (var writer = new StreamWriter(stream))
{
writer.Write(text);
}
Why not just use the helper methods in FileIO class? You could call:
FileIO.WriteTextAsync(selectedFileToSave, newTextContents);
If you really need a StreamWriter, first truncate the file by calling
FileIO.WriteBytesAsync(selectedFileToSave, new byte[0]);
And then continue with your existing code.
i wrote a function for download a webpage : function like:
public string GetWebPage(string sURL)
{
System.Net.WebResponse objResponse = null;
System.Net.WebRequest objRequest = null;
System.IO.StreamReader objStreamReader = null;
string sResultPage = null;
try
{
objRequest = System.Net.HttpWebRequest.Create(sURL);
objResponse = objRequest.GetResponse();
objStreamReader = new System.IO.StreamReader(objResponse.GetResponseStream());
sResultPage = objStreamReader.ReadToEnd();
return sResultPage;
}
catch (Exception ex)
{
return "";
}
}
But my problem is that. when this function working at that time application goto freeze (not response) and that time my can't not do any thing. How can i solve this problem. when downloading at time user can do other thing in my application.
Welcome to the world of blocking IO.
Consider the following:
You want your program to download a web page and then return the first 10 letters it finds in the source html. Your code might look like this:
...
string page = GetWebPage("http://example.com"); // download web page
page = page.Substring(0, 10);
Console.WriteLine(page);
....
When your program calls GetWebPage(), it must WAIT for the web page to be fully downloaded before it can possibly try to call Substring() - else it may try to get the substring before it actually downloads the letters.
Now consider your program. You've got lots of code - maybe a GUI interface running - and it's all executing line by line one instruction at a time. When your code calls GetWebPage(), it can't possibly continue executing additional code until that request is fully finished. Your entire program is waiting on that request to finish.
The problem can be solved in a few different ways, and the best solution depends on exactly what you're doing with your code. Ideally, your code needs to execute asynchronously. c# has methods that can handle a lot of this for you, but one way or another, you're going to want to start some work - downloading the web page in your case - and then continue executing code until your main thread is notified that the webpage is fully downloaded. Then your main thread can begin parsing the return value.
I'm assuming that since you've asked this question, you are very new to threads and concurrency in general. You have a lot of work to do. Here are some resources for you to read up about threading and implementing concurrency in c#:
C# Thread Introduction
.NET Asynchronous IO Design
the best was is to use thread
new Thread(download).Start(url);
and if your download page size is large use chunk logic.
HttpWebRequest ObjHttpWebRequest = (HttpWebRequest)HttpWebRequest.Create(Convert.ToString(url));
ObjHttpWebRequest.AddRange(99204);
ObjHttpWebRequest.Timeout = Timeout.Infinite;
ObjHttpWebRequest.Method = "get";
HttpWebResponse ObjHttpWebResponse = (HttpWebResponse)ObjHttpWebRequest.GetResponse();
Stream ObjStream = ObjHttpWebResponse.GetResponseStream();
StreamReader ObjStreamReader = new StreamReader(ObjStream);
byte[] buffer = new byte[1224];
int length = 0;
while ((length = ObjStream.Read(buffer, 0, buffer.Length)) > 0)
{
downloaddata += Encoding.GetEncoding(936).GetString(buffer);
I feel embarrassed to ask this question as I feel like I should already know. However, given I don't....I want to know how to read large files from disk to a database without getting an OutOfMemory exception. Specifically, I need to load CSV (or really tab delimited files).
I am experimenting with CSVReader and specifically this code sample but I'm sure I'm doing it wrong. Some of their other coding samples show how you can read streaming files of any size, which is pretty much what I want (only I need to read from disk), but I don't know what type of IDataReader I could create to allow this.
I am reading directly from disk and my attempt to ensure I don't ever run out of memory by reading too much data at once is below. I can't help thinking that I should be able to use a BufferedFileReader or something similar where I can point to the location of the file and specify a buffer size and then CsvDataReader expects an IDataReader as it's first parameter, it could just use that. Please show me the error of my ways, let me be rid of my GetData method with it's arbitrary file chunking mechanism and help me out with this basic problem.
private void button3_Click(object sender, EventArgs e)
{
totalNumberOfLinesInFile = GetNumberOfRecordsInFile();
totalNumberOfLinesProcessed = 0;
while (totalNumberOfLinesProcessed < totalNumberOfLinesInFile)
{
TextReader tr = GetData();
using (CsvDataReader csvData = new CsvDataReader(tr, '\t'))
{
csvData.Settings.HasHeaders = false;
csvData.Settings.SkipEmptyRecords = true;
csvData.Settings.TrimWhitespace = true;
for (int i = 0; i < 30; i++) // known number of columns for testing purposes
{
csvData.Columns.Add("varchar");
}
using (SqlBulkCopy bulkCopy = new SqlBulkCopy(#"Data Source=XPDEVVM\XPDEV;Initial Catalog=MyTest;Integrated Security=SSPI;"))
{
bulkCopy.DestinationTableName = "work.test";
for (int i = 0; i < 30; i++)
{
bulkCopy.ColumnMappings.Add(i, i); // map First to first_name
}
bulkCopy.WriteToServer(csvData);
}
}
}
}
private TextReader GetData()
{
StringBuilder result = new StringBuilder();
int totalDataLines = 0;
using (FileStream fs = new FileStream(pathToFile, FileMode.Open, System.IO.FileAccess.Read, FileShare.ReadWrite))
{
using (StreamReader sr = new StreamReader(fs))
{
string line = string.Empty;
while ((line = sr.ReadLine()) != null)
{
if (line.StartsWith("D\t"))
{
totalDataLines++;
if (totalDataLines < 100000) // Arbitrary method of restricting how much data is read at once.
{
result.AppendLine(line);
}
}
}
}
}
totalNumberOfLinesProcessed += totalDataLines;
return new StringReader(result.ToString());
}
Actually your code is reading all data from file and keep into TextReader(in memory). Then you read data from TextReader to Save server.
If data is so big, data size in TextReader caused out of memory. Please try this way.
1) Read data (each line) from File.
2) Then insert each line to Server.
Out of memory problem will be solved because only each record in memory while processing.
Pseudo code
begin tran
While (data = FilerReader.ReadLine())
{
insert into Table[col0,col1,etc] values (data[0], data[1], etc)
}
end tran
Probably not the answer you're looking for but this is what BULK INSERT was designed for.
I would just add using BufferedFileReader with the readLine method and doing exatcly in the fashion above.
Basically understanding the resposnisbilties here.
BufferedFileReader is the class reading data from file (buffe wise)
There should be a LineReader too.
CSVReader is a util class for reading the data assuming that its in correct format.
SQlBulkCopy you are anywsay using.
Second Option
You can go to the import facility of database directly. If the format of the file is correct and thw hole point of program is this only. that would be faster too.
I think you may have a red herring with the size of the data. Every time I come across this problem, it's not the size of the data but the amount of objects created when looping over the data.
Look in your while loop adding records to the db within the method button3_Click(object sender, EventArgs e):
TextReader tr = GetData();
using (CsvDataReader csvData = new CsvDataReader(tr, '\t'))
Here you declare and instantiate two objects each iteration - meaning for each chunk of file you read you will instantiate 200,000 objects; the garbage collector will not keep up.
Why not declare the objects outside of the while loop?
TextReader tr = null;
CsvDataReader csvData = null;
This way, the gc will stand half a chance. You could prove the difference by benchmarking the while loop, you will no doubt notice a huge performance degradation after you have created just a couple of thousand objects.
pseudo code:
while (!EOF) {
while (chosenRecords.size() < WRITE_BUFFER_LIST_SIZE) {
MyRecord record = chooseOrSkipRecord(file.readln());
if (record != null) {
chosenRecords.add(record)
}
}
insertRecords(chosenRecords) // <== writes data and clears the list
}
WRITE_BUFFER_LIST_SIZE is just a constant that you set... bigger means bigger batches and smaller means smaller batches. A size of 1 is RBAR :).
If your operation is big enough that failing partway through is a realistic possibility, or if failing partway through could cost someone a non-trivial amount of money, you probably want to also write to a second table the total number of records processed so far from the file (including the ones you skipped) as part of the same transaction so that you can pick up where you left off in the event of partial completion.
Instead of reading csv rows one by one and inserting into db one by one I suggest read a chunk and insert it into database. Repeat this process until the entire file has been read.
You can buffer in memory, say 1000 csv rows at a time, then insert them in the database.
int MAX_BUFFERED=1000;
int counter=0;
List<List<String>> bufferedRows= new ...
while (scanner.hasNext()){
List<String> rowEntries= getData(scanner.getLine())
bufferedRows.add(rowEntries);
if (counter==MAX_BUFFERED){
//INSERT INTO DATABASE
//append all contents to a string buffer and create your SQL INSERT statement
bufferedRows.clearAll();//remove data so it could be GCed when GC kicks in
}
}
I'm trying to post some files using OpenRasta. I've gotten as far as getting my handler called, but by all appearances the stream in the entity is empty. Here's my handler:
public OperationResult Post( IEnumerable<IMultipartHttpEntity> entities)
{
var foo = entities.ToList();
foreach (var entity in foo)
{
if (entity.Stream != null && entity.ContentType != null)
{
var memoryStream = new MemoryStream();
entity.Stream.CopyTo(memoryStream);
}
}
return new OperationResult.Created();
}
Each time through the loop memoryStream has a length of 0. What am I doing wrong?
Nothing like posting on StackOverflow to make the answer immediately obvious. Apparently you only get one enumeration of the entities in order to grab the stream. I had added the "foo" variable above to make debugging easier, but it was causing the streaming to fail. As I stored the stream to the database, I had also failed to reset memoryStream to the beginning before writing it. Fixing these two issues got the file to upload correctly.