Ignore header line when parsing CSV file - ruby-on-rails-3

How can the header line of the CSV file be ignored in ruby on rails while doing the CSV parsing!! Any ideas

If you're using ruby 1.8.X and FasterCSV, it has a 'headers' option:
csv = FasterCSV.parse(your_csv_file, {:headers => true}) #or false if you do want to read them
If you're using ruby 1.9.X, the default library is basically FasterCSV, so you can just do the following:
csv = CSV.parse(your_csv_file, {headers: true})

csv = CSV.read("file")
csv.shift # <-- kick out the first line
csv # <-- the results that you want

I have found the solution to above question. Here is the way i have done it in ruby 1.9.X.
csv_contents = CSV.parse(File.read(file))
csv_contents.slice!(0)
csv=""
csv_contents.each do |content|
csv<<CSV.generate_line(content)
end

Easier way I have found is by doing this:
file = CSV.open('./tmp/sample_file.csv', { :headers => true })
# <#CSV io_type:File io_path:"./tmp/sample_file.csv" encoding:UTF-8 lineno:0 col_sep:"," row_sep:"\n" quote_char:"\"" headers:true>
file.each do |row|
puts row
end

Here is the simplest one worked for me. You can read a CSV file and ignore its first line which is the header or field names using headers: true:
CSV.foreach(File.join(File.dirname(__FILE__), filepath), headers: true) do |row|
puts row.inspect
end
You can do what ever you want with row. Don't forget headers: true

To skip the header without the headers option (since that has the side-effect of returning CSV::Row rather than Array) while still processing a line at a time:
File.open(path, 'r') do |io|
io.readline
csv = CSV.new(io, headers: false)
while row = csv.shift do
# process row
end
end

Related

Karate: Multipart file : read works but passing content in value fails

I have a endpoint with a multipart request which takes two files as part of the request parameter.
I tried with the below snippet with read and it worked, but my use case is to take content from a file and pass it to the value parameter.
Can I pass the content as a file?
The working code if my I try to read the file from directory is
Working feature::
Scenario:
* configure headers = {'Content-Type' : 'multipart/form-data', 'Authorization': 'Bearer sgahshshshs'}
Given url "http://filecompare.com/compare"
And multipart file oldfile = { read: './oldfile.json', filename: 'oldfile.json'}
And multipart file newfile = { read: './newfile.json', filename: newfile.json'}
When method post
Then status 200
Not working feature::
Scenario:
Given url "http://download-oldfile/oldfile"
When method get
* def oldfile = response
Given url "http://download-newfile/newfile"
When method get
* def newfile = response
* configure headers = {'Content-Type' : 'multipart/form-data', 'Authorization': 'Bearer sgahshshshs'}
Given url "http://filecompare.com/compare"
And multipart file oldfile = { value: '#(oldfile)', filename: 'oldfile.json'}
And multipart file newfile = { value: '#(newfile)', filename: newfile.json'}
When method post
Then status 200
The contents are printed correctly but api returns error when I use value
Please let me know if I am missing something as part of running with the value keyword in multipart file.
Thank you
Can't think of anything other than try to convert the value to a string:
* string oldfile = response
If still stuck, follow this process: https://github.com/intuit/karate/wiki/How-to-Submit-an-Issue
Had the same issue, resolved without needing to save to file by defining a variable holding the JSON object and using set to manipulate its value.
In your case this would look like:
* def temp = {value: '', filename: 'oldfile.json'}
* set temp.value = oldFile
#[...]
And multipart file oldFile = temp

NiFi: Remove fixed number of header lines from file

I'm processing a file and I'd like to remove (trim) the first X header lines to keep only data, possibly avoiding using regular expressions.
Thanks
You can remove the first X header lines by using ExecuteScript procesor in Nifi.
The following is a example Jython script which I wrote for myself:
import json
import java.io
from org.apache.commons.io import IOUtils
from java.nio.charset import StandardCharsets
from org.apache.nifi.processor.io import StreamCallback
class PyStreamCallback(StreamCallback):
def __init__(self):
pass
def process(self, inputStream, outputStream):
text = IOUtils.readLines(inputStream, StandardCharsets.UTF_8)
for line in text[3:]:
outputStream.write(line + "\n")
flowFile = session.get()
if (flowFile != None):
flowFile = session.write(flowFile,PyStreamCallback())
flowFile = session.putAttribute(flowFile, "filename", flowFile.getAttribute('filename').split('.')[0]+'_translated.json')
session.transfer(flowFile, REL_SUCCESS)
This obviously removes the first 3 lines but you can easily modify it to remove more or less lines.
Hope that helps.

Using find_or_create_by to update table

I have a table that I am trying to update after I have made changes to the url column. Currently, the data is seeded into the databse when I run rake db:seed, but if I make changes to the csv, in this case to the url, I want to update the table to reflect that change. Currently, the table will not update that value.
require 'csv'
datafile = Rails.root + 'db/data.csv'
CSV.foreach(datafile, headers: true) do |row|
Data.find_or_create_by({address: row[0]}) do |hr|
hr.address = row[0]
hr.city = row[1]
hr.state = row[2]
hr.zip = row[3]
hr.name = row[4]
hr.url = row[5]
end
end
CSV.foreach(datafile, headers: true) do |row|
Data.find_or_create_by({url: row[5]}) do |hr|
hr.url = row[5]
end
end
I tried doing the find_or_create_by on just the url (row[5]), but those changes are not being reflected. How can I make this seed file update any changes or new entries in the CSV in the postgreSQL database?
CSV.foreach(datafile, headers: true) do |row|
Data.find_or_create_by({url: row[5]}) do |hr|
hr.url = row[5]
# call the save method, to saved the change
hr.save
end
end

Rails - How to test that ActionMailer sent a specific attachment?

In my ActionMailer::TestCase test, I'm expecting:
#expected.to = BuyadsproMailer.group_to(campaign.agency.users)
#expected.subject = "You submitted #{offer_log.total} worth of offers for #{offer_log.campaign.name} "
#expected.from = "BuyAds Pro <feedback#buyads.com>"
#expected.body = read_fixture('deliver_to_agency')
#expected.content_type = "multipart/mixed;\r\n boundary=\"something\""
#expected.attachments["#{offer_log.aws_key}.pdf"] = {
:mime_type => 'application/pdf',
:content => fake_pdf.body
}
and stub my mailer to get fake_pdf instead of a real PDF normally fetched from S3 so that I'm sure the bodies of the PDFs match.
However, I get this long error telling me that one email was expected but got a slightly different email:
<...Mime-Version: 1.0\r\nContent-Type: multipart/mixed\r\nContent-Transfer-Encoding: 7bit...> expected but was
<...Mime-Version: 1.0\r\nContent-Type: multipart/mixed;\r\n boundary=\"--==_mimepart_50f06fa9c06e1_118dd3fd552035ae03352b\";\r\n charset=UTF-8\r\nContent-Transfer-Encoding: 7bit...>
I'm not matching the charset or part-boundary of the generated email.
How do I define or stub this aspect of my expected emails?
Here's an example that I copied from my rspec test of a specific attachment, hope that it helps (mail can be creating by calling your mailer method or peeking at the deliveries array after calling .deliver):
mail.attachments.should have(1).attachment
attachment = mail.attachments[0]
attachment.should be_a_kind_of(Mail::Part)
attachment.content_type.should be_start_with('application/ics;')
attachment.filename.should == 'event.ics'
I had something similar where I wanted to check an attached csv's content. I needed something like this because it looks like \r got inserted for newlines:
expect(mail.attachments.first.body.encoded.gsub(/\r/, '')).to(
eq(
<<~CSV
"Foo","Bar"
"1","2"
CSV
)
)

Character encoding issue exporting rails data to CSV

I'm exporting data to a CSV file in rails and in some of my fields, I'm getting character encoding issues like this when I open in Excel:
didn’t
I borrowed this code from an example and I'm assuming the encoding is off. Any idea what it should be?
send_data csv_data,
:type => 'text/csv; charset=iso-8859-1; header=present',
:disposition => "attachment; filename=#{filename}.csv"
When Excel opens the CSV file it just assumes an "iso-8859-1" character encoding. I guess it doesn't even know about the encoding information you send along within your HTTP reply. That's why setting this to UTF-8 doesn't work.
So in order to export your CSV file for Excel in Rails you could do this:
send_data Iconv.conv('iso-8859-1//IGNORE', 'utf-8', csv_data),
:type => 'text/csv; charset=iso-8859-1; header=present',
:disposition => "attachment; filename=#{filename}.csv"
This re-encodes your UTF-8 data string (that's the Rails default) to ISO-8859 and sends it. Along goes the information that this reply is actually ISO-8859-1 encoded (which won't make a difference for Excel but is technically correct if you should open it in a browser etc.).
This worked for me, with Chinese characters!excel csv fromat (BOM + UTF8)
def export_csv_excel
....
# Add BOM to make excel using utf8 to open csv file
head = 'EF BB BF'.split(' ').map{|a|a.hex.chr}.join()
csv_str = CSV.generate(csv = head) do |csv|
csv << [ , , , ...]
#invoices.each do |invoice|
csv << [ , , , ...]
end
end
send_data csv_str, filename: "Invoices-#{Time.now.strftime("%y%m%d%H%M%S")}.csv", type: "text/csv"
end
source(Chinese): http://blog.inheart.tw/2013/09/rubyraisl-csv-excel.html
The answers above did not work for me on Mac Excel:
Using iso-8859-1 would require I replace/remove weird characters, which is not a good enough solution for me, and using BOM with UTF8 worked under Windows but not under Mac Excel.
What worked for me is the WINDOWS-1252 encoding as suggested by https://stackoverflow.com/a/20194266/226255
def self.to_csv(options = {})
(CSV.generate(options) do |csv|
csv << self.headers
all.each do |e|
csv << e.values
end
end).encode('WINDOWS-1252', :undef => :replace, :replace => '')
end
module DownloadService
def student_list
File.open("#{file_name}", "w+:UTF-16LE:UTF-8") do |f|
file = CSV.generate({:col_sep => "\t"}) do |c|
c << ['Canción ', 'años', 'etc']
end
f.write "\xEF\xBB\xBF"
f.write(file)
end
end
end