January 22, 2020
Sometimes is rails, you have to do an import from a large csv file. The largest csv file I had to read had a million lines…
As you could imagine, I ran into some memory issues on my worker. So here is a some code that will allow you to read a large csv file.
require 'csv'
def csv_each(file)
open(file) do |csv|
count = 0
csv.each_line do |line|
@headers ||= line.strip.split(',').map{|col| col.split(' ').join }
if count != 0
d = build_hash(line, @headers)
yield({
id: d['id'].downcase,
name: d['name'],
}, count)
end
count += 1
end
end
end
end
def build_hash(line, headers)
line.strip.split(',').each_with_index.reduce({}) do |result, (value, index)|
result[headers[index]] = value
result
end
end
csv_each('example.csv') do |item|
# TODO: do what ever you need with item object
end