Hello all,
I’m trying to import some csv data (that has latin characters) using
faster csv but I keep getting a MalformedCSVError:
“Unclosed quoted field on line 1.”
“/ruby/lib/ruby/gems/1.8/gems/fastercsv-1.2.3/lib/faster_csv.rb:1592:in
`shift’”
I’ve tested the same code outside of the Rails application an it works
fine.
Can anyone help me?
Here is the code:
csv_options = {
:headers => false,
:return_headers => false,
:skip_blanks => false
}
csv_file_name = File.dirname(FILE) + “/…/…/Files.TXT”
data = “”
FasterCSV.foreach(csv_file_name, csv_options) do |row|
data << row.to_s + “
”
end
And the first two lines in the file:
“20.1”,“Maria José”,“9”
“10.2”,“José Maria”,“10”
I’m using fastercsv 1.2.3 and rails 2.1.2
Note: I’ve tried adding $KCODE = “utf8” but it still does not work.
Thanks.
Best regards,
Migrate
it can be issue with your rails version for current application.
Hi Migrate,
On Thu, 2009-06-18 at 00:51 +0200, Hu Ma wrote:
Hello all,
I’m trying to import some csv data (that has latin characters) using
faster csv but I keep getting a MalformedCSVError:
“Unclosed quoted field on line 1.”
“/ruby/lib/ruby/gems/1.8/gems/fastercsv-1.2.3/lib/faster_csv.rb:1592:in
`shift’”
From the error message, it looks like shift has a backtick as the
opening quote and a single quote as the closing quote.
HTH,
Bill
Dharmdip R. wrote:
it can be issue with your rails version for current application.
Hello Dharmdip,
Do you know any issues between faster csv and rails?
Best regards,
Migrate
Hello Bill,
I’m not sure I understood you. Can you explain further?
The file content has only double quotes.
Best regards,
Migrate
bill walton wrote:
Hi Migrate,
On Thu, 2009-06-18 at 00:51 +0200, Hu Ma wrote:
Hello all,
I’m trying to import some csv data (that has latin characters) using
faster csv but I keep getting a MalformedCSVError:
“Unclosed quoted field on line 1.”
“/ruby/lib/ruby/gems/1.8/gems/fastercsv-1.2.3/lib/faster_csv.rb:1592:in
`shift’”
From the error message, it looks like shift has a backtick as the
opening quote and a single quote as the closing quote.
HTH,
Bill
Yes i had it , my rails version was issue .
FasterCSV was running smooth on my local but i deployed my application
to server FasterCSV was not running so please check server operating
system , and install gem related to OS. If your problem is not solved
feel free to ask !
ba bye…
First off, a quick note on $KCODE - the values it takes are a little
weird. ‘utf8’ is not valid; you’re looking for one of these:
None (n or N)
EUC (e or E)
Shift_JIS (s or S)
UTF-8 (u or U)
(taken from
http://blog.grayproductions.net/articles/bytes_and_characters_in_ruby_18
)
You can also pass an :encoding option (same choice of values) to
FasterCSV.
I’d also make sure that the file is really UTF8, as many CSVs produced
by Excel will instead be in ISO8859-1 (technically, an incompatible MS
variant of that standard [CP1252]). You may have to crunch the data
somewhat to get it to behave, as the MS Office products are also
notorious for stuffing “smart quotes” and other non-standard
characters into files…
–Matt J.
This confirms my original suspicion; your file is almost certainly in
either ISO8859-1 or Windows-1252. The accented characters are being
misread, as Ruby’s default value (or maybe Rails?) is $KCODE = ‘u’. So
the accented letters are interpreted as the first character of a 3-
byte UTF-8 sequence.
You might want to try passing the :encoding => ‘n’ option to FasterCSV
and see what that does; getting Windows-1252 characters to display
correctly on other platforms is left as an exercise… 
–Matt J.
Hello all,
I’ve update to Rails 2.3.2 and the problem remains.
I’ve also created a new application, and added just one controller
and FasterCSV to environment.rb (no other plugins are present in the
application) but still it does not work.
I already said here that the error only happens in rails. I also found
out the following weird behavior: this error only happens if the latin
character is in the last two positions in the string.
Example 1 (Not working):
“1”,“Maria José”,“9”
Example 2 (Not working):
“1”,“Helder Serrão”,“9”
Example 3 (working):
“2”,“José Maria”,“10”
Also I checked that it works if the latin caracters exist only in the
last column:
Example 4 (working):
“1”,“9”,“Maria José”
Note: I’ve removed the KCODE part from the code.
Hello all,
Matt thanks for the advice but I’m using faster csv 1.2.3 and only
version 1.4 has the option :encoding => ‘n’. I cannot upgrade to 1.4
because I’m using ruport and ruport only accepts 1.2.3.
Matthew I’ve done what you suggested and it worked. Thanks!
Best regards,
Migrate
On Jun 22, 3:23 pm, Matt J. [email protected] wrote:
You might want to try passing the :encoding => ‘n’ option to FasterCSV
and see what that does; getting Windows-1252 characters to display
correctly on other platforms is left as an exercise… 
If it is indeed the case that your accented characters are coming out
in ISO8859-1, consider converting the character set in your input
first:
csv_data = Iconv.iconv(‘utf-8’, ‘ISO_8859-1’, file.read).to_s,
Frankly the whole thing seems rather like black magic to me, but the
above seems to solve the problem I was having with Excel files in a
CSV importer that I’ve recently created.
-Matt
On Jun 17, 6:51 pm, Hu Ma [email protected] wrote:
Hello all,
I’m trying to import some csv data (that has latin characters) using
faster csv but I keep getting a MalformedCSVError:
“Unclosed quoted field on line 1.”
“/ruby/lib/ruby/gems/1.8/gems/fastercsv-1.2.3/lib/faster_csv.rb:1592:in
`shift’”
[snip]
I’m using fastercsv 1.2.3 and rails 2.1.2
You may want to try fastercsv 1.5. It was released just a few days
ago.
I updated our project to use trunk back in May because of a similar
problem.
Jonathan
Hello Jonathan,
Thanks for the info but I cannot upgrade to 1.5
because I’m using ruport and ruport only accepts 1.2.3.
Best regards,
Migrate