Fwd: "invalid byte sequence in UTF-8" on user-provided parameter - how to resolve?

Rafa_F · September 6, 2013, 8:30pm

Hi all,

I have a Rails 3.2.12 app running under JRuby 1.7.2.

In working with a particular payment gateway, I have a user who is
entering
a name with a French accented character as her user name (e.g. Labbé).
When
the gateway posts back to my Rails app to update the user’s profile, I’m
getting the following:

An ArgumentError occurred in users#update_profile:

invalid byte sequence in UTF-8
org/jruby/RubyString.java:3074:in `gsub’

I can see the parameters being passed to my app as:

Parameters: {“data_key”=>"(Edited)", “cust_id”=>“316”,
“date_stamp”=>“2013-09-06”, “time_stamp”=>“09:12:08”,
“response_code”=>“001”, “message”=>“Successfully registered CC
details.”,
“res_success”=>“true”, “payment_type”=>“cc”, “cardholder”=>“K Labb\xE9”,
“expiry_date”=>“1312”, “f4l4”=>“4242***4242”, “result”=>“1”,
“avs_response_code”=>"", “cvd_response_code”=>"", “controller”=>“users”,
“action”=>“update_profile”}

In my app, I’m recording the name provided as follows:

user = User.find(user_id)
if user.present?
  #Record the name as it was entered on the Hosted Vault
  user.name_on_card = params[:cardholder]  Page
  user.datakey = params[:data_key] #This is the data key used for

subsequent transactions
…

I already have my app configured for UTF-8 (in environment.rb)

Encoding.default_external = Encoding::UTF_8
Encoding.default_internal = Encoding::UTF_8

How do I properly record this user’s name without triggering the
“invalid
byte sequence” error?

Many thanks!
Tim

For reference, the stack trace:

Backtrace:

org/jruby/RubyString.java:3074:in gsub' org/jruby/RubyString.java:3042:ingsub’

vendor/cache/activerecord-jdbc-adapter-91d319a99c18/lib/arjdbc/mssql/adapter.rb:221:in
`quote_string’

vendor/cache/activerecord-jdbc-adapter-91d319a99c18/lib/arjdbc/mssql/adapter.rb:210:in
quote' arel (3.0.2) lib/arel/visitors/to_sql.rb:416:inquote’
arel (3.0.2) lib/arel/visitors/to_sql.rb:331:in
visit_Arel_Nodes_Assignment' org/jruby/RubyBasicObject.java:1665:insend’
org/jruby/RubyKernel.java:2090:in send' arel (3.0.2) lib/arel/visitors/visitor.rb:19:invisit’
arel (3.0.2) lib/arel/visitors/to_sql.rb:61:in
visit_Arel_Nodes_UpdateStatement' org/jruby/RubyArray.java:2361:inmap’
arel (3.0.2) lib/arel/visitors/to_sql.rb:61:in
visit_Arel_Nodes_UpdateStatement' org/jruby/RubyBasicObject.java:1665:insend’
org/jruby/RubyKernel.java:2090:in send' arel (3.0.2) lib/arel/visitors/visitor.rb:19:invisit’
arel (3.0.2) lib/arel/visitors/visitor.rb:5:in accept' arel (3.0.2) lib/arel/visitors/to_sql.rb:19:inaccept’
activerecord (3.2.13)
lib/active_record/connection_adapters/abstract/database_statements.rb:7:in
to_sql' activerecord (3.2.13) lib/active_record/connection_adapters/abstract/database_statements.rb:96:inupdate’
activerecord (3.2.13)
lib/active_record/connection_adapters/abstract/query_cache.rb:19:in
update' activerecord (3.2.13) lib/active_record/persistence.rb:359:inupdate’
activerecord (3.2.13) lib/active_record/locking/optimistic.rb:68:in
update' activerecord (3.2.13) lib/active_record/attribute_methods/dirty.rb:74:inupdate’
activerecord (3.2.13) lib/active_record/timestamp.rb:71:in update' activerecord (3.2.13) lib/active_record/callbacks.rb:272:inupdate’
activesupport (3.2.13) lib/active_support/callbacks.rb:406:in
_run__1472338488__update__1343517959__callbacks' org/jruby/RubyBasicObject.java:1659:insend’
org/jruby/RubyKernel.java:2086:in send' activesupport (3.2.13) lib/active_support/callbacks.rb:405:in__run_callback’
activesupport (3.2.13) lib/active_support/callbacks.rb:390:in
_run_update_callbacks' org/jruby/RubyBasicObject.java:1659:insend’
org/jruby/RubyKernel.java:2086:in send' activesupport (3.2.13) lib/active_support/callbacks.rb:81:inrun_callbacks’
activerecord (3.2.13) lib/active_record/callbacks.rb:272:in update' vendor/cache/hobo-9dacbc4465df/hobo/lib/hobo/model/permissions.rb:146:inupdate_with_hobo_permission_check’
activerecord (3.2.13) lib/active_record/persistence.rb:348:in
create_or_update' activerecord (3.2.13) lib/active_record/callbacks.rb:264:increate_or_update’
activesupport (3.2.13) lib/active_support/callbacks.rb:439:in
_run__1472338488__save__1343517959__callbacks' org/jruby/RubyBasicObject.java:1659:insend’
org/jruby/RubyKernel.java:2086:in send' activesupport (3.2.13) lib/active_support/callbacks.rb:405:in__run_callback’
activesupport (3.2.13) lib/active_support/callbacks.rb:390:in
_run_save_callbacks' org/jruby/RubyBasicObject.java:1659:insend’
org/jruby/RubyKernel.java:2086:in send' activesupport (3.2.13) lib/active_support/callbacks.rb:81:inrun_callbacks’
activerecord (3.2.13) lib/active_record/callbacks.rb:264:in
create_or_update' activerecord (3.2.13) lib/active_record/persistence.rb:104:insave!’
activerecord (3.2.13) lib/active_record/validations.rb:56:in save!' activerecord (3.2.13) lib/active_record/attribute_methods/dirty.rb:33:insave!’
activerecord (3.2.13) lib/active_record/transactions.rb:264:in save!' activerecord (3.2.13) lib/active_record/transactions.rb:313:inwith_transaction_returning_status’
activerecord (3.2.13)
lib/active_record/connection_adapters/abstract/database_statements.rb:192:in
transaction' torquebox-transactions (2.3.0) lib/torquebox/active_record_adapters.rb:38:intransaction’
activerecord (3.2.13) lib/active_record/transactions.rb:208:in
transaction' activerecord (3.2.13) lib/active_record/transactions.rb:311:inwith_transaction_returning_status’
activerecord (3.2.13) lib/active_record/transactions.rb:264:in save!' app/controllers/users_controller.rb:202:inupdate_profile’
org/jruby/RubyBasicObject.java:1659:in __send__' org/jruby/RubyKernel.java:2086:insend’
actionpack (3.2.13)
lib/action_controller/metal/implicit_render.rb:4:in
send_action' actionpack (3.2.13) lib/abstract_controller/base.rb:167:inprocess_action’
actionpack (3.2.13) lib/action_controller/metal/rendering.rb:10:in
process_action' actionpack (3.2.13) lib/abstract_controller/callbacks.rb:18:inprocess_action’
activesupport (3.2.13) lib/active_support/callbacks.rb:461:in
_run__260237797__process_action__885198886__callbacks' org/jruby/RubyBasicObject.java:1659:insend’
org/jruby/RubyKernel.java:2086:in send' activesupport (3.2.13) lib/active_support/callbacks.rb:405:in__run_callback’
activesupport (3.2.13) lib/active_support/callbacks.rb:390:in
_run_process_action_callbacks' org/jruby/RubyBasicObject.java:1665:insend’
org/jruby/RubyKernel.java:2090:in send' activesupport (3.2.13) lib/active_support/callbacks.rb:81:inrun_callbacks’
actionpack (3.2.13) lib/abstract_controller/callbacks.rb:17:in
process_action' actionpack (3.2.13) lib/action_controller/metal/rescue.rb:29:inprocess_action’
actionpack (3.2.13)
lib/action_controller/metal/instrumentation.rb:30:in
process_action' activesupport (3.2.13) lib/active_support/notifications.rb:123:ininstrument’
activesupport (3.2.13)
lib/active_support/notifications/instrumenter.rb:20:in instrument' activesupport (3.2.13) lib/active_support/notifications/instrumenter.rb:19:ininstrument’
activesupport (3.2.13) lib/active_support/notifications.rb:123:in
instrument' actionpack (3.2.13) lib/action_controller/metal/instrumentation.rb:29:inprocess_action’
actionpack (3.2.13)
lib/action_controller/metal/params_wrapper.rb:207:in
process_action' activerecord (3.2.13) lib/active_record/railties/controller_runtime.rb:18:inprocess_action’
actionpack (3.2.13) lib/abstract_controller/base.rb:121:in process' actionpack (3.2.13) lib/abstract_controller/rendering.rb:45:inprocess’
actionpack (3.2.13) lib/action_controller/metal.rb:203:in dispatch' actionpack (3.2.13) lib/action_controller/metal/rack_delegation.rb:14:indispatch’
actionpack (3.2.13) lib/action_controller/metal.rb:246:in action' org/jruby/RubyProc.java:249:incall’
actionpack (3.2.13) lib/action_dispatch/routing/route_set.rb:73:in
dispatch' actionpack (3.2.13) lib/action_dispatch/routing/route_set.rb:36:incall’
journey (1.0.4) lib/journey/router.rb:68:in call' org/jruby/RubyArray.java:1613:ineach’
journey (1.0.4) lib/journey/router.rb:56:in call' actionpack (3.2.13) lib/action_dispatch/routing/route_set.rb:612:incall’
exception_notification (3.0.1) lib/exception_notifier.rb:41:in call' actionpack (3.2.13) lib/action_dispatch/middleware/best_standards_support.rb:17:incall’
rack (1.4.5) lib/rack/etag.rb:23:in call' rack (1.4.5) lib/rack/conditionalget.rb:35:incall’
actionpack (3.2.13) lib/action_dispatch/middleware/head.rb:14:in
call' actionpack (3.2.13) lib/action_dispatch/middleware/params_parser.rb:21:incall’
actionpack (3.2.13) lib/action_dispatch/middleware/flash.rb:242:in
call' torquebox-web-2.3.0 (java) lib/torquebox/session/servlet_store.rb:31:incall’
actionpack (3.2.13) lib/action_dispatch/middleware/cookies.rb:341:in
call' activerecord (3.2.13) lib/active_record/query_cache.rb:64:incall’
activerecord (3.2.13)
lib/active_record/connection_adapters/abstract/connection_pool.rb:479:in
call' actionpack (3.2.13) lib/action_dispatch/middleware/callbacks.rb:28:incall’
activesupport (3.2.13) lib/active_support/callbacks.rb:408:in _run__ 2063131183__call__1343517959__callbacks' org/jruby/RubyBasicObject.java:1659:insend’
org/jruby/RubyKernel.java:2086:in send' activesupport (3.2.13) lib/active_support/callbacks.rb:405:in__run_callback’
activesupport (3.2.13) lib/active_support/callbacks.rb:390:in
_run_call_callbacks' org/jruby/RubyBasicObject.java:1659:insend’
org/jruby/RubyKernel.java:2086:in send' activesupport (3.2.13) lib/active_support/callbacks.rb:81:inrun_callbacks’
actionpack (3.2.13) lib/action_dispatch/middleware/callbacks.rb:27:in
call' actionpack (3.2.13) lib/action_dispatch/middleware/remote_ip.rb:31:incall’
actionpack (3.2.13)
lib/action_dispatch/middleware/debug_exceptions.rb:16:in call' actionpack (3.2.13) lib/action_dispatch/middleware/show_exceptions.rb:56:incall’
railties (3.2.13) lib/rails/rack/logger.rb:32:in call_app' railties (3.2.13) lib/rails/rack/logger.rb:16:incall’
activesupport (3.2.13) lib/active_support/tagged_logging.rb:22:in
tagged' railties (3.2.13) lib/rails/rack/logger.rb:16:incall’
actionpack (3.2.13) lib/action_dispatch/middleware/request_id.rb:22:in
call' rack (1.4.5) lib/rack/methodoverride.rb:21:incall’
rack (1.4.5) lib/rack/runtime.rb:17:in call' rack (1.4.5) lib/rack/lock.rb:15:incall’
rack-cache (1.2) lib/rack/cache/context.rb:136:in forward' rack-cache (1.2) lib/rack/cache/context.rb:143:inpass’
rack-cache (1.2) lib/rack/cache/context.rb:155:in invalidate' rack-cache (1.2) lib/rack/cache/context.rb:71:incall!’
rack-cache (1.2) lib/rack/cache/context.rb:51:in call' railties (3.2.13) lib/rails/engine.rb:479:incall’
railties (3.2.13) lib/rails/application.rb:223:in call' org/jruby/RubyBasicObject.java:1665:insend’
org/jruby/RubyKernel.java:2090:in send' railties (3.2.13) lib/rails/railtie/configurable.rb:30:inmethod_missing’

Tim_Griffin · September 6, 2013, 9:00pm

Hi Tim,

Looks like that string is ISO-8859-1 encoded, not UTF-8. You might want
to
try GitHub - whitequark/rack-utf8_sanitizer: Rack::UTF8Sanitizer is a Rack middleware which cleans up invalid UTF8 characters in request URI and headers. rather than
dealing
with it in your app.

Rohit

Tim_Griffin · September 6, 2013, 9:13pm

Great, thanks for the nudge, Rohit. I’ll give it a shot and report back
in
case anyone else runs into this.

Thanks for the quick response!
Tim

Tim_Griffin · September 7, 2013, 2:04am

Maybe that can be related…or can help…
https://groups.google.com/forum/#!topic/jruby-users/YEMV9uvynYs

Regards
El 06/09/2013 20:30, “Tim Griffin” [email protected] escribi:

Tim_Griffin · September 7, 2013, 5:15pm

Hi Manuel,

I don’t think this issue is related to the one in the link you posted.

$ irb
jruby-1.7.4 :001 > s = “K Labb\xE9”
=> “K Labb\xE9”
jruby-1.7.4 :002 > s.encode(“UTF-8”, “ISO-8859-1”)
=> “K Labb”

Rohit

Tim_Griffin · September 7, 2013, 8:58pm

Hi Rigth… Its different
El 07/09/2013 17:15, “Rohit N.” [email protected]
escribi:

Tim_Griffin · September 8, 2013, 12:23am

Hi Rohit;

From your test statement, it looks like I might not even have to go as
far
as integrating the rack-utf8-sanitizer (provided I know that the input
is
always arriving in ISO-8859-1), right? Your s.encode statement appears
to
do the trick.

Of course, when I can’t guarantee the encoding, rack-utf8-sanitizer is
the
better solution.

I am a little surprised that with Rails support for UTF-8 and
internationalization that there isn’t a more built-in check/conversion
against non-UTF-8 strings. I really didn’t think that I’d encounter this
problem and have to look for a solution outside of Rails.

Tim