Avoiding filesorts in MySQL with tagging

I recently started with implementing tagging using the various tagging
plug-ins and gems circulating.

All worked to some degree, but I found that the queries (has_many ->
through) that use any eager loading when coupled with order by clauses
tend to cause a lot of temp table creation and file sorts in MySQL.

Has anyone encountered this type of performance issue? If so, are there
any recommendations?