Load Time With Relationships Fields

We have a site we build that has a lot of relationships. And the first time you hit a page with a given entry that has relationships associated with it it takes a considerable time to load. If you return to that entry again after the long load time it is much faster. I have went in to the tools > SQL Manager and narrowed it down to this query that is getting hung up in the process list.

“Copying to tmp table” :

SELECT DISTINCT L0.field_id as L0_field, L0.grid_field_id as L0_grid_field_id, L0.`grid_co

Any help would be appreciated on finding a solution.