"Fóru þeir þá út á skip, en at því hugði Ánn, hvar hann skyldi
þess koma sér, at öllum þætti sér mest mein at."

I think I understand this (though I wouldn't want to bet
money on it) but it's hard to translate.

Fóru þeir þá út á skip en at því hugði Ánn,
Went they then aboard the ship and about that thought Ánn

hvar hann skyldi þess koma sér
where he should in it place himself

at öllum þætti sér mest mein at.
so that each would feel himself most harmed.

1. "Hyggja at" is a phrasal verb meaning something like "consider" or, to
use an English phrasal verb, "think about".

2. The mysterious "þess" must stand for "skips" in its locative sense.
Like in "miðskips". Or English "amidships".

3. The "at" before "öllum" I understand as "svá at".

4. The "sér" could be left out but that would slightly alter the meaning.
The "sér" tells us that it's each man by himself who is inconvenienced.

5.
Mér þykkir mein at einhverju.
I feel harm from something.

Kveðja,
Haukur