It's just been pointed out to me that the oe-ligature is actually a
19th century invention. The symbol used by the First Grammarian
(author of the First Grammatical Treatise) was 'ø' with a mark to
indicate length. This practice is sometimes used in modern grammars
too [ http://lexicon.ff.cuni.cz/png/oi_noreen/b0413.png ].

Among his examples of vowels which differ only by length, the First
Grammarian has: Vel líkuðu Goþrøþe góþ rø´þe "Well did G. like good
oars." In normalised ON spelling, Norse Course style: Vel líkuðu
Goðrøði góð roeði. From this, in case any doubt remains, it seems
pretty clear that the sound described was a simple long vowel and not
a diphthong at this time.

The presumed pronunciation of 'oe' in Classical Latin and 'oi' in
Classical Greek is not really relevant to the question of how an
Icelandic sound, written with a completely different symbol, was
pronounced in the 12th and 13th centuries. Besides, the Greek and
Latin diphthongs had become monophthongs in their respective languages
many hundreds of years before the First Grammatical Treasise was written.