Don't put anything in the last slot of your Ender Chest. It will get replaced when saving the offhand item. Also, if you put an item in your offhand while the elytra is equipped, it will replace the previous item in storage. This means you will get the more recent item back, and any before will be deleted.