I guess the slowdown is related to the part when you assign image to VLC picture. Try timing just FImg := TSingleImage.CreateFromFile(AFileName); part and compare to Irfan. Both Imaging and Irfan use LibTiff library to load TIFFs from files so there should't be that much of a difference.
Regarding the memory, just don't do anything where data size of the image changes - resizing, data format conversions, etc.