Running a 125B model on used parts during the RAMapocalypse
How we got a 125-billion-parameter mixture-of-experts model answering at 55-68 tokens a second on a used RX 7900 XTX, a used i9-13900KF and 96 GB of mismatched second-hand DDR4, using the open-source Strata runtime.
Read post →