• 2 Posts
  • 2.55K Comments
Joined 11 months ago
cake
Cake day: August 21st, 2024

help-circle















  • yes, the models are bigger, but Wh/prompt is still the metric to look at. 300W for 3 seconds is the same amount of energy as 14.3kW for 0.021 seconds, roughly. i don’t know how fast a machine like that can spit out a single response because right now i’m assuming they’re time-slicing them to fuck, but at least gpt4o through duck.ai responds in about the same time.
    if it running an 800GB model (which i think is about where gpt4o is) takes the same amount of time to respond as me running an 8GB model (i know the comparison is naive) then it would be about… twice as efficient? 0.25Wh for me compared to 11.9Wh/100 for them. and that’s without knowing how many conversations one of those things can carry on at the same time.

    Edit: also, this is me ignoring for the sake of the discussion that the training is where all the energy use comes from.


  • it takes my 7900XTX about three seconds to generate a longish reply when running at 300w, so that’s 0.24Wh for a single response to a “thank you”. let’s round up so that four “thank yous” costs 1Wh. so he’d have to consistently send almost three million messages a day just containing “thank you”.

    and that’s assuming these huge server farms have the same efficiency per watt as my single GPU.