Fine kicking from Simione Kuruvoli put Fiji 12-8 up at half-time before Josua Tuisova crossed after the restart.
Lalakai Foketi went over late on to cut the deficit to 22-15 but Eddie Jones' side could not mount a comeback.
Fiji's first-ever World Cup win over the Wallabies moves them above their opponents into second in Pool C.
The pair are locked together on six points after two matches with Fiji ahead thanks to their head-to-head record.
"I'm emotionally drained at the moment," said Fiji head coach Simon Raiwalui after full-time.
"It was a great contest. It kept us guessing until the end. We pushed the boys to the limits and they've never once complained.
"I think there will be a few boys sore but I'm super proud."
