Earlier, Southee bagged his 200th test wicket as Bangladesh crumbled to 100 for five at tea.
Southee had two for 30 with Bangladesh just 35 runs ahead with only five wickets remaining in their second innings at Hagley Oval.
As the sun returned after day three was washed out, a test-best 98 by Henry Nicholls led New Zealand to 354 and a first-innings lead of 65.
Southee struck early with the dismissal of Tamim Iqbal for eight and later removed Shakib Al Hasan for seven to become the fifth New Zealander to take 200 test wickets on a list headed by Richard Hadlee with 431.
In between Southee's wickets, Colin de Grandhomme took the wicket of Bangladesh's first innings dangerman Soumya Sarkar for 36, caught by Jeet Raval diving to the left in the gully.
It brought Shakib to the crease and the senior Bangladeshi showed no interest in playing for time with a four and two off the first two balls he faced.
He slashed again at the third ball which flew to Raval who was unable to pocket the chance and for the 11th time in the match a catch was spilled.
Mahmudullah made 36, his best test score on this tour, before he was bowled by Wagner and Sabbir Rahman did not get off the mark before he was caught behind.
The 18-year-old Nazmul Hossain, on debut, was not out 10 with Nurul Hasan yet to score.
After New Zealand resumed the day at 260 for seven, Nicholls ensured they overhauled Bangladesh's 289 before the innings ended with the unusual dismissal of Wagner for 26.
Although he had grounded his bat behind the crease after completing a run, in the fleeting moment when wicketkeeper Nurul Hasan whipped the bails off, Wagner was mid-stride with neither foot on the ground and had lifted his bat.
Nicholls, who inherited the No 5 slot from former skipper Brendon McCullum, was in need of runs after a string of low scores before the Bangladesh series.
He overtook his previous test best 76 in a near chanceless innings until he hit the nineties when the nerves appeared and, with a maiden test century in sight, he sent an inside edge from offspinner Mehedi Hasan on to his stumps.