The winger was chopped down from behind by former Ayr full-back Willie Lyle as he dashed into the box.
Conway stepped up to wrong foot keeper Ally Brown from the spot and seal the home team's victory.
Raith looked the better side early on with former Ayr veteran Scott Crabbe twice testing keeper Mark McGeown with good efforts.
Raith should have gone ahead in 27 minutes when the tricky Brian Fairbairn drove the ball across the face of goal only for no Rovers strikers to take advantage.
Ayr made them pay in 34 minutes when Chris Robertson rose unchallenged to head home a Conway corner.
However, it took Raith only two minutes to draw level when Fairbairn's free-kick was turned back by Neil Jablonski and swept home on the turn by Eddie Annand.
Conway's second-half penalty put Ayr back in front and this time they didn't relinquish their lead despite Annand going close with a diving header from a Laurie Ellis cross.