Muslić led Cercle Brugge to eighth place in the Belgian First Division in his first season with the team.
Last season, he improved on that as he finished fifth in the regular season before leading them to a fourth-place finish in the season-ending playoffs.
This saw them reach the Europa League qualifying rounds this season, where they beat Kilmarnock in two matches before losing to Norway's Molde and being relegated to the Conference League.
He was sacked at the beginning of last month with Cercle in the relegation positions after losing to bottom club Beerschot.
Muslic is Argyle's fourth permanent manager in 13 months.
Steven Schumacher, who led the Pilgrims to the Premier League title in 2023, left the club in December that year to take over as Stoke City manager.
He was replaced by former England youth coach Ian Foster, but his three-month tenure saw the Pilgrims drop into the relegation zone, and it was director of football Neil Deusnep and Nanskivill who guided the team to safety on the final day of the season.
Former England captain Rooney took over in the summer but has only managed four league wins during his half-season which ended on New Year's Eve with the club sitting last in the Championship and with the worst defensive record in the English Premier League.
