I found a one-way to Prague for ~180 Euros... Is it close to what it would cost to travel via bus/train?
180 Euro is a big sum for that part of Europe, so it's not even a problem to travel there and back for this money
And pretty much you have quite a few options available. For instance, you can use a rather exotic option - ferry from Patra/Igoumenitsa to Southern Italy(Brindisi or Bari) and then fly from there to Prague (for example, WizzAir has tickets Bari-Prague for exactly in a week date - 9/11- for only 39.99 €, while deck/air type seat on the ferry isn't that expensive comparing to 180 EUR.
But at the same time you can travel in a more standard fashion - train to Thessaloniki (it's not a problem to catch 19 € ticket if purchased in advance), then you can go either via FYR Macedonia-Serbia or Bulgaria-Serbia/Romania. The later apparently is a bit easier, since buses from Thessaloniki to Sofia is much more frequent (cost 25-30 € depending on carrier). Or you can even catch direct Athens-Sofia buses, but that a bit odd to travel all the way bus, when you can use train for half of the route. To Skopje we found currently one bus from Thessaloniki three times per week, but here you can also try getting towards the border (for instance Doirani), crossing it on foot/taxi and then continue by domestic transport to Skopje. While from Skopje or Sofia it's already really easy (and relatively cheap (<50 EUR) as far as you catch discounted Budapest-Prague ticket (cost 19 €)) to travel to Prague via Budapest. In case you are interested - choose a route and we will advise more precise itinerary.
We are planning on coming back through Austria/Slovenia/Croatia/Serbia/Bulgaria. Should I book in advance to save money or does it really matter?
Actually it does matter, at least in case of Prague-Austria and Austria-Slovenia. But again everything depends on exact criteria. For instance, you can buy discounted tickets Prague-Vienna or don't buy anything, but use Einfach-Raus-Ticket etc. So unless you give us exact destinations and number of people traveling, it's difficult to say anything decent
Also, since you plan to travel in Serbia and Bulgaria + transit either/both on your way to Prague, you might consider buying Balkan flexi pass. It's issued for 5,7, 10 or 15 days validity withing a month. Pass allows unlimited travel throughout whole Balkans, thus at the end can be a deal for you.
Author