Sorry for late reply, kind of on the road right now
I will be in Budapest about 2 weeks before I take the train - can I buy a single ticket there do you think? Any idea of price for sleeper of couchette?
Yes, however ticket fare when bought in Hungary will be couple times higher compared to the same Podgorica-Subotica ticket bought in Serbia or Montenegro (should be about 25 euro there). Perhaps makes sense to ask for couchette/sleeper reservation only (9,40 euro for 6 bed couchette, 15,40 euro for 3 bed sleeper or 23,10 euro for double sleeper) and then just purchase ticket part when you already in Montenegro
Can I buy single ticket Podgorica - Novi Sad from the station in Belgrade? Or do I need to wait and buy it in Montenegro (risky, I think)
Sure, in Belgrade (or any other Serbian station) you would easily buy such ticket. Sleeper reservations in Serbian carriages cost 6 euro (6 bed couchette), 15 euro (3 bed sleeper) or 20 euro (double sleeper)
In not then plan C is... well that's something to think about
Well, as was said above, might make sense to ensure purchase of reservation and don't worry about ticket part until you get to SRB/MNT. Or, if those Hungarian carriages are in EPA system, then you don't even need to hunt for it in Hungary, but try buying it elsewhere with access (e.g. Slovakia), no guarantees here, but worth a try