How can I detect, that an application was loaded from DOS?
I would like an application to return to DOS, when there is a DOS, but end in infinite loop, if there is no DOS (for example if it was started in emulator by loading XEX directly).
The simple solution is to check BOOT? register ($09). 1 - drive boot succeed, 2 - cas boot succeed, 3 - both.
The more elegant solution is to check hatabs table ($31a) for "D" entry. After bare boot there is only P, C, E, S, K handlers. After booting dos, the device "D" appears.
Sometimes game was booted using some kind of disk loader, or game boots directly from disk (whole-disc game) - I'd prefer then check BOOT? register. It depends what do you want to check.
Correct return from application is the jmp (dosvec) - not rts. You can't worry about dos then. By default OS set dosvec to $e471 (self test jump at system jump table).