After a search I've come up with some suggestive solutions to try: http://www.macattorney.com/tutorial.html
for Jaguar or for Panther
http://www.macattorney.com/panther.html

I've seen that error when the download was incomplete. The download got bonked or truncated somehow. Try re-dwnlding. The file may be corrupt at its source.

Got it to mount after downloading it on a PC

While the Finder refuses to mount these files (a reboot is the only solution to the problem I could find), you can still mount these files from the command line using hdiutil, run as root:

$ sudo hdituil attach file name.dmg

Replace file name.dmg with the name of the disk image file. This will properly mount the DMG files, which can then be unmounted normally using the Finder.

It just so happens that we already built an AppleScript solution using this method; I'll paste it below for those who are Terminally challenged. As for the issue with password protected disks, this script will indeed trigger a GUI window asking for any such passwords not already stored in Keychain. I have yet to find an instance where presentation of the password box triggered the failure we're looking to overcome in this hing, but the -stdinpass switch would likely resolve it, and could be added to this script.

set theDMG to POSIX path of (choose file with prompt ¬
"Where is the .DMG file or files you wish to mount?" of type {"devi", "devr"} ¬
with multiple selections allowed)
set {a, b, c} to {"Cancel", "Verify", "Skip Verification"}

set verifyOption to button returned of ¬
(display dialog "Would you like to skip verification? (recommended)" & return & return & ¬
"CAUTION: verification can take a very long time, and you will receive no feedback or opportunity to cancel once selected." buttons {a, b, c} ¬
default button 3 with icon 2)

if verifyOption is b then set verifyOption to " "
if verifyOption is c then set verifyOption to " -noverify "
set mountDMG to "sudo hdiutil attach" & verifyOption & theDMG
try
do shell script mountDMG with administrator privileges
display dialog "Disk Mount of disk image " & (theDMG as text) ¬
& " is complete."
on error errMSG
display dialog errMSG
end try