Adding Duke3D - #289
Conversation
Adding Duke3D to supported systems
Some slight ESP32 improvements, moves certain actions to internal ram rather than the much slower PSRAM. Helps a little but still not amazing. Some other cosmetic fixes.
Adds support for additional grp files, tested Plutonium and seems to work fine with the limited testing (first level)
Also resolves some other potential issues later regarding fading
Now correctly saves duke3d.cfg and reads from it when starting.
Music was too slow as it was coupled to the game sound, but now its decoupled and plays back at the correct speed. It doesnt sound as 'full' as the PC version, its a little more 'simple' but its there! Music is now enabled on the ESP32, doesnt seem to tank performance too much, but can be disabled in the main menu if needed, ESP32 can also increase res back to 320x240 but does decrease performance. Amended controls to add crouch as required to progress in the game, added a 'hotkey' as explained in controls.md
|
What the flip?! This is awesome! Didn't try it out yet but keeping an eye on this :-) |
|
I didnt do a huge amount personally except tell AI to do things but happy how its came out. Happy to build an image for whatever device you have, I added a few for the devices I have here: https://github.com/DynaMight1124/retro-go/releases/tag/Quake%26Duke |
|
Which version of retro-go did you use as a base for this port? Btw great port, it works quite well (although I have modified the code a bit for 1.46 and it works wonderfully) Edit: typo |
|
Hi I used the very latest dev branch as the base and I compiled using IDF v5.5. I should probably go down to v5.1 (edit, 5.1 doesnt support the P4, needs 5.2 min) as the main dev is basing future dev on that version, when I first put together a ESP32-P4, the dev was using v4.4.8 which didnt support the P4 so used v5.5 based on the original P4 work. It should compile warning free on the latest dev and v5.5 :) |
|
Oh so then it makes sense why it didn't work out of the box for master branch 1.46, but I have done some modifications (very little) so it works properly on 1.46 now too! The changes are: `const rg_config_t config = { rg_system_init(&config); rg_system_init(sample_rate, &handlers, NULL);` and I just added an app declaration in main.c in launcher It runs on S3 very well too, great job you and AI did! |
|
I'm glad it worked with minimum changes, its a pain compiling at the best of times let alone between different versions! My biggest rule to AI was not to make any adjustments outside of Duke3D, as I wanted it to work in the Retro-Go system as it was. I'll probably keep it based on the latest Dev since that will be the version going forward assuming the developer is happy to accept. The poor ESP32 performance may not satisfy an acceptance, which I would accept as its not great. Either way, I'll keep it on my fork just so its out there. |
Just tweaked the colours so its not the exact same colour as Doom was
|
Tested DynaMight1124:Duke3D on the fri3d-2024 hardware (esp32s3 with 296x240 SPI display and 8MB PSRAM) with v1.3d Full / Retail (26MB, CRC32: BBC9CE44) and it worked beautifully! Hats off to all of you! |
|
Awesome glad it worked for you too. Mostly a testament to how great Retro-Go is, different devices with different screens, control methods etc and they all just work without needing per app adjustment! I did compile plenty of devices a few days ago to check they still compiled. Heres both Duke and Quake, if you wanted to test. |
|
I was thinking - for "true" 3D games like Duke3D and Quake, it becomes increasingly necessary to have an aim mechanism. Duke3D has autoaim for enemies but a big part of the fun are the secrets and needless destruction, for which being able to properly aim is a must. Also, being able to look up and down would help for situational awareness. The keyboard settings have an "aim" button setting (pageup and pagedown) but since we're very low on buttons, those aren't really an option, I guess. So perhaps a kind of "modifier" button that, when pressed, makes the joystick behave like a "mouse" aim would be nice. To implement this, just thinking out loud: Simple way:
Better way:
Curious to hear if others have given this some thought! |
|
Duke3D was especially hard as it has a lot of inputs required! I had to try to cater for most device combinations as much as I could but mostly based around the Odroid setup of 2 action and 4 other buttons, even with that Duke was struggling as theres shoot, jump and crouch, which are pretty key as well as use/interact & weapon cycle, plus you need to bring up the menu and thats pretty much all our buttons used :) Duke3D does also have an inventory system plus things like jetpack and kick. It was a bit of a nightmare trying to get as much in as possible, at least as much that allows you to complete the game. I ended up using something similar to your suggestion of a modifier button (Use/interact), if held for 1 sec+ it allows the dpad to be used as additional buttons, which I used for inventory cycle, inventory use and jetpack. Theres info on the controls here: https://github.com/DynaMight1124/retro-go/blob/Duke3D/duke3d-go/CONTROLS.md Buttons could be modified a little further to allow look up/down, but it certainly couldnt be used during intense gameplay and I'd imagine it could be very clunky with the current setup of holding use for 1sec. Quake was easier as theres a surprisingly low number of buttons needed! I was a little thrown back after Duke, but theres just not a lot of buttons needed as most things can be interacted with by shooting it or walking into it. One thing I didnt think about which may require lookup/down are things that need to be shot to open certain doors, if they actually require you to look up/down to do this, if they're higher up etc? In the ideal world, we've have an analog sticks and a dpad like modern games and essentially use the dpad for quick select/toggles |
|
Sidequest alert! For the Fri3d Camp 2026 badge, we are limited in internal flash storage so the original Duke Nukem 3D Shareware at 11MB was just too big. I couldn't find a smaller one so set out to create it myself, by compressing image textures, audio, data files, analyzing the level maps to figure out which textures to include, etc. Thus, DukeNano3D was born! You can play the complete Duke Nukem 1.3D Shareware with just 4.7MiB of storage space, or gradually compromise more and more until you get all the way down to 1.3MiB for just the first level and limited sound effects. For general purpose compression, it uses retro-go's built-in .zip support, which seems to work for .grp files smaller than ~4MiB. I had to do quite some work on your duke3d-go branch and also a little bit on EDuke32 for this, so hopefully that can get upstreamed at some point. |
|
Thats awesome, I did see your Duke3D updates the other day actually while scanning through recent Retro-Go forks, I wasnt 100% sure at the time about all the compression changes but now makes sense and very cool, thats some awesome savings! Happy for any improvements, assuming it still works with stock grps (shareware and full), some of the commits sounds like there were quite a few performance gains? so maybe even faster? I'm certainly gonna be running this to test :) sounds like its just for shareware? any sidequest sidequests to add Atomic? :) |
I doubt it, I think those performance gains were just to optimize the decompression, not performance gains to your duke3d-go branch, but I could be mistaken. There were a few minor fixes to yours, for example, when using a non-standard GRP file, duke3d-go was giving a warning on the console and asking the user to press Y or N, which of course hung. I just commented that out.
It kind of works for atomic, it's playable, but some of the textures (like the pistol, the crosshair,...) don't show up. I think that's because they have different tile numbers, so they don't get included from the whitelist. If the duke3d_compact_grp.py script would analyze the game scripts (like .CON and .DEF) in a smarter way, that could be made dynamic :-) But I didn't spend time on it because I don't see much use for it... Atomic is around 43MiB so even if we compress it down to 16-20MiB then it's still too big to fit most internal flash storage... unless someone else has a genius out-of-the-box idea to make this work! It can be done by removing all sound and textures, but at some point that also removes some of the fun :-D |
|
Oh yes makes sense, I was initially thinking that the about the compression of the png's and the conversion of audio to WAV could help improve the performance (mostly on the ESP32). The original ESP32 does struggle quite a bit and I think a lot of it is due to the much slower PSRAM and the data its constantly moving around, the S3 flies through this easily. Plus thats kinda where I was going with the Atomic conversion, not so much a full on reduction to try to fit into internal storage but image compression and sound conversion. The tools are there, so I think its good time for me to try it myself :) I'll grab your grp files and updated duke3d codebase and try it on the lowly ESP32 over the next few days or so. |
|
Initial tests indicate it didnt break anything :) but unfortunately didnt show any notable speed improvements on the ESP32. I tried E1L1-6.grp and E1L1-6_tiny.grp, but thats fine, theres probably more holding it back! Its possibly something a skilled coder could tackle, but maybe one of those things that requires WAY too many hours of tiny optimisations in hundreds of places to get an extra 5fps to take it just over the barely playable to mostly playable :) I havent found AI too great that stuff, but maybe thats partly due to my lack to knowledge too, if I directed it to certain things it could do it, rather than me saying "Just make it faster" It would be cool if you're happy to upstream your Duke3D improvements, esp the improved GRP loading and other improvements, I dont think the additions of the PNG/WAV support breaks anything either but its up to you what you want to upstream. No rush. I had been meaning to revisit Duke3D as I have learnt a little more about the Retro-Go ecosystem and since Duke3D was my first AI project, I pretty much just let it do what it wants, so I'd imagine theres a lot of hardcoded stuff like filepaths that probably should be replaced with rg_x calls instead. |
|
Happy that it works! @DynaMight1124 now that the .grp files are small enough to load from internal filesystem, I started testing that and noticed it crashed due to a failing assert(). I used LittleFS2 but looks like it would have the same issue for the default VfsFat. So this fix is quite important, and now it works fine for internal storage. I didn't notice any adverse side-effects, but only tested on ESP32S3, not ESP32 nor ESP32P4, nor VfsFat. As for upstreaming, do you agree I should submit my pull requests to your DynaMight1124/Duke3D branch, we continue testing with that one, until hopefully one day @ducalex merges your branch into his retro-go? |
|
I have tested that fix, well it was already active on the ESP32 anyway so it will be fine, but I have tested on the P4 and works without issues, which I kinda expected! I worked on this port the wrong way, probably, I thought it would make life easier if I used the most powerful device first, so got it working on the P4, hence the larger stack etc then went down to the S3, which I think mostly just worked with minimal/joint changes, after that went to the ESP32, which required a lot of adjustment. I was concerned about breaking what was already working, hence all the ifdefs etc. I tend to start with the slowest device now. So I've kinda gone down a rabbit hole yesterday! what started off with a simple clean up of the serial log and removing all the hardcoded directories and replacing with proper Retro-Go ones, then quickly turned into a larger change by moving a lot of the static BSS to dynamic pointers, which should help make it more flexible and also inline with Retro-Go. I did incorporate your two main changes, non-standard GRP files & multiple GRP files too. I'll merge the simple serial log clean up and hardcoded directory (plus your two changes, and the latest fix) shortly as its small changes and easily mergable but I'm not sure how easily mergable the other changes will be. I havent git compared it yet so may not be as many changes as it feels. Yes for upstreaming, I am happy if you want to send to my branch, then hopefully if its accepted into the main repo, it'll have all the updates. |
Changed ESP32 to 320x200, the lower resolution looked bad and didnt really help performance, also re-enabled music as again, didnt really change performance. Cleaned up the serial log, mostly so the rg_alloc didnt spam it but also some of the art files. Amended the hardcoded directories to proper Retro-Go calls. Added some fixes/improvements from ThomasFarstrike, non-standard GRP files shouldnt crash, GRP files with different names will now load and unified the stack size for all devices as it caused issues with loading from internal flash.
If the .GRP file contains a duke3d.def file with lines like:
tilefromtexture 1234 { file tile1234.png }
...and of course also the actual tile1234.png file itself,
then that PNG file will be used for that tile, instead of
looking for them in the TILESNNN.ART files.
This makes loading slower, but allows for big filesize optimizations,
which opens up the possibility of including an optimized and stripped
down version of DUKE3D.GRP on the internal flash storage.
Store duke3d.cfg under retro-go config and use storage-backed existence checks so setup loads reliably across boots.
Remove trailing blank line introduced during config path debugging cleanup.
Improve config file handling
- dofrontscreens(): drop premature scrsize restore so loading screen stays up when control returns to enterlevel - enterlevel(): move fade-out + vscrn + scrsize restore to after cacheit()/docacheit() so loading screen covers tile work - Add _idle() calls in docacheit() and precachenecessarysounds() work loops so ESP-IDF audio/input tasks stay responsive - loadplayer(): show loading screen for single-player saves too (after audio cleanup to avoid competing for DMA memory)
Interestingly doesnt improve performance on the ESP32... but at least it now works!
duke3d: keep loading screen visible through cacheit/docacheit
Adds a Retro-Go menu in Duke3D's own settings menu, allows changes to: Backlight Volume Sound Driver Screen Scaling Overclock (Overclock doesnt persist so will need to be set each time the game is run) Also made an amendment to how the game sees grp files that is doesnt recognise (via its CRC32 value) prev it would treat them as 1.5 Atomic but now it treats as 1.3 SW.
The path VBE_setPalette -> SDL_SetColors converted Duke3D's 6-bit palette values through an unnecessary 8-bit intermediate, creating an asymmetric precision issue analogous to the QEMU st7789 backlight fix (a159x36/qemu#4): Old: r6/b6 -> 8-bit via (v<<2)|(v>>4) -> >>3 -> r5 (net: v6>>1, floor) g6 -> 8-bit via (v<<2)|(v>>4) -> >>2 -> g6 (net: identity, lossless) For any palette entry with odd source values, R5=B5=floor(v/2)=0 while G6=v remains non-zero — producing a pure-green pixel instead of near-black. This is visible as a greenish cast in dark/shadowed areas. Fix: add SDL_SetPalette565() which converts directly from the 6-bit source, zeroing the green LSB to give it the same effective 5-bit precision as R/B: r5 = v6 >> 1 g6 = (v6 >> 1) << 1 // 5-bit precision in the 6-bit RGB565 green field b5 = v6 >> 1 For neutral input (r=g=b=val): r5/31 == (g6>>1)/31 == b5/31 -> neutral gray. At full brightness (val=63): r5=31, g6=62, b5=31 — green is 1/63 (~1.6%) shy of maximum, which is perceptually indistinguishable. Also removes the 1KB static SDL_Color fmt_swap[256] intermediate buffer and the SDL_SetColors() call, reducing per-palette-update work by ~50%.
…+ ceiling round
The previous fix (zeroing green LSB to match R/B precision) addressed neutral
gray greenishness but missed the root cause of the 'incredibly green' dark areas:
Duke3D's warm wall/floor palette entries have b6=1 (1/63 blue), which anchors
them as warm brown rather than yellow-green. Both the old code and the first fix
discarded that component: b6=1 -> b5 = floor(1/2) = 0, leaving zero blue.
The proper pipeline:
1. Fill-expand 6-bit to 8-bit: v8 = (v6 << 2) | (v6 >> 4)
Maps 0->0 and 63->255 (full scale). For b6=1: b8=4.
2. Ceiling-round to 5-bit (R,B): r5 = ((v8 + 4) >> 3) & 0x1F
For b8=4: b5 = (4+4)>>3 = 1. Blue survives.
3. Ceiling-round to 6-bit (G): g6 = ((v8 + 2) >> 2) & 0x3F
Green keeps full RGB565 precision.
The & mask (instead of min/clamping) is branchless; it only fires for v6=63
where v8=255 and the rounded value would otherwise be 32/64.
Concrete impact on the most common shaded wall colors:
pal[145] (26,20,1) -> was (13,20,0), now (13,20,1) — blue restored
pal[146] (29,22,1) -> was (14,22,0), now (15,22,1) — blue restored
pal[147] (32,25,1) -> was (16,24,0), now (16,25,1) — blue restored
These are the palette entries that dominate mid-shade corridor rendering
and were the direct cause of the yellow-green tint in dark areas.
The original DOS game was designed for CRT monitors with ~2.2 hardware gamma. On a modern LCD, dark linear palette values are crushed much darker than intended, causing the tiny blue components in warm wall/floor colors (b6=1) to disappear into zero even after the fill-expand fix — leaving yellow-green instead of warm brown in shaded areas. Fix: apply a power-curve gamma lift in SDL_SetPalette565, keyed to the game's existing curbrightness value (set by the Options brightness slider). The formula mirrors eDuke32's britable exactly: gamma_lut[v] = round(255 * pow(v/255, a)) where a = 8.0/(curbrightness+8) curbrightness=0 (slider min): a=1.0, identity — no correction curbrightness=4 (default, ud=16): a=0.667, moderate CRT-like shadow lift curbrightness=14 (slider max): a=0.364, strong lift for bright setups The LUT is 256 bytes and is recomputed only when curbrightness changes (i.e. when the user moves the brightness slider) — zero per-frame overhead. Concrete impact at default brightness on the dominant dim-corridor wall color pal[145]=(26,20,1): before: b8=4 -> b5=1 (3%) after: b8=4 -> gamma_lut[4]=15 -> b5=2 (6%) eDuke32 reference: b8=4 -> britable[7]=27 -> 11% The brightness slider now behaves as a proper gamma control, giving the same shadow detail and color balance as eDuke32 at equivalent brightness settings.
The gamma_lut in SDL_SetPalette565 was applying a second power-curve correction on top of the britable[] table (loaded from tables.dat), which already encodes the exact same brightness formula as eDuke32. The two applications compounded to pow(v, ~0.44), over-brightening dark palette entries by ~56% vs eDuke32 at default brightness. This made dark/shaded areas appear washed out and greenish: dim wall colours that should be dark brown were lifted to visible mid-grey, making the slight warm tint in Duke3D's shade ramp far more prominent. Fix: remove the gamma_lut entirely. SDL_SetPalette565 now only does precision-correct fill-expand (6->8 bit) and ceiling-rounded quantise (8->5/6 bit). britable alone matches eDuke32 output to within 1-2 counts per channel at every brightness and shade level.
…pping to 0 The ceiling rounding formula used & 0x1F as a 'branchless cap' but this was incorrect: when input is 63 (max 6-bit), fill-expand gives 255, ceiling round gives (255+4)>>3=32, and 32 & 0x1F = 0, not 31! This caused bright colors with max channel values to lose that channel entirely, making bright reds/yellows appear green (R=0 while G stays). Palette entries 159, 223, 79 were visibly affected. Fix: use proper clamping (min) instead of bitwise masking.
…sh-dark-colors Minor tweak to improve greenish tint in dark areas — precision-correct 6-bit palette to RGB565 conversion
This adds the CHEATS to a similar location as in eDuke32. Extract cheat effect logic into activate_cheat(int k) so cheat effects can be triggered without keyboard input. The old DN-prefix keyboard path still works unchanged. Add a CHEATS entry as the last item in the Game Options menu (case 702), opening a new submenu (case 10100) with: - God Mode, No-Clip, All Stuff, Monsters (ON/OFF toggle) - Warp shortcuts for E1L2 through E1L6 (DNSCOTTY102-106) Cheats are always visible but greyed out when no game is active. Toggle cheats show their current state (ON/OFF) to the right. Monsters cheat cycles between ON and OFF (invisible/no-AI) only.
Add cheats menu under Options -> Game Options
Adds Lookup/down via Hotkey (see controls.md) Now shows an onscreen 'Hotkey Active' message when the hotkey...is well active! Amended the hotkey activation period to 500ms, tested plenty and its plenty of difference between a normal 'click/push' to a 'hold' Inverted Swim & Fly so fly/swim up was the upper button, as before it was back to front.
Small change for the rare instance the button was held for exactly 500ms
Expanded the cheat menu to allow any level from any episode/volume. It will show levels based on the grp used e.g. shareware only shows episode 1, full only 1-3 and atomic has the full 1-4. Also added support for the 3 official expansions (Duke it out DC, Duke Caribbean & Nuclear Winter) essentially you need to ensure you have duke3d.grp (Atomic version) and also one of the expansions (need to be called dukedc.grp, vacation.grp and nwinter.grp) and select one from the RetroGo menu, it will load the expansion grp and also the atomic. Tested on ESP32-S3 which works fine on all expansions, ESP32 runs out of memory when starting the bigger expansions (Duke it out DC seems to work tho!)
Don't annoy the player with warnings about missing sound files.
…g-sound duke3d-go: move missing sound warning from on-screen to console
Only touch PLL when actually overclocking
Large speed improvements, esp to the original ESP32, over a 50% increase. S3 and P4 also get performance boosts.
Had screen corruption when the player died, fixed. Also on the P4 it would not cap properly to 60FPS, sometimes peaking at 125FPS! great for testing but wasted CPU cycles/battery so now caps at 60FPS with roughly 50% CPU usage. ESP32 and S3 will still likely run 100% CPU to try and get to 60FPS.
3 participants
Hi
Just to be upfront, this is entirely AI completed! Its insane on how well it can do things like this. I took your old Duke3D branch for IDF4.x (I got a crazy amount of warnings and compile errors on 5.x) and told AI what I wanted and little by little is all started working.
Tested on:
ESP32 - Now very playable thanks for ChatGPT 5.6! See below for average FPS.
ESP32-S3 - Actually a very surprising increase, gets a good 30+fps and very playable.
ESP32-P4 - Better again, gets a solid 60+fps.
I tried to keep AI close the Retro-Go 'eco-system' by referencing existing apps like prboom and using RG_x as much as possible so its future proof. I kept it away from making any amendments to existing Retro-Go system (it gets out of the control sometimes!)
It'll need a human eye to cast over it, but it runs well for me!
I'll paste some AI generated stuff below:
Core Enhancements (All Devices)
/sd/retro-go/config/).
logs a warning and skips the non-essential sprite, allowing gameplay to continue.
Device-Specific Adjustments
ESP32
ESP32-S3
ESP32-P4
Content Compatibility (.GRP Files)
The port features an auto-adaptive version system that identifies your game files via CRC32 checksums. The following versions are fully compatible:
Note: I've personally tested Atomic v1.5 and Shareware v1.3D and both work fine. Some shareware versions are v0.99, v1.0 and v1.1, which dont work.
Setup: Simply place your duke3d.grp file in /sd/roms/duke3d/. The engine will automatically detect the version.