So I tried importing a DSiWare injected by this, and it doesn't work. I took that same DSiWare and tried decrypting it with the python TADPole. It said it couldn't decrypt. Turns out, the normal key generated by the 3DS injector is just a tiny bit different than the normal key generated by the python injector. For example:
Normal key:
F247CA716B8BB38A484894F5844803B7 <-- TADPole Python
F247CA716B8BB38A484894F5845E83B7 <-- TADPole 3DS
Normal key CMAC:
69081275B05E8FC7911E6AB8B85C10B8 <-- TADPole python
6981275B05E8FC7911E6AB8B85C10B8 <-- TADPole 3DS
I don't know why this happens. I tried switching to a different key scrambler (this one to be exact: https://github.com/luigoalma/3ds_keyscrambler/blob/master/src/UnScrambler.c#L50) in the hopes that it was something wrong with the uint128_t library or something of the sort, but nope.
Well I'll stick with the new algorithm since it reduces the overhead a bit.
A LOT of changes to the project. I completely replaced vectors with C arrays because of the allocation issues.
This has completely fixed the problem I had before and the project compiles.
Note however that it doesn't mean the project actually works. The produced DSiWare export doesn't get accepted by Frogtool, so our work here is not done.
Basically I noticed that the process of re-encrypting a section and placing it back was common and I was doing it manually 3 times for no reason, so I wrote a new function called placeSection which does exactly that. It takes a section, calculates the CMAC of the sha256 of it, places it there, encrypts the section, places it there, and sorts out the all-zero IV
Oh and I rewrote the SHA256 function to be more "proper"
I need to figure out how to integrate this ninty-233 library into the project. I'm thinking of compiling it to .a myself and linking to it at the end, cause automatically compiling it every time looks like it will be a chore (cpp files inside of folders inside of folders)
I'm thwarted by the disabled CMAC algorithm in the 3DS's mbedtls, so now I need to rebuild it with the "#define MBEDTLS_CMAC_C" enabled in the config.h
Either that or maybe the devkitpro guys can help me out, idk