-Wno-shift-negative-value isn't available on gcc-5.4, and since both
flags were combined into a single string, both were tested at the same
time instead of testing each flag individually.
Test the flags individually so we can suppress the -Wempty-body
warnings in unice68 on Xubuntu 16.04.
Reverse-seeking past the output buffer can take a while, and the time
increases exponentially the further we read into the file. The only
reverse seeking done by MicroTAR that causes this problem is seeking
back to the header after reading the file data.
Instead of doing that, cache the file header. This eliminates virtually
all reverse-seeking past the output buffer.
Simple benchmarks with debug builds:
- Linux (-Og): 5487 ms -> 2125 ms
- Windows (/Od): 48199 ms -> 39656 ms
This uses zstd-compressed .tar files to reduce the amount of space in
the repository.
[microtar] Use strncpy() and snprintf() to avoid buffer overflows.
This might reduce the total name length from 100 to 99, which may cause
problems later...
[libromdata/tests] microtar_zstd.c: Minimal zstd wrapper for MicroTAR.
It's not perfect, since MicroTAR occasionally seeks backwards and
zstd doesn't support reversing, but we can seek backwards in the current
input buffer most of the time.
For MegaDrive, it took around 1,080ms for uncompressed .tar and 2,680ms
for compressed .tar. The slight slowdown due to rewinding is worth the
massive space savings.
[libromdata] RomDataFactory.hpp: #include "dll-macros.hpp".
Needed for use by RomHeaderTest.
[librpbase] TextOut: Add OF_JSON_NoPrettyPrint to disable JSON
pretty-printing. This is needed for proper comparisons of the JSON data.
Parsing the JSON with RapidJSON and using operator==() didn't seem to
work, so this is the next best option.
RomData subclasses currently tested:
- Console/MegaDrive (and 32X)
- Console/N64
- Handheld/DMG
Code coverage differences:
Before: (top-level)
- Lines: 8059/51885, 15.5%
- Functions: 573/3631, 15.8%
After: (top-level)
- Lines: 9910/51885, 19.1%
- Functions: 684/3631, 18.8%
minizip-ng:
extlib/minizip-ng/mz_os_posix.c: In function ‘mz_os_utf8_string_create’:
extlib/minizip-ng/mz_os_posix.c:94:63: warning: unused parameter ‘encoding’ [-Wunused-parameter]
94 | uint8_t *mz_os_utf8_string_create(const char *string, int32_t encoding) {
| ~~~~~~~~^~~~~~~~
rapidjson:
src/librpbase/TextOut_json.cpp:31:
extlib/rapidjson/include/rapidjson/document.h:102:19: warning:
‘template<class _Category, class _Tp, class _Distance, class _Pointer, class _Reference>
struct std::iterator’ is deprecated [-Wdeprecated-declarations]
102 | : public std::iterator<std::random_access_iterator_tag
| ^~~~~~~~
In file included from /usr/lib/gcc/x86_64-pc-linux-gnu/12/include/g++-v12/bits/stl_algobase.h:65,
from /usr/lib/gcc/x86_64-pc-linux-gnu/12/include/g++-v12/algorithm:60,
from src/librpbase/stdafx.h:28,
from src/librpbase/TextOut_json.cpp:10:
/usr/lib/gcc/x86_64-pc-linux-gnu/12/include/g++-v12/bits/stl_iterator_base_types.h:127:34: note: declared here
127 | struct _GLIBCXX17_DEPRECATED iterator
| ^~~~~~~~
- Split debug symbols doesn't work properly due to an issue with `strip`:
error: symbols referenced by indirect symbol table entries that can't be stripped in: [library]
- Add "arm64" for M1/M2 Macs.
- zstd: "-fdeclspec" is needed for Clang due to use of __declspec() even
on non-MS compilers or OSes. Note that we have to check for both
"Clang" and "AppleClang", since Apple uses its own customized version
of LLVM/clang.
- msvc.cmake: Make sure "Clang" is quoted when STREQUAL is used.
This fixes the last failing test in ImageDecoderTest:
TCtest_ASTC/ImageDecoderTest.decodeTest/tctest_example_astc_dds_gz_Image,
where GetParam() = tctest/example-astc.dds.gz
We're not using cjk.h right now, but better to ensure that it *is* usable
in case we decide to use it later.
[debian] copyright: Update copyrights for uniwidth.
uniwidth provides a uc_width() function, which is similar to wcwidth().
Our version removes the 'encoding' parameter, which isn't needed because
we're always using UTF-8.
Fixes#353: rpcli: bad table alignment on multibyte characters
Reported by @DankRank.
Was missing "-${tinyxml2_VERSION_MAJOR}".
Without this, "tinyxml2-9.pdb" didn't get installed in the release ZIP
file in the Windows build, and no error occurred due to the use of
"optional" to prevent errors caused by Debug vs. Release.
- Use le32_to_cpu() for the source data.
- Rearrange Pixel32 on BE.
Added non-intrinsic macros using byteorder.h. Can't easily add
intrinsic macros without CMake detection. (Might do that later.)
ImageDecoderTest BE failures before: 15
ImageDecoderTest BE failures after: 11
This broke release builds on AppVeyor.
This fixes a regression from commit 107aa39005.
([zlib-ng] CMakeLists.txt: Hacks to get zlibstatic to work when using the internal zlib on Linux.)
Very minor space optimization. `position_slots` is only referenced once
during lzxd initialization, so this shouldn't cause a performance issue.
Code size differences: (64-bit Gentoo Linux, gcc-12.1.0, release build, no LTO)
text data bss dec hex filename
14091 0 0 14091 370b ./extlib/libmspack-xenia/CMakeFiles/mspack.dir/lzxd.c.o
14053 0 0 14053 36e5 ./extlib/libmspack-xenia/CMakeFiles/mspack.dir/lzxd.c.o
-38 0 0 -38 -26 Difference
I would have expected the difference to be 22, since position_slots has
11 elements, but I guess reducing the size may have reduced some padding
as well.
For most libraries, this merely required adding hidden visibility
flags to the CMakeLists.txt files. zlib-ng was a bit finicky,
and TinyXML2 needed a new macro, TINYXML2_NO_GCC_EXPORT.
TODO: zstd isn't currently used by libromdata.so. If I make use of it
later, either directly or indirectly via minizip, update it to not
export symbols when statically linking to it on Linux.
APNG_dlopen.c: Rework this file so it's always compiled regardless of
USE_INTERNAL_PNG and USE_INTERNAL_PNG_DLL. It exports two symbols,
APNG_ref() and APNG_unref(). When using the statically-linked libpng,
these functions are now no-ops, but they still need to be exported.
TinyXML2 is a C++ library, so it definitely needs the SOVERSION in the
DLL name to prevent ABI mismatches.
[libwin32common] DelayLoadHelper.c: Update for TinyXML2.
TODO: Get the SOVERSIONs instead of hard-coding them where necessary.
This causes issues on Xubuntu 16.04 if a system-wide zstd is installed
because extlib uses the internal version while minizip-ng tries using
the system version, and the system version isn't recent enough for
minizip-ng. (Requires 1.4.0; Xubuntu 16.04 has 1.3.1.)
This makes it easier to read and reduces code size a bit.
Code size differences: (64-bit Gentoo Linux, gcc-11.2.0, release build, no LTO)
text data bss dec hex filename
16889 0 0 16889 41f9 basisu_astc_decomp.cpp.o [before]
16877 0 0 16877 41ed basisu_astc_decomp.cpp.o [after]
-12 0 0 -12 -c Difference
AppVeyor's MinGW-w64 build is showing -Wformat warnings that aren't
showing up locally. Unfortunately, I can't seem to find an equivalent
for `make -k` with CMake, so I'll have to keep pushing individual
fixes until I get it fully built.
Reference: https://fedoraproject.org/wiki/Format-Security-FAQ
-Werror=format-nonliteral is *not* enabled because there are some
legitimate uses of non-literal format strings.
Separated the warning flags into multiple variables.
Updated everything to build with this change.
These arrays don't have values that exceed 255, so there's no point
in wasting all the space.
Code size differences: (64-bit Gentoo Linux, gcc-11.2.0, release build, no LTO)
text data bss dec hex filename
23065 0 0 23065 5a19 basisu_astc_decomp.cpp.o [before]
16889 0 0 16889 41f9 basisu_astc_decomp.cpp.o [after]
-6176 0 0 -6176 -1820 Difference
Also enabled the following flags:
- large address aware
- high entropy VA (64-bit only)
- dynamic base
- nxcompat
For some reason, these weren't enabled when I built them before, which
broke when running ROM Properties from a network share on Windows 10
due to the "Prohibit Dynamic Code" policy. (Probably nxcompat.)
Also, strip the binaries in bin.i386/.
This was originally fixed in commit b243f47341.
([tinyxml2] Fix DLL installation in Windows packaging.)
However, it regressed when TinyXML2 was upgraded to 8.1.0 in
commit 05aed01ddf.
([tinyxml2] Updated to tinyxml2-8.1.0.)
The end result was the 32-bit TinyXML2 being installed in bin/,
and the 64-bit TinyXML2 not being installed at all. This resulted in
crashes, which shouldn't have happened either; this means something
is wrong with DelayLoad, so that will need to be fixed next.
Fixes#313: tinyxml2.dll isn't packaged correctly
Reported by @ccawley2011.
extlib\zstd\compress\zstd_lazy.c(1399): error C4703: potentially uninitialized local pointer variable 'dmsTagRow' used
extlib\zstd\compress\zstd_lazy.c(1407): error C4703: potentially uninitialized local pointer variable 'dmsRow' used
gcc-11.1 complained about uninitialized variables, and googletest has
-Werror specified:
In file included from ../../extlib/googletest/googletest/src/gtest-all.cc:42:
../../extlib/googletest/googletest/src/gtest-death-test.cc: In function ‘bool testing::internal::StackGrowsDown()’:
../../extlib/googletest/googletest/src/gtest-death-test.cc:1301:24: error: ‘dummy’ may be used uninitialized [-Werror=maybe-uninitialized]
1301 | StackLowerThanAddress(&dummy, &result);
| ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~
../../extlib/googletest/googletest/src/gtest-death-test.cc:1290:13: note: by argument 1 of type ‘const void*’ to ‘void testing::internal::StackLowerThanAddress(const void*, bool*)’ declared here
1290 | static void StackLowerThanAddress(const void* ptr, bool* result) {
| ^~~~~~~~~~~~~~~~~~~~~
../../extlib/googletest/googletest/src/gtest-death-test.cc:1299:7: note: ‘dummy’ declared here
1299 | int dummy;
| ^~~~~
cc1plus: all warnings being treated as errors
I tested a release build and it saved a total of 10,752 bytes.
On the other hand, it had a rather huge maintenance overhead, since I had
to ensure that all extlibs had __cdecl set up in the headers properly,
and this had to be redone on every update.
The i386 build of LZ4 on AppVeyor was failing in tests because of missing
stdcall symbols. I decided not to bother adding stdcall support to LZ4
and simply revert stdcall entirely.
I somehow missed these when updating zstd earlier. It only seemed to
break the MinGW-w64 builds, which is odd, since I would have expected
this to break every build in cases where we didn't have a system zstd...
https://github.com/nmoinvaz/minizip
NOTE: iconv support is disabled for now, since it isn't needed by any
of the test cases. It may be re-enabled later if minizip is needed for
the main program.
We're technically installing both the Debug and Release files, but only
one will exist at any given time, so it has to be OPTIONAL.
[libpng] CMakeLists.txt: Minor formatting improvements.
This broke the Mac OS X build:
In file included from /Users/travis/build/GerbilSoft/rom-properties/extlib/lz4/lib/lz4.c:110:
/Users/travis/build/GerbilSoft/rom-properties/extlib/lz4/contrib/cmake_unofficial/../../lib/lz4.h:119:1: error: '__declspec' attributes are not enabled; use '-fdeclspec' or '-fms-extensions' to enable support for __declspec attributes
LZ4LIB_API int LZ4LIB_CALL LZ4_versionNumber (void); /**< library version number; useful to check dll version */
^
/Users/travis/build/GerbilSoft/rom-properties/extlib/lz4/contrib/cmake_unofficial/../../lib/lz4.h💯22: note: expanded from macro 'LZ4LIB_API'
.# define LZ4LIB_API __declspec(dllexport) LZ4LIB_VISIBILITY
^
- TextOut: Added CRLF linebreak options, which are needed for
copying to clipboard on Windows.
[rapidjson] prettywriter.h: Added a CRLF linebreak option.
The JSON output code will be rewritten to use rapidjson, which will allow
us to add more stuff without having to worry if the resulting text has
the correct formatting.
This adds around 20 KB to the compiled binary.
I chose MiniLZO instead of regular LZO because we only need to be able to
decompress LZO1X blocks.
[libromdata] CisoPspReader: Don't call lzo_init() if this JISO isn't
actually using LZO. Otherwise, if the DLL is missing on Windows, the
program will crash.
This should fix the MinGW-w64 build:
[ 16%] Building RC object extlib/lz4/contrib/cmake_unofficial/CMakeFiles/lz4_shared.dir/__/__/visual/VS2017/liblz4-dll/liblz4-dll.rc.obj
C:\projects\rom-properties\extlib\lz4\visual\VS2017\liblz4-dll\liblz4-dll.rc:6:21: fatal error: verrsrc.h: No such file or directory
.#include "verrsrc.h"
^
compilation terminated.
C:\mingw-w64\i686-6.3.0-posix-dwarf-rt_v5-rev1\mingw32\bin\windres.exe: preprocessing failed.
Main changes:
- gettext now has wbindtextdomain() on Windows, so we can remove our
custom Unicode patch.
- The 64-bit version seems to always build DllMain from the relocatable
package, even though we're disabling that...
New patches:
122-Use-LF-as-newline-in-envsubst.patch
- From: https://github.com/Alexpux/MINGW-packages/tree/master/mingw-w64-gettext
- Use LF as newlines instead of CRLF.
1002-dcigettext-c-include-wchar-h.patch:
- #include <wchar.h> for _wgetcwd().
1003-gettextsrc-exports-missing-formatstring-ruby.patch:
- Fix a missing export for formatstring_ruby, which fixes an undefined
__imp_* symbol error.
1004-gettext-runtime-disable-iconv.patch:
- Disable iconv support in gettext runtime so we don't need to bundle
iconv.dll in the final package.
- Was applied previously, but I forgot to include it as a patch.
It turns out we need it because other modules are compiled with /Gz,
even if zstd isn't.
FIXME: Compiling zstd with /Gz fails due to C2373 type redefinition
errors, which doesn't make any sense...
Otherwise, the travis-ci build fails because BUILD_SHARED_LIBS is OFF
here, so CMake tries building it as a static library, and then fails
because the INSTALL() rule lacks an ARCHIVE target.
- TARGET_INCLUDE_DIRECTORIES(): Needs to be PUBLIC, not INTERFACE.
Otherwise, rc.exe can't find zstd.h.
- verrsrc.h: Added VS_FF_DEBUG, which is used by libzstd-dll.rc but
wasn't present in the original version upstream.
- Remove "/Ob0 /Od /RTC1" from debug flags. In particular, "/Od" and
"/RTC1" conflict with each other:
cl : Command line error D8016: '/O2' and '/RTC1' command-line options are incompatible
- Add "/Ob2" to debug CFLAGS.
- Set -DZSTD_DLL_EXPORT when building the DLL. Otherwise, symbols aren't
exported and the import library isn't built.
- Set -DZSTD_DLL_IMPORT when importing the DLL. This isn't strictly
necessary, but it does reduce some overhead.
- Install the PDB files. (Copied from zlib.)
MiniZip 2.10.0 added support for zstd as a compression method, so we
should support it.
Note that a system version of zstd is preferred on Linux systems.
Enabled the MiniZip test tools for manual builds only. Info-ZIP hasn't
been updated in a while, so I'm going to use the MiniZip test tools to
compress ZIP archives using zstd.
Enabled MiniZip compression in order to compress stuff using the minizip
test program.
mz_strm_zstd.c: FIXME: Compression level doesn't work. Hard-coded the
maximum compression level for now.
The variables are CMAKE_PROJECT_VERSION* for the top-level version
PROJECT_VERSION_* is set if a subproject sets a version number.
- CMP0048 is needed in order to use VERSION. This might be implied by
using CMAKE_MINIMUM_REQUIRED(), but we'll explicitly specify it anyway.
- Set policies before calling PROJECT().
- Update the Win32 DO_SPLIT_DEBUG() code to use PROJECT_VERSION_*
variables if set, or CMAKE_PROJECT_VERSION* if not.
This should fix the Win32 build.
This fixes a regression from commit a1a2e0895c.
(Set all projects to require CMake 3.0.)
- Added language specifications.
- Set the top-level project version in the PROJECT() call.
- Set CMP0063 to NEW if it's available. This enables the symbol
visibility presets on *all* target types, not just shared libraries,
module libraries, and executables with ENABLE_EXPORTS set.
Compiled on Ubuntu 20.04 using MinGW-w64 with gcc-9.3.0-7ubuntu1+22~exp1ubuntu4.
Some changes:
- The libraries are significantly smaller than before. Not sure what
really changed...
- The .def files no longer have ordinals.
MinGW-w64 defines FSCTL_GET_REPARSE_POINT in winioctl.h (or a file
included by it), so we end up with a redefinition warning.
C:\projects\rom-properties\extlib\minizip\mz_os_win32.c:23:0: warning: "FSCTL_GET_REPARSE_POINT" redefined
. # define FSCTL_GET_REPARSE_POINT CTL_CODE(FILE_DEVICE_FILE_SYSTEM, 42, METHOD_BUFFERED, FILE_ANY_ACCESS)
In file included from C:\projects\rom-properties\extlib\minizip\mz_os_win32.c:22:0:
C:/mingw-w64/i686-6.3.0-posix-dwarf-rt_v5-rev1/mingw32/i686-w64-mingw32/include/winioctl.h:1280:0: note: this is the location of the previous definition
. #define FSCTL_GET_REPARSE_POINT CTL_CODE(FILE_DEVICE_FILE_SYSTEM,42,METHOD_BUFFERED,FILE_ANY_ACCESS
ffs() is not declared in headers, even though it's a gcc built-in.
extlib/libmspack-xenia/xenia_lzx.c: In function 'bit_scan_forward':
extlib/libmspack-xenia/xenia_lzx.c:39:10: warning: implicit declaration of function 'ffs' [-Wimplicit-function-declaration]
39 | int i = ffs(v);
| ^~~
extlib/libmspack-xenia/xenia_lzx.c:39:14: warning: 'ffs' argument 1 promotes to 'uint32_t' {aka 'unsigned int'} where 'int' is expected in a call to built-in function declared without prototype [-Wbuiltin-declaration-mismatch]
39 | int i = ffs(v);
| ^
<built-in>: note: built-in 'ffs' declared here
https://github.com/nmoinvaz/minizip
This includes a potential buffer overflow fix in mz_os_win32.c when
determining the pathname of kernel32.dll. We don't actually need it,
since kernel32.dll is always loaded, so use GetModuleHandle() instead.
Upstream changes:
[CPU] Use window size for LZX ref_data_size
71780838f0
[CPU] Properly clear LZX window
c213f7d7a3
Added xe::bit_scan_forward, referenced in these commits:
[CPU] Minor LZX code cleanup.
29665ae79c
[CPU] Fix lzx_decompress misuse of xe::bit_count.
e14639c6c0
There's also a fix for the non-Windows implementation (off-by-one).
Fixes PVS-Studio V127 (Med): An overflow of the 32-bit 'window_bits'
variable is possible inside a long cycle which utilizes a memsize-type
loop counter.
We need to install in ${INSTALL_BIN_DIR}, not ${CMAKE_INSTALL_BINDIR}.
Otherwise, it ends up in bin/ instead of i386/ or amd64/.
[libpng] Use ${INSTALL_BIN_DIR} for the Framework destination. This will
be used if I ever get any Mac OS X frontends done (besides rpcli).
This fixes issue #194: package.cmd: tinyxml2.dll isn't copied over correctly.
Main changes:
- Color A and Color B now share an opaque flag.
- Color B: The low bit of alpha is always 1, not 0.
- Tiles are stored in linear order, not Morton order.
To make it easier to support both formats with minimal changes, the
original PVRTC code has been changed to use templated functions.
Simple non-templated wrapper functions are provided for external use.
PowerVR3, KhronosKTX: Handle PVRTC-II. Note that there's no FourCC
defined for DDS, so we're not updating DirectDrawSurface.
TODO:
- The Hard transition flag isn't supported. PVRTexToolCLI didn't use the
hard transition flag when creating the test images.
[libromdata/tests] ImageDecoderTest: Added some PVRTC-II test images.
Based on tctest's example.png. Note that the PVRTC-II test images are
the full 512x512, whereas the PVRTC-I test image was 256x256.
We have our own ETC1 decompression code, which might be somewhat faster
than PowerVR's due to use of lookup tables. (Maybe I should benchmark
this later...)
It seems that the R and B channels are backwards, so we'll need to fix
that next. Other than that, both 2bpp and 4bpp decoding seems to work.
(A lot better than my terrible attempt, at least.)
[cmake] options.cmake: Added an option for PVRTC. The code is licensed
under the MIT license, but we might as well provide an option for it
because it's third-party code instead of my own code.
TODO:
- Fix R/B channel ordering.
- PVRTC-II decoding?
- Add PVRTC decoding to KTX and DDS.
This now has the correct 32-bit DLL and supporting libraries.
The __imp_libintl_version symbol is present again.
This fixes a 32-bit Windows build regression from
commit bc41a4e845.
([gettext.win32] Recompiled gettext-0.20.1.)
Compiled on Ubuntu 19.10 using MinGW-w64 with gcc-9.2.1-9ubuntu1+22~exp1ubuntu2.
The previous build was compiled on Ubuntu 19.04 using MinGW-w64
with gcc-8.3.0.
Some changes:
- The i386 .def file no longer has __imp_libintl_version.
- The .debug files are no longer compressed. I'm not sure how many
Windows tools support compressed debug files.
This should have been added before, but apparently it wasn't added to
the repository due to the .gitignore rule, and I forgot to specify the
--force option. Oh well.
It was only used on Windows, and since we're using GDI+ for JPEG decoding
on Windows now, it's not needed.
If a Mac OS X UI frontend is added at some point, I'll probably use the
system JPEG decoder instead of re-adding libjpeg-turbo.
It's not working properly on AppVeyor, even though we did set
_WIN32_WINNT=0x0600. (Then again, maybe it just needed winioctl.h...)
Removed the _WIN32_WINNT definition, since it's no longer needed.
[librpbase] TextFuncs.hpp:
- Don't #include "TextFuncs_libc.h". Anything that uses strnlen() or
memmem() has to include this header file directly.
- Moved RP_WIS16 from config.librpbase.h to TextFuncs.hpp. This wasn't
actually configured by cmake; it's set on Windows and unsaet on
everything else.
[libromdata] WiiWAD.cpp: #include "librpbase/config.librpbase.h"
for ENABLE_DECRYPTION.
PROJECT_NAME() was removed in commit 9ea341f40c:
[minizip] Updated to v2.8.4 from the MiniZip fork.
...and re-added with `minizip` in commit 9ea341f40c:
[minizip] CMakeLists.txt: Restore target_include_directories() changes.
Since this didn't update the main extlib CMakeLists.txt, this broke the
extlib properties for minizip. In particular, EXCLUDE_FROM_ALL wasn't
being set, so minizip was being built even when it wasn't needed.
Compiled on Ubuntu 19.04 using MinGW-w64 with gcc-8.3.0.
Some patches were dropped:
- 05-always-use-libintl-vsnprintf.mingw.patch: Doesn't apply cleanly
and isn't needed.
- 09-asm-underscore-mingw.patch: Merged upstream.
All other patches were applied, though some manual adjustments were
needed for a few patches, including the ctype/unistd include patch,
06-dont-include-ctype-after-gnulibs-wctype.mingw.patch.
Removed ini_parse_w(), since this should be handled by the caller.
[librpbase] ConfReader.cpp: On Windows, use _wfopen() to open the file
using a Unicode pathname, then use ini_parse_file() to parse the opened
INI file.
This caused the Windows build to fail due to use of functionality
added in Windows 8 (CERT_FIND_HAS_PRIVATE_KEY) that isn't available
due to the WINVER and _WIN32_WINNT settings.
We don't need encryption enabled for rom-properties, anyway.
https://github.com/nmoinvaz/minizip
[libromdata/tests] CMakeLists.txt: MiniZip now uses the target name
'minizip' instead of 'libminizip'.
[extlib] CMakeLists.txt: MiniZip's options are now prefixed with "MZ_".
