librpbyteswap is still embedded into libromdata and has exported
functions.
librpcpuid is a standalone static library and will be linked into
anything that needs CPU flags, instead of using __builtin_cpu_supports().
librpcpuid's version is more efficient because it has fewer tests and
it doesn't do string comparisons.
Remove #include "byteorder.h" from a few files, since it isn't actually used:
- librpbase/img/RpPng.cpp
- librpbase/img/RpPngWriter.cpp
- libromdata/disc/xdvdfs_structs.h
- libromdata/Media/hsfs_structs.h
[gtk,librpbase] Remove #include "librpcpu/cpu_dispatch.h" from stdafx.h,
since it's only used by a few files.
[gtk3] CMakeLists.txt: Removed SSSE3 checks. SSSE3 is only used by
GdkImageConv, which is only used by the XFCE (GTK2) UI frontend.
[gtk3] CairoImageConv.hpp: Remove #include "librpcpu/cpu_dispatch.h",
since it isn't actually used here.
[xfce] GdkImageConv_ifunc.cpp, [librpbyteswap] byteswap_ifunc.c:
- #include "config.librpcpuid.h" before checking for HAVE_IFUNC.
This was indirectly included before, but explicitly including it
allows us to skip the other inclusion if IFUNC is not available.
gcc-4.9 doesn't support std::is_trivially_copy_constructible<T> or
std::is_trivially_destructible<T>, so use __has_trivial_copy(T)
and __has_trivial_destructor(T) on old gcc.
It's not a perfect fix, but it gets the code to compile.
(I'm using Debian 8 to test big-endian using qemu-system-ppc.
Surprisingly, it's faster than Wii U Linux, even with all of
the CPU emulation overhead!)
Fixes the following 64-bit MSVC warnings:
unice68_pack.c(243,43): warning C4244: '=': conversion from '__int64' to 'dreg_t', possible loss of data
unice68_pack.c(319,28): warning C4244: '=': conversion from '__int64' to 'dreg_t', possible loss of data
unice68_pack.c(325,36): warning C4244: '=': conversion from '__int64' to 'dreg_t', possible loss of data
unice68_pack.c(456,40): warning C4244: '=': conversion from '__int64' to 'dreg_t', possible loss of data
unice68_pack.c(488,36): warning C4244: '=': conversion from '__int64' to 'dreg_t', possible loss of data
This matches msvc.cmake, which disables thread-safe statics for both
i386 and amd64, even though we aren't really supporting 64-bit versions
of Windows XP or Windows Server 2003.
rp-config suddenly started crashing on XP when trying to decode PNG
images. It turns out that *something* caused zlib-ng to start using
TLS, though the code indicates it may have always used it...
It's entirely possible that this was broken on XP for quite a while,
and I just didn't notice it until now.
TLS usage was added in zlib-ng 2.1.0-beta1. rom-properties updated
from v2.0.7 to v2.1.2 in rom-properties 2.2, so the previous release
will randomly crash on Windows XP.
It's going to have a lot of local customizations, so it isn't really
a vendored library anymore. Among other things, I'm converting it from
header-only to .cpp files.
In order to use "dark mode" in Win10 1809+ in a standard Win32 program,
we have to do all sorts of weird shenanigans.
Currently, dark mode is "enabled" in rp-config, but it only works in
context menus and scroll bars.
Reference: https://github.com/ysc3839/win32-darkmode
rapidjson hasn't had a proper release since August 2016, even though it's
had patches as recently as this past September. This patch is from
December 2017 and reworks GenericMemberIterator to not inherit from
std::iterator.
This fixes several warnings in MSVC when compiling in C++17 mode:
extlib\rapidjson\include\rapidjson/document.h(111,5): warning C4996:
'std::iterator<std::random_access_iterator_tag,internal::MaybeAddConst
<Const,rapidjson::GenericMember<Encoding,Allocator>>::Type,ptrdiff_t,
internal::MaybeAddConst<Const,rapidjson::GenericMember<Encoding,Allocator>>
::Type*,internal::MaybeAddConst<Const,rapidjson::GenericMember<Encoding,Allocator>>::Type&>':
warning STL4015: The std::iterator class template (used as a base class
to provide typedefs) is deprecated in C++17. (The <iterator> header is NOT
deprecated.) The C++ Standard has never required user-defined iterators to
derive from std::iterator. To fix this warning, stop deriving from
std::iterator and start providing publicly accessible typedefs named
iterator_category, value_type, difference_type, pointer, and reference.
Note that value_type is required to be non-const, even for constant
iterators. You can define _SILENCE_CXX17_ITERATOR_BASE_CLASS_DEPRECATION_WARNING
or _SILENCE_ALL_CXX17_DEPRECATION_WARNINGS to suppress this warning.
One of the afl-fuzz tests (000003?) was failing due to the source
range being out of bounds.
Also, add assert() to the checked chk_src_range() and chk_dst_range()
functions.
Otherwise, attempting to dereference a5 will segfault.
This fixes all of the unice68 segfaults found with afl-fuzz so far...
at least in debug builds. Release builds are still faulting for some
reason...
TODO: Continue running afl-fuzz. I suspect I'll need to add something
similar to all uses of chk_src_range() and chk_dst_range().
- Define _DEFAULT_SOURCE in addition to _BSD_SOURCE.
In file included from /usr/include/bits/libc-header-start.h:33,
from /usr/include/stdlib.h:26,
from /home/david/programming/rom-properties/extlib/minizip-ng/mz.h:158,
from /home/david/programming/rom-properties/extlib/minizip-ng/mz_strm_mem.c:19:
/usr/include/features.h:196:3: warning: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [-Wcpp]
196 | # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE"
| ^~~~~~~
- mz_os_posix.c, mz_os_utf8_string_create(): `encoding` is unused.
This ensures any project that pulls in TinyXML2 also defines it.
I don't think this caused any actual problems, since libromdata.so wasn't
exporting TinyXML2 symbols when -DUSE_INTERNAL_XML=ON, but it does silence
a warning:
In file included from src/kde/config/AboutTab.cpp:48:0:
extlib/tinyxml2/tinyxml2.h:85:25: warning: "TINYXML2_NO_GCC_EXPORT" is not defined [-Wundef]
.#elif __GNUC__ >= 4 && !TINYXML2_NO_GCC_EXPORT
^
Windows 2000 support only partially worked in rp-config. The actual
shell extension component didn't work.
Support for ANSI Windows was mostly just compile-tested, and it was
becoming a burden to support something almost no one will use.
Removed most `#ifdef UNICODE` and `#ifdef _UNICODE` branches in
favor of using the Unicode path exclusively.
Removed oldwincompat, which was an attempt to get things working
properly on Windows 9x and 2000 with the MSVC 2010 runtime.
rom-properties doesn't compile with MSVC 2010 anymore, so this
probably wasn't very useful anyways.
Consolidated tstring macros in tcharx.h.
Removed GUID_fns.c since it was only useful in ANSI builds.
Compiled using gcc-13.2.0 and MinGW-w64 11.0.0.
Removed the *.dll.debug files, since I'm not planning on debugging
gettext, and no one seems to care about them anyway.
NOTE: The libgnuintl-8.dll files are more than double the size of
the previous version. This might be due to using a newly-built
Gentoo crossdev MinGW-w64 toolchain on my laptop instead of the
old MinGW-w64 toolchain on Ubuntu 22.04, or it might be because
gettext-0.22 has more wchar_t functionality on Windows now.
TODO: Build for Windows on ARM using LLVM MinGW-w64.
[libmspack-xenia] lzxd.c: Cast pointer differences to int.
[microtar] microtar.c: fwrite() and fread() return size_t.
[libcachecommon] CacheKeys.hpp: Silence C4834 and gcc -Wunused-result
for cacheKeys.c_str().
[libromdata] ELF.cpp: Cast val_dtag[] to uint32_t.
[libromdata] EXE_NE.cpp: Cast ao::uvector<> sizes to uint32_t.
[libromdata/tests] microtar_zstd.c: Cast toCopy to unsigned when passing
it to microtar functions.
[librpfile] RpFile_scsi_win32.cpp: Windows 8.1 SDK's ntddscsi.h doesn't
properly declare _NV_SEP_WRITE_CACHE_TYPE as an enum.
[librptext/tests] TextFuncsTest: Silence warnings about implicit
conversoin from double to unsigned int by only doing integer
calculations. (It's done at compile time, not runtime.)
[librptexture] PowerVR3, ValveVTF3: Explicitly cast some size_t
calculations to unsigned int.
[win32] KeyManagerTab: Explicitly cast the return value from
std::count_if() to int.
CMAKE_DEBUG_POSTFIX wasn't being set, so the PDB filename ended up being
gtestpdb_debug_postfix-NOTFOUND.pdb.
Set CMAKE_DEBUG_POSTFIX="d" and CMAKE_RELEASE_POSTFIX="" when building
with MSVC. For other compilers, both are set to "".
Also disable some warnings that merely clutter up the logs and don't
help at all.
extlib/.clang-tidy: Attempt to disable header file scanning, but it
doesn't seem to work...
zlib-ng v2.1 has significant performance improvements compared to v2.0.
In particular, decompression can be up to 56% faster on amd64 when
using AVX2 instructions.
FIXME: RomHeaderTest infinite-loops on Xubuntu 16.04 32-bit with 1.5 GB
RAM due to memory allocation issues. microtar_zstd.c doesn't ever receive
an "out of memory" error...
Ubuntu 16.04 has CMake 3.5.1:
- DESCRIPTION was added in CMake 3.9.
- HOMEPAGE_URL was added in CMake 3.12.
CMake 3.5.1 incorrectly interprets these as programming languages and
gets very confused when it can't find .cmake files that describe how
to handle those "languages".
This was added in minizip-ng 4.0.0, so no rom-properties releases are
affected by this.
-Wno-shift-negative-value isn't available on gcc-5.4, and since both
flags were combined into a single string, both were tested at the same
time instead of testing each flag individually.
Test the flags individually so we can suppress the -Wempty-body
warnings in unice68 on Xubuntu 16.04.
Reverse-seeking past the output buffer can take a while, and the time
increases exponentially the further we read into the file. The only
reverse seeking done by MicroTAR that causes this problem is seeking
back to the header after reading the file data.
Instead of doing that, cache the file header. This eliminates virtually
all reverse-seeking past the output buffer.
Simple benchmarks with debug builds:
- Linux (-Og): 5487 ms -> 2125 ms
- Windows (/Od): 48199 ms -> 39656 ms
This uses zstd-compressed .tar files to reduce the amount of space in
the repository.
[microtar] Use strncpy() and snprintf() to avoid buffer overflows.
This might reduce the total name length from 100 to 99, which may cause
problems later...
[libromdata/tests] microtar_zstd.c: Minimal zstd wrapper for MicroTAR.
It's not perfect, since MicroTAR occasionally seeks backwards and
zstd doesn't support reversing, but we can seek backwards in the current
input buffer most of the time.
For MegaDrive, it took around 1,080ms for uncompressed .tar and 2,680ms
for compressed .tar. The slight slowdown due to rewinding is worth the
massive space savings.
[libromdata] RomDataFactory.hpp: #include "dll-macros.hpp".
Needed for use by RomHeaderTest.
[librpbase] TextOut: Add OF_JSON_NoPrettyPrint to disable JSON
pretty-printing. This is needed for proper comparisons of the JSON data.
Parsing the JSON with RapidJSON and using operator==() didn't seem to
work, so this is the next best option.
RomData subclasses currently tested:
- Console/MegaDrive (and 32X)
- Console/N64
- Handheld/DMG
Code coverage differences:
Before: (top-level)
- Lines: 8059/51885, 15.5%
- Functions: 573/3631, 15.8%
After: (top-level)
- Lines: 9910/51885, 19.1%
- Functions: 684/3631, 18.8%
minizip-ng:
extlib/minizip-ng/mz_os_posix.c: In function ‘mz_os_utf8_string_create’:
extlib/minizip-ng/mz_os_posix.c:94:63: warning: unused parameter ‘encoding’ [-Wunused-parameter]
94 | uint8_t *mz_os_utf8_string_create(const char *string, int32_t encoding) {
| ~~~~~~~~^~~~~~~~
rapidjson:
src/librpbase/TextOut_json.cpp:31:
extlib/rapidjson/include/rapidjson/document.h:102:19: warning:
‘template<class _Category, class _Tp, class _Distance, class _Pointer, class _Reference>
struct std::iterator’ is deprecated [-Wdeprecated-declarations]
102 | : public std::iterator<std::random_access_iterator_tag
| ^~~~~~~~
In file included from /usr/lib/gcc/x86_64-pc-linux-gnu/12/include/g++-v12/bits/stl_algobase.h:65,
from /usr/lib/gcc/x86_64-pc-linux-gnu/12/include/g++-v12/algorithm:60,
from src/librpbase/stdafx.h:28,
from src/librpbase/TextOut_json.cpp:10:
/usr/lib/gcc/x86_64-pc-linux-gnu/12/include/g++-v12/bits/stl_iterator_base_types.h:127:34: note: declared here
127 | struct _GLIBCXX17_DEPRECATED iterator
| ^~~~~~~~
- Split debug symbols doesn't work properly due to an issue with `strip`:
error: symbols referenced by indirect symbol table entries that can't be stripped in: [library]
- Add "arm64" for M1/M2 Macs.
- zstd: "-fdeclspec" is needed for Clang due to use of __declspec() even
on non-MS compilers or OSes. Note that we have to check for both
"Clang" and "AppleClang", since Apple uses its own customized version
of LLVM/clang.
- msvc.cmake: Make sure "Clang" is quoted when STREQUAL is used.
This fixes the last failing test in ImageDecoderTest:
TCtest_ASTC/ImageDecoderTest.decodeTest/tctest_example_astc_dds_gz_Image,
where GetParam() = tctest/example-astc.dds.gz
We're not using cjk.h right now, but better to ensure that it *is* usable
in case we decide to use it later.
[debian] copyright: Update copyrights for uniwidth.
uniwidth provides a uc_width() function, which is similar to wcwidth().
Our version removes the 'encoding' parameter, which isn't needed because
we're always using UTF-8.
Fixes#353: rpcli: bad table alignment on multibyte characters
Reported by @DankRank.
Was missing "-${tinyxml2_VERSION_MAJOR}".
Without this, "tinyxml2-9.pdb" didn't get installed in the release ZIP
file in the Windows build, and no error occurred due to the use of
"optional" to prevent errors caused by Debug vs. Release.
- Use le32_to_cpu() for the source data.
- Rearrange Pixel32 on BE.
Added non-intrinsic macros using byteorder.h. Can't easily add
intrinsic macros without CMake detection. (Might do that later.)
ImageDecoderTest BE failures before: 15
ImageDecoderTest BE failures after: 11
This broke release builds on AppVeyor.
This fixes a regression from commit 107aa39005.
([zlib-ng] CMakeLists.txt: Hacks to get zlibstatic to work when using the internal zlib on Linux.)
Very minor space optimization. `position_slots` is only referenced once
during lzxd initialization, so this shouldn't cause a performance issue.
Code size differences: (64-bit Gentoo Linux, gcc-12.1.0, release build, no LTO)
text data bss dec hex filename
14091 0 0 14091 370b ./extlib/libmspack-xenia/CMakeFiles/mspack.dir/lzxd.c.o
14053 0 0 14053 36e5 ./extlib/libmspack-xenia/CMakeFiles/mspack.dir/lzxd.c.o
-38 0 0 -38 -26 Difference
I would have expected the difference to be 22, since position_slots has
11 elements, but I guess reducing the size may have reduced some padding
as well.
For most libraries, this merely required adding hidden visibility
flags to the CMakeLists.txt files. zlib-ng was a bit finicky,
and TinyXML2 needed a new macro, TINYXML2_NO_GCC_EXPORT.
TODO: zstd isn't currently used by libromdata.so. If I make use of it
later, either directly or indirectly via minizip, update it to not
export symbols when statically linking to it on Linux.
APNG_dlopen.c: Rework this file so it's always compiled regardless of
USE_INTERNAL_PNG and USE_INTERNAL_PNG_DLL. It exports two symbols,
APNG_ref() and APNG_unref(). When using the statically-linked libpng,
these functions are now no-ops, but they still need to be exported.
TinyXML2 is a C++ library, so it definitely needs the SOVERSION in the
DLL name to prevent ABI mismatches.
[libwin32common] DelayLoadHelper.c: Update for TinyXML2.
TODO: Get the SOVERSIONs instead of hard-coding them where necessary.
This causes issues on Xubuntu 16.04 if a system-wide zstd is installed
because extlib uses the internal version while minizip-ng tries using
the system version, and the system version isn't recent enough for
minizip-ng. (Requires 1.4.0; Xubuntu 16.04 has 1.3.1.)
This makes it easier to read and reduces code size a bit.
Code size differences: (64-bit Gentoo Linux, gcc-11.2.0, release build, no LTO)
text data bss dec hex filename
16889 0 0 16889 41f9 basisu_astc_decomp.cpp.o [before]
16877 0 0 16877 41ed basisu_astc_decomp.cpp.o [after]
-12 0 0 -12 -c Difference
AppVeyor's MinGW-w64 build is showing -Wformat warnings that aren't
showing up locally. Unfortunately, I can't seem to find an equivalent
for `make -k` with CMake, so I'll have to keep pushing individual
fixes until I get it fully built.
Reference: https://fedoraproject.org/wiki/Format-Security-FAQ
-Werror=format-nonliteral is *not* enabled because there are some
legitimate uses of non-literal format strings.
Separated the warning flags into multiple variables.
Updated everything to build with this change.
These arrays don't have values that exceed 255, so there's no point
in wasting all the space.
Code size differences: (64-bit Gentoo Linux, gcc-11.2.0, release build, no LTO)
text data bss dec hex filename
23065 0 0 23065 5a19 basisu_astc_decomp.cpp.o [before]
16889 0 0 16889 41f9 basisu_astc_decomp.cpp.o [after]
-6176 0 0 -6176 -1820 Difference
Also enabled the following flags:
- large address aware
- high entropy VA (64-bit only)
- dynamic base
- nxcompat
For some reason, these weren't enabled when I built them before, which
broke when running ROM Properties from a network share on Windows 10
due to the "Prohibit Dynamic Code" policy. (Probably nxcompat.)
Also, strip the binaries in bin.i386/.
This was originally fixed in commit b243f47341.
([tinyxml2] Fix DLL installation in Windows packaging.)
However, it regressed when TinyXML2 was upgraded to 8.1.0 in
commit 05aed01ddf.
([tinyxml2] Updated to tinyxml2-8.1.0.)
The end result was the 32-bit TinyXML2 being installed in bin/,
and the 64-bit TinyXML2 not being installed at all. This resulted in
crashes, which shouldn't have happened either; this means something
is wrong with DelayLoad, so that will need to be fixed next.
Fixes#313: tinyxml2.dll isn't packaged correctly
Reported by @ccawley2011.
extlib\zstd\compress\zstd_lazy.c(1399): error C4703: potentially uninitialized local pointer variable 'dmsTagRow' used
extlib\zstd\compress\zstd_lazy.c(1407): error C4703: potentially uninitialized local pointer variable 'dmsRow' used
gcc-11.1 complained about uninitialized variables, and googletest has
-Werror specified:
In file included from ../../extlib/googletest/googletest/src/gtest-all.cc:42:
../../extlib/googletest/googletest/src/gtest-death-test.cc: In function ‘bool testing::internal::StackGrowsDown()’:
../../extlib/googletest/googletest/src/gtest-death-test.cc:1301:24: error: ‘dummy’ may be used uninitialized [-Werror=maybe-uninitialized]
1301 | StackLowerThanAddress(&dummy, &result);
| ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~
../../extlib/googletest/googletest/src/gtest-death-test.cc:1290:13: note: by argument 1 of type ‘const void*’ to ‘void testing::internal::StackLowerThanAddress(const void*, bool*)’ declared here
1290 | static void StackLowerThanAddress(const void* ptr, bool* result) {
| ^~~~~~~~~~~~~~~~~~~~~
../../extlib/googletest/googletest/src/gtest-death-test.cc:1299:7: note: ‘dummy’ declared here
1299 | int dummy;
| ^~~~~
cc1plus: all warnings being treated as errors
I tested a release build and it saved a total of 10,752 bytes.
On the other hand, it had a rather huge maintenance overhead, since I had
to ensure that all extlibs had __cdecl set up in the headers properly,
and this had to be redone on every update.
The i386 build of LZ4 on AppVeyor was failing in tests because of missing
stdcall symbols. I decided not to bother adding stdcall support to LZ4
and simply revert stdcall entirely.
I somehow missed these when updating zstd earlier. It only seemed to
break the MinGW-w64 builds, which is odd, since I would have expected
this to break every build in cases where we didn't have a system zstd...
https://github.com/nmoinvaz/minizip
NOTE: iconv support is disabled for now, since it isn't needed by any
of the test cases. It may be re-enabled later if minizip is needed for
the main program.
We're technically installing both the Debug and Release files, but only
one will exist at any given time, so it has to be OPTIONAL.
[libpng] CMakeLists.txt: Minor formatting improvements.
This broke the Mac OS X build:
In file included from /Users/travis/build/GerbilSoft/rom-properties/extlib/lz4/lib/lz4.c:110:
/Users/travis/build/GerbilSoft/rom-properties/extlib/lz4/contrib/cmake_unofficial/../../lib/lz4.h:119:1: error: '__declspec' attributes are not enabled; use '-fdeclspec' or '-fms-extensions' to enable support for __declspec attributes
LZ4LIB_API int LZ4LIB_CALL LZ4_versionNumber (void); /**< library version number; useful to check dll version */
^
/Users/travis/build/GerbilSoft/rom-properties/extlib/lz4/contrib/cmake_unofficial/../../lib/lz4.h💯22: note: expanded from macro 'LZ4LIB_API'
.# define LZ4LIB_API __declspec(dllexport) LZ4LIB_VISIBILITY
^
- TextOut: Added CRLF linebreak options, which are needed for
copying to clipboard on Windows.
[rapidjson] prettywriter.h: Added a CRLF linebreak option.
