on building jpeg-xl for wasm
I was planning on working on a redesign of a photo site, and wanted to use JPEG-XL as the preferred image format for storage. The only implementation I know of is the libjxl reference implementation written in C++. Alas, it hasn’t been a good week for JPEG successors, and I would much prefer not to run this code on my server.
While looking at libjxl a short while ago, I did notice that the build system supports compiling to WASM. Seems like this could be the needed solution to my safety concerns. Hook this into my go server with something like wazero, and I’d feel a lot more comfortable connecting this to the internet. I didn’t investigate too thoroughly at the time, but it seemed quite straightforward.
libjxl
It starts off pretty easy. We’ll just follow the building WASM artifacts instructions. I’m building on debian because I know better than to try this with openbsd.
Clone the emscripten sdk, install it, and woah, what’s happening? It immediately downloads some 300MB binary package of clang. Should have connected to wifi instead of tethering to my phone, but it gets the job done.
Start the build and away!
CMake Error at third_party/CMakeLists.txt:35 (message):
Highway library (hwy) not found. Install libhwy-dev or download it to
third_party/highway from https://github.com/google/highway . Highway is
required to build JPEG XL. You can run
/home/tedu/work/libjxl/deps.sh to download this dependency.Okay, I see. This is going to be one of those builds that requires intervention.
highway
The version of debian I was running didn’t have a libhwy-dev package available, so source build it is. Just a quick detour.
Turns out highway takes an extremely long time to build. About four minutes per c++ file on my laptop, of which there are many. It also takes a lot of memory. My first attempt at building failed when one of the compiler processes was OOM killed. Restart with reduced parallelism.
As a side note, cmake/ninja doesn’t immediately stop when a command fails. It lets all the other concurrent commands continue. I guess every build system does this, although I hadn’t given it much thought before. Running make -j8 to build an openbsd kernel does the same, but it’s less noticeable when the seven leftovers finish in two seconds. I think there’s an equilibrium between time saved finishing what can be done vs time spent waiting to restart. Four minutes is an awkward amount of time to wait.
While the scaled back cmake process dragged on, I did some more research, and found there was a deb package, but it’s located in bookworm, instead of bubbles or bananas or whatever I’m running. Would love to meet the debian big brain balliterater someday.
Install that, but it’s no good, because it’s highway version 1.0.3 and libjxl requires only the freshest 1.0.7. I guess it’s sid time. Fortunately, these install without much fuss. Also, they are very small packages. So much compilation for so little output.
But this is all for naught. The libjxl cmake error keeps coming back. I don’t know whether cmake is broken or lying, but no amount of installing highway packages will appease it.
I prefer to minimize the number of bundled native dependencies, because it only makes it harder to keep track of what’s what and where. (And it inevitably causes trouble building for openbsd.) Looking to make progress, I finally relent and run dep.sh. Despite the name, this is not a shell script; it’s a bash script. It will not run using sh. This downloads a bunch of things. And we’re back to compiling libjxl.
compiling
Having sorted out the dependencies, we’re making progress on the main build. Well, actually, not quite yet. Despite installing many other prereqs as instructed, the brotli build (yes, we’re building brotli too) failed because dot could not be found to build its documentation. I mean, okay, it’s trivial for me to run apt install graphviz, but it’s kinda frustrating for the build to fail again in such manner, and why are we even building documentation for dependencies of dependencies that will never be read?
Alright, we sort that, and at long last the c++ code from libjxl itself is entering a compiler. Everything is going about as well as one can expect.
the abyss
In file included from ../lib/jxl/enc_frame.cc:38:
../lib/jxl/enc_aux_out.h:68:3: error: definition of implicit copy assignment operator for 'AuxOut' is deprecated because it has a user-declared copy constructor [-Werror,-Wdeprecated-copy]
68 | AuxOut(const AuxOut&) = default;
| ^
/home/tedu/work/emsdk/upstream/emscripten/cache/sysroot/include/c++/v1/__algorithm/move.h:41:17: note: in implicit copy assignment operator for 'jxl::AuxOut' first required here
41 | *__result = _IterOps<_AlgPolicy>::__iter_move(__first);
| ^
/home/tedu/work/emsdk/upstream/emscripten/cache/sysroot/include/c++/v1/__algorithm/copy_move_common.h:107:19: note: in instantiation of function template specialization 'std::__move_loop<std::_ClassicAlgPolicy>::operator()<std::reverse_iterator<jxl::AuxOut *>, std::reverse_iterator<jxl::AuxOut *>, std::reverse_iterator<jxl::AuxOut *>>' requested here
107 | auto __result = _Algorithm()(std::move(__range.first), std::move(__range.second), std::__unwrap_iter(__out_first));
| ^
/home/tedu/work/emsdk/upstream/emscripten/cache/sysroot/include/c++/v1/__algorithm/copy_move_common.h:152:19: note: in instantiation of function template specialization 'std::__unwrap_and_dispatch<std::__move_loop<std::_ClassicAlgPolicy>, std::reverse_iterator<jxl::AuxOut *>, std::reverse_iterator<jxl::AuxOut *>, std::reverse_iterator<jxl::AuxOut *>, 0>' requested here
152 | return std::__unwrap_and_dispatch<_NaiveAlgorithm>(std::move(__first), std::move(__last), std::move(__out_first));
| ^
/home/tedu/work/emsdk/upstream/emscripten/cache/sysroot/include/c++/v1/__algorithm/move.h:113:15: note: in instantiation of function template specialization 'std::__dispatch_copy_or_move<std::_ClassicAlgPolicy, std::__move_loop<std::_ClassicAlgPolicy>, std::__move_trivial, std::reverse_iterator<jxl::AuxOut *>, std::reverse_iterator<jxl::AuxOut *>, std::reverse_iterator<jxl::AuxOut *>>' requested here
113 | return std::__dispatch_copy_or_move<_AlgPolicy, __move_loop<_AlgPolicy>, __move_trivial>(
| ^
/home/tedu/work/emsdk/upstream/emscripten/cache/sysroot/include/c++/v1/__algorithm/move.h:123:15: note: in instantiation of function template specialization 'std::__move<std::_ClassicAlgPolicy, std::reverse_iterator<jxl::AuxOut *>, std::reverse_iterator<jxl::AuxOut *>, std::reverse_iterator<jxl::AuxOut *>>' requested here
123 | return std::__move<_ClassicAlgPolicy>(std::move(__first), std::move(__last), std::move(__result)).second;
| ^
/home/tedu/work/emsdk/upstream/emscripten/cache/sysroot/include/c++/v1/__memory/uninitialized_algorithms.h:639:17: note: in instantiation of function template specialization 'std::move<std::reverse_iterator<jxl::AuxOut *>, std::reverse_iterator<jxl::AuxOut *>>' requested here
639 | return std::move(__first1, __last1, __first2);
| ^
/home/tedu/work/emsdk/upstream/emscripten/cache/sysroot/include/c++/v1/vector:917:27: note: in instantiation of function template specialization 'std::__uninitialized_allocator_move_if_noexcept<std::allocator<jxl::AuxOut>, std::reverse_iterator<jxl::AuxOut *>, std::reverse_iterator<jxl::AuxOut *>, jxl::AuxOut, void>' requested here
917 | __v.__begin_ = std::__uninitialized_allocator_move_if_noexcept(
| ^
/home/tedu/work/emsdk/upstream/emscripten/cache/sysroot/include/c++/v1/vector:1048:9: note: in instantiation of member function 'std::vector<jxl::AuxOut>::__swap_out_circular_buffer' requested here
1048 | __swap_out_circular_buffer(__v);
| ^
/home/tedu/work/emsdk/upstream/emscripten/cache/sysroot/include/c++/v1/vector:1910:15: note: in instantiation of member function 'std::vector<jxl::AuxOut>::__append' requested here
1910 | this->__append(__sz - __cs);
| ^
../lib/jxl/enc_frame.cc:1246:16: note: in instantiation of member function 'std::vector<jxl::AuxOut>::resize' requested here
1246 | aux_outs.resize(num_threads);
| ^
1 error generated.
em++: error: '/home/tedu/work/emsdk/upstream/bin/clang++ -target wasm32-unknown-emscripten -fignore-exceptions -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -D__EMSCRIPTEN_SHARED_MEMORY__=1 -DEMSCRIPTEN --sysroot=/home/tedu/work/emsdk/upstream/emscripten/cache/sysroot -Xclang -iwithsysroot/include/fakesdl -Xclang -iwithsysroot/include/compat -DFJXL_ENABLE_AVX512=0 -DHWY_DISABLED_TARGETS=(HWY_SSSE3|HWY_AVX3|HWY_AVX3_ZEN4|HWY_SVE|HWY_SVE2|HWY_SVE_256|HWY_SVE2_128|HWY_RVV) -DJPEGXL_MAJOR_VERSION=0 -DJPEGXL_MINOR_VERSION=9 -DJPEGXL_PATCH_VERSION=0 -DJXL_INTERNAL_LIBRARY_BUILD -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted" -I../ -I../third_party/highway -I../third_party/brotli/c/include -I../third_party/skcms -isystem lib/include -pthread -fno-rtti -funwind-tables -Xclang -mrelax-all -Xclang -mconstructor-aliases -fno-omit-frame-pointer -O3 -DNDEBUG -O2 -fPIC -fvisibility=hidden -fvisibility-inlines-hidden -fmacro-prefix-map=/home/tedu/work/libjxl=. -Wno-builtin-macro-redefined -Wall -Werror -fmerge-all-constants -fno-builtin-fwrite -fno-builtin-fread -Wextra -Wc++11-compat -Warray-bounds -Wformat-security -Wimplicit-fallthrough -Wno-register -Wno-unused-function -Wno-unused-parameter -Wnon-virtual-dtor -Woverloaded-virtual -Wvla -Wdeprecated-increment-bool -Wfloat-overflow-conversion -Wfloat-zero-conversion -Wfor-loop-analysis -Wgnu-redeclared-enum -Winfinite-recursion -Wliteral-conversion -Wno-c++98-compat -Wno-unused-command-line-argument -Wprivate-header -Wself-assign -Wstring-conversion -Wtautological-overlap-compare -Wthread-safety-analysis -Wundefined-func-template -Wunreachable-code -Wunused-comparison -fsized-deallocation -fno-exceptions -fmath-errno -fnew-alignment=8 -fno-cxx-exceptions -fno-slp-vectorize -fno-vectorize -disable-free -disable-llvm-verifier -DJPEGXL_ENABLE_SKCMS=1 -DJPEGXL_BUNDLE_SKCMS=1 -DJPEGXL_ENABLE_TRANSCODE_JPEG=1 -DJPEGXL_ENABLE_BOXES=1 -std=c++11 -MD -MT lib/CMakeFiles/jxl_enc-obj.dir/jxl/enc_frame.cc.o -MF lib/CMakeFiles/jxl_enc-obj.dir/jxl/enc_frame.cc.o.d -c -matomics -mbulk-memory ../lib/jxl/enc_frame.cc -o lib/CMakeFiles/jxl_enc-obj.dir/jxl/enc_frame.cc.o' failed (returned 1)
Figuring that surely at some point somebody must have compiled this code successfully, and it’s not some elaborate hoax, I downloaded the previous release of libjxl and tried building that.
Same angry complaints about the missing highway. Okay, run deps.sh again. That’ll fix it.
-- Could NOT find LCMS2 (missing: LCMS2_LIBRARY LCMS2_INCLUDE_DIR) (Required is at least version "2.13")
CMake Error at third_party/CMakeLists.txt:116 (message):Nope. Apparently the old deps.sh doesn’t actually know how to download all the required pieces. Fortunately for me, it wasn’t a total waste of time starting with the head branch, because now I had an lcms directory I could copy into this libjxl. The failed build shall find new life.
After this final minor transplant operation, the build really did proceed smoothly. I now have a WASM file I will be guarding with my life.
recap
Download libjxl. Install as many prereqs as you can manage. Ignore the cmake lies and run deps.sh to download more stuff. Install the prereqs you inevitably missed because they weren’t documented. Run the build as far as you can. Switch to a release tag. Start over until it fails. Copy missing pieces over from head branch. Complete build.
I am astonished the browsers aren’t all over JPEG-XL; this is the kind of build nonsense they absolutely love.
result
Now that I’ve got a WASM output file, it’s just a simple matter of picking a runtime and building a test program to perform some conversions between image formats.
Spoiler alert: that program doesn’t exist yet.
But preview: “/usr/bin/ld: warning: x86_64.o: missing .note.GNU-stack section implies executable stack” makes an appearance in the sequel. We are in this very deep rabbit hole because we wanted a modicum of security, and now we’re going to mark the stack executable like it’s 1999.
Tagged: programming