Converting Bertini2 to a unity build

this follows templates part3

May 8, 2017

Having failed utterly at using extern template to reduce compile time, I am attempting to use a "unity" build (master build, etc) to improve compile time.

The first step was to manually produce a unity.cpp file which #includes all the .cpp files for the core library. No problem, < 3 mins manual effort on that one.

// this file includes all cpp files for this project, and represents the Unity style build.

#include "src/basics/limbo.cpp"
#include "src/basics/mpfr_complex.cpp"
#include "src/basics/mpfr_extensions.cpp"

#include "src/bertini2/bertini.cpp"

#include "src/function_tree/node.cpp"
#include "src/function_tree/operators/arithmetic.cpp"
#include "src/function_tree/operators/operator.cpp"
#include "src/function_tree/operators/trig.cpp"

#include "src/function_tree/roots/function.cpp"
#include "src/function_tree/roots/jacobian.cpp"

#include "src/function_tree/symbols/differential.cpp"
#include "src/function_tree/symbols/number.cpp"
#include "src/function_tree/symbols/special_number.cpp"
#include "src/function_tree/symbols/symbol.cpp"
#include "src/function_tree/symbols/variable.cpp"

#include "src/parallel/initialize_finalize.cpp"
#include "src/parallel/parallel.cpp"


#include "src/system/precon.cpp"
#include "src/system/slice.cpp"
#include "src/system/start_base.cpp"
#include "src/system/system.cpp"

#include "src/system/start/total_degree.cpp"
#include "src/system/start/mhom.cpp"
#include "src/system/start/user.cpp"

#include "src/tracking/explicit_predictors.cpp"

// this file should be dynamically generated, but that's a step for another day

Then, to make a new Makemodule.am file (name arbitrary) to describe how to build libbertini2.la with the unity source file.

#this is src/unity/Makemodule.am
# mostly copy-pasta from my original Makemodule for bertini's core.

lib_LTLIBRARIES += libbertini2.la

unity_build_source_files = \
    src/unity/unity.cpp
#yep, just that one file

libbertini2_la_SOURCES = \
    $(unity_build_source_files) \
    $(all_headers)
#make explicit the dependence on all header files.  this should be adjusted to be tighter

rootinclude_HEADERS += include/bertini2/bertini.hpp


# depends on a lot of libraries
libbertini2_la_LIBADD = $(BOOST_LDFLAGS) \
    $(BOOST_FILESYSTEM_LIB) $(BOOST_SYSTEM_LIB)  $(BOOST_CHRONO_LIB) $(BOOST_REGEX_LIB) $(BOOST_TIMER_LIB) $(MPI_CXXLDFLAGS) $(BOOST_UNIT_TEST_FRAMEWORK_LIB) $(BOOST_SERIALIZATION_LIB) $(BOOST_LOG_LIB) $(BOOST_LOG_SETUP_LIB) $(BOOST_THREAD_LIB) 


bin_PROGRAMS += bertini2

bertini2_SOURCES = \
    src/bertini2/bertini.cpp \
    include/bertini2/blackbox/main_mode_switch.hpp src/blackbox/main_mode_switch.cpp \
    include/bertini2/blackbox/argc_argv.hpp src/blackbox/argc_argv.cpp \
    include/bertini2/parallel.hpp \
    include/bertini2/parallel/initialize_finalize.hpp src/parallel/initialize_finalize.cpp

bertini2_LDADD = $(BOOST_LDFLAGS) $(BOOST_FILESYSTEM_LIB) $(BOOST_SYSTEM_LIB)  $(BOOST_CHRONO_LIB) $(BOOST_REGEX_LIB) $(BOOST_TIMER_LIB) $(MPI_CXXLDFLAGS) $(BOOST_SERIALIZATION_LIB) $(BOOST_LOG_LIB) $(BOOST_LOG_SETUP_LIB) $(BOOST_THREAD_LIB) libbertini2.la

bertini2_CXXFLAGS = $(BOOST_CPPFLAGS)




Results

Unity has been implemented, let's test it out! I compiled with -O2 in Clang coming with XCode 8.

Unity, single thread: 1m15s. Incremental, single thread: 4m08s.

30% of original time, saving 70%. This is just for the core library and nearly-trivial blackbox program, and not including building the tests (I only made up a unity.cpp file for the core).

Since my little virtual build server farm (Thanks CRC!!!) consists of virtual machines without a lot of CPU's, this may prove beneficial. I do, however, need to make sure that the memory usage doesn't go crazy. Sometimes my .cpp files can take a ton of ram, due to the extensive use of templating...

Conclusions

I am pretty happy with the "unity build" thing. It appears to have lead to a 70% reduction in single-thread compile time for the core.

A last note: multithreaded compilation was still faster... Because I can use 8 threads on my Mac. So, this style of build should remain an option, not a requirement. I will add a --with-unity_build option to the configure.ac file, so a user can choose to use it. A last question: which to make default, unity or incremental?