1. Overview

Sweet Build is a build tool that tracks dependencies between files; using their timestamps to determine which are out of date; and then carrying out actions to bring those files up to date.

Sweet Build is similar to GNU Make, Perforce Jam, SCons, Waf, Lake, and other dependency based build tools. Its main difference to these tools is that it allows scripts to make arbitrary passes over the dependency graph to carry out actions. For example a clean action in Sweet Build is implemented by traversing the dependency graph and invoking the clean function for all visited targets while in a more traditional build tool the clean action would be expressed by creating additional phony targets.

Sweet Build handles source trees spanning multiple directories, with multiple variants and compilers. Multiple processor machines are utilized by allowing dependency scanning and execution of external processes to happen in parallel.

Features:

  • Single executable with no external dependencies.

  • Lua scripting language to specify dependency graph and actions.

  • Support for builds spread across multiple directories.

  • Platform independent path and file system operations.

  • Scanning source files for regular expressions to generate dependencies.

  • Filtering the output of external processes using regular expressions.

  • Dependency graph save and load for faster incremental builds.

  • Parallel execution to make use of multiple processors.

  • Variant builds.

Anti-features:

  • Not in widespread use.

1.1. GNU Make vs SCons vs Sweet Build

These are timing tests based on the tests run by Noel Llopis in his "The Quest for the Perfect Build System" blog entries (parts 1, 2, and 3). In the test 50 libraries with 100 classes each are built. Each source file includes 15 headers from its own library and 5 headers from other libraries.

The tests are run on an Intel Core2 Duo P8600 @2.4GHz, 4GB RAM, with a Samsung SSD RBX series 128GB hard drive.

Tool Full Incremental Single

GNU Make

456.3s

4.2s

3.2s

SCons

427.2s

43.1s

8.7s

Sweet Build

135.4s

1.8s

1.6s

Sweet Build performs well on a full rebuild because it passes all files on the command line to a single invocation of the compiler while both GNU Make and SCons pass each file to a separate invocation of the compiler.

2. Installation

Sweet Build consists of a single executable and the build scripts that define how to build C/C++ programs.

The executable has no external dependencies and can be installed by copying it anywhere that your build process can execute it from. The build scripts need to be loaded from a file named build.lua in each project being built. They can be copied anywhere your build script can locate them.

The recommended installation method is to store the executable and build scripts in a directory that is version controlled along with the rest of the source code in your project. From there the executable and the build scripts can be easily located by the build process. This method versions the executable and the build scripts along with the rest of the code in the project. Reverting back to a previous revision of the project also reverts to the build process used to build that revision.

Another installation method is to copy the executable into a directory in the executable path specified by the PATH environment variable and the build scripts into a directory in the Lua search path specified by the LUA_PATH environment variable. In this scenario the executable and the build scripts are available to all projects on the machine. The build.lua file can easily load the build scripts using Lua’s require() function. This method allows the executable and the build scripts to be shared across multiple projects.

Precompiled binaries for MacOSX and Windows and source code for the latest version of Sweet Build can be downloaded from http://www.sweetsoftware.co.nz/.

3. Usage

Usage: build [options] [variable=value] [command] ...
Options:
  -h, --help         Print this message and exit.
  -v, --version      Print the version and exit.
  -f, --file         Set the script file to load.
  -W, --warn         Set the warning level.
  -s, --stack-trace  Enable stack traces in error messages.

Sweet Build is invoked by running the build executable from the command line. When invoked the executable searches up from the current working directory until it finds a file named build.lua. Once found this file is executed to load and initialize the build system.

3.1. Command Line

Information is passed from the command line in the form of assignments and commands. Expressions of the form variable=value are interpreted as assignments. Any other identifiers on the command line are interpreted as commands.

The assignments listed on the command line specify values that are assigned to global variables before build.lua is loaded. The assignments provide a way to pass arguments to the build system. For example the variant to build can be specified by setting the value of the variant variable e.g. "build variant=release".

The commands listed on the command line specify functions that are called after build.lua has been loaded. The commands provide a way to specify which build actions the build system should carry out. For example a clean action can be carried out by passing the clean command e.g. "build clean".

3.1.1. Commands

The default command, used when no command is passed on the command line, is "default". This carries out a build action that traverses the dependency graph and brings outdated files up to date.

Valid commands are:

  • default - builds targets. This is the default.

  • clean - remove generated and intermediate files created during a build.

  • dependencies - print the targets in the dependency graph.

  • namespace - print the namespace of targets in the dependency graph.

  • reconfigure - regenerate the per-machine configuration in local_settings.lua.

  • sln - generate a Visual Studio solution and projects.

  • xcodeproj - generate an Xcode project.

Invocation from the top level directory of a project with an empty command line builds everything in that project for the "debug" variant and the "msvc" or "llvmgcc" platform depending on operating system.

D:\sweet\sweet_build_tool\sweet> build

Invocation with the "clean" command removes files generated during a previous build:

D:\sweet\sweet_build_tool\sweet> build clean

3.1.2. Variables

The variant variable can be set to control the settings used when building by passing "variant=variant" on the command line. The default value, used when no variant is passed on the command line, is "debug". Other accepted values are:

  • debug - Build with debug information and no optimization to produce executables and static libraries suitable for debugging. This is the default.

  • debug_dll - Build with debug information and no optimization to produce execuables and dynamic libraries suitable for debugging.

  • release - Build with optimization and runtime debugging functions to produce executables and static libraries suitable for testing.

  • release_dll - Build with optimization and runtime debugging functions to produce executables and dynamic libraries suitable for testing.

  • shipping - Build with optimization to produce executables and static libraries for shipping.

  • shipping_dll - Build with optimization to produce executables and dynamic libraries for shipping.

Invocation with the "release" variant builds using release options:

D:\sweet\sweet_build_tool\sweet> build variant=release

The platform variable can be set to control the tools used when building by passing "platform=platform" on the command line. The default value, used when no platform is passed on the command line, is either "msvc" or "llvmgcc" depending on whether the build is being run on Microsoft Windows or Apple MacOSX.

  • llvmgcc - build using Apple LLVM-GCC. This is the default when running on MacOSX.

  • msvc - build using Microsoft Visual C++ 9.0 or 10.0. This is the default when running on Microsoft Windows.

  • mingw - build using MinGW.

The goal variable can be set to to specify the target to build by passing "goal=goal" on the command line. The goal is interpreted as a path to the target to build. Relative values are considered relative to the current working directory. The target should always be specified using forward slashes as it is a target path not an operating system path. The default, used when no goal is passed on the command line, is to use the goal that corresponds to the current working directory.

The executables and libraries in the "build_tool" directory can be built (as opposed to building all of the executables and libraries under the "sweet" directory) by specifying the relative path to the "build_tool" directory from the command line using the goal variable:

D:\sweet\sweet_build_tool\sweet> build goal=build_tool

Alternatively the executables and libraries in the "build_tool" directory can be built by changing to the "build_tool" directory before invoking the build:

D:\sweet\sweet_build_tool\sweet> cd build_tool
D:\sweet\sweet_build_tool\sweet\build_tool> build

The version variable can be set to control the value of the preprocessor macro BUILD_VERSION by passing "version=version" on the command line. The default, used when no version is set on the command line, is to use the date, time, variant, and platform of the current build.

The jobs variable can be set to control the maximum number of jobs to allow in parallel by passing "jobs=jobs" on the command line. The default, used when jobs is not set on the command line, is four.

3.2. Buildfiles

Sweet Build is configured using build scripts and buildfiles.

Build scripts are the *.lua files that appear at the root of a project (the build.lua and local_settings.lua files) and in the build/lua/build directory of the Sweet Build distribution. They are Lua scripts that define the actions that the build system carries out and the functions that implement the declarative domain specific language that appears in the buildfiles of a project.

Buildfiles are the *.build files that appear throughout the directory hierarchy of a project. Buildfiles are Lua scripts from which the full syntax of Lua is available - they are different from build scripts only in the logical sense that they contain the Lua script that uses the domain specific language that has been defined by the build scripts.

The difference between build scripts and buildfiles is that build scripts define the actions carried out by the build system and buildfiles define the dependency graph that represents the files and dependency relationships of a specific project. The difference is a purely logical one. It is possible to define actions from within buildfiles and file and dependency relationships from within build scripts.

The build.lua file located at the root of a project has the special distinction of being the build script that is executed to load and initialize the build system. The location of the build.lua file also implicitly defines the root directory of the project (available to build scripts and buildfiles through the root() API function).

The build.lua file is expected to set the package path so that it contains the directories in which the build scripts reside; require the build scripts necessary to carry out the build; define a function named initialize() to initialize the build system on every run; and define a function named buildfiles() to load the dependency graph when it needs to be reloaded.

3.2.1. Syntax

The language used in both build scripts and buildfiles is Lua. The language is described in detail in the online manual at http://www.lua.org/manual/5.1/manual.html/.

For readers in a rush Lua is similar to other programming languages. It is procedural and contains expressions (+, -, /, etc) and statements (if, for, while, etc) with semantics that should be familiar to those with knowledge of another programming language.

Some differences between Lua and other programming languages that show up in build scripts and buildfiles are:

  • The ".." operator is used to concatenate strings.

  • The "%" operator is used to interpolate strings with a printf-style format string and arguments as described by that string. This is defined in the build system (therefore it isn’t available until after the build scripts have been loaded).

  • Strings can be delimited with double square brackets ([[ and ]]) as well as the usual single and double quotes. Strings delimited with double square brackets don’t do normal escaping while single and double quoted strings do.

  • The main data type in Lua is the table - an associative array constructed using curly braces and an optional list of fields. Tables can be treated as arrays, a maps, or objects interchangeably. Be aware that array indices are numbered from 1 and not 0 like you might expect. Don’t worry! It works well in practice.

  • Functions are first class values and so show up as fields within tables, local variables, parameters, and return values.

  • Lexical scoping is used to determine the visibility of variables - local variables are available from within functions defined within the same scope even after that scope has been released from the stack.

  • An identifier followed by a string literal or an identifier followed by a table constructor is syntactic sugar for a function call taking a string or table as its only argument.

  • An identifier followed by colon ":" followed by another identifier is syntactic sugar for a function call on a table. The first identifier refers to the table and second identifier to a function stored in that table. The table is passed as the implicit first parameter to the function. The ":" syntax can also be used when the function is defined in which case it implies a hidden first parameter named "self".

  • Lua is a prototype based language. A table can set another table to act as its prototype and have lookups for fields that aren’t defined automatically redirected to be looked up in the prototype table. The table inherits the fields that it doesn’t define itself from its prototype.

  • Settings inheritance in the build system is implemented using prototypes. Per target settings are inherited from their parent target’s settings or the global settings. The global settings are inherited the default settings specified by the build system and the project being built.

  • The behaviour of targets in the build system is also defined using prototypes. In this case each target has its own target prototype and the target prototype defines the functions that specify which actions are taken for that type of target in a traversal of the dependency graph.

3.2.2. Hello World!

The following build.lua builds the classic "Hello World!" program. It creates targets to compile its single source file and to link the resulting object file into the final executable:

package.path = root("../../../build/lua/?.lua")..";"..root("../../../build/lua/?/init.lua");
require "build";
require "build/msvc";

function initialize()
    local settings = build.initialize {};
    msvc.initialize( settings );
end

function buildfiles()
    Executable {
        id = "hello_world";
        Source {
            "hello_world.cpp"
        };
    }
end

The first three lines set the search path for packages loaded by the require() function so that build scripts within the "build/lua/build" directory (relative to the project root) are available. The require() calls load the build scripts that implement the core of the build system and support the Microsoft Visual C++ compiler.

The initialize() function initializes the build system. The build.initialize() function takes a table of project specific settings that override the default settings. The msvc.initialize() function sets up the build system to use the Microsoft Visual C++ compiler.

The buildfiles() function defines the files and dependency relationships specific to building the "Hello World!" example. The Executable identifier is a short form function call taking a table as its sole argument. That table defines the name of the executable and the source that is compiled and linked to build that executable (via the Source identifier which is again a short form function call taking a table).

3.2.3. More Hello World!

In an even more contrived example the following "build.lua" builds the classic "Hello World!" example using a separate library. It creates targets to compile the source files in the library and the executable, archive the library, and link the executable. The executable also specifies a dependency on the library which results in the library being linked in with the executable’s object files:

package.path = root("../../../build/lua/?.lua")..";"..root("../../../build/lua/?/init.lua");
require "build";
require "build/msvc";

function initialize()
  local settings = build.initialize {};
  msvc.initialize( settings );
end

function buildfiles()
  Executable {
      id = "executable";
      libraries = {
          "library"
      };
      Source {
          "executable.cpp"
      };
  };

  Library {
      id = "library";
      Source {
          "library.cpp"
      };
  };
end

The first part of this build.lua file is the same as the previous example. It loads the required build scripts and provides an initialize() function that initializes the build system.

The buildfiles() function in this case specifies two targets to build. An executable and a library. The executable depends on the library through the identifier of the library being listed in the libraries table of the executable. The values listed in the libraries table are build system paths to libraries relative to the root of the project. Specifying this dependency relationship causes the library to be built before the executable, links the library when linking the executable, and makes sure that the executable is linked if the library is changed.

3.2.4. Real World Example

A more typical build.lua file loads modules to provide support for several compilers, provides default values for the variables that are set from the command line, overrides some of the default settings provided by the build system, and loads its dependency graph from buildfiles spread throughout the directory hierarchy of the project.

The following build.lua file is used to build the Sweet Lua project. It loads several buildfiles to define its dependency graph:

package.path = root("build/lua/?.lua")..";"..root("build/lua/?/init.lua");
require "build";
require "build/llvmgcc";
require "build/mingw";
require "build/msvc";
require "build/visual_studio";
require "build/xcode";

function initialize()
    platform = platform or build.switch { operating_system(); windows = "msvc"; macosx = "llvmgcc" };
    variant = variant or "debug";
    version = version or "%s %s %s" % { os.date("%Y.%m.%d %H:%M:%S"), platform, variant };
    goal = goal or "";
    jobs = jobs or 4;

    local settings = build.initialize {
        bin = root( "../bin" );
        lib = root( "../lib" );
        obj = root( "../obj" );
        include_directories = {
            root( ".." )
        };
        sln = root( "../sweet_lua.sln" );
        xcodeproj = root( "../sweet_lua.xcodeproj" );
    };

    if operating_system() == "windows" then
        mingw.initialize( settings );
        msvc.initialize( settings );
        visual_studio.initialize( settings );
    elseif operating_system() == "macosx" then
        llvmgcc.initialize( settings );
        xcode.initialize( settings );
    end
end

function buildfiles()
    buildfile( "assert/assert.build" );
    buildfile( "error/error.build" );
    buildfile( "lua/lua.build" );
    buildfile( "rtti/rtti.build" );
    buildfile( "traits/traits.build" );
    buildfile( "unit/unit.build" );
end

The first few lines are similar to the previous example. They load the required build scripts. In this case they also load build scripts that support building using Apple LLVM-GCC and MinGW and also generation of Visual Studio and Xcode project files for the project.

The initialize() function is again similar just with a few additions. The assignments to variables at the start of the function are setting the default values of variables that may be set on the command line. The or idiom used takes advantage of short circuiting to preserve any value that may already have been set from the command line otherwise it uses the default value on the right. The build.switch() function sets the default platform to either "msvc" (on Microsoft Windows) or "llvmgcc" (on Apple MacOSX).

The call to build.initialize() also provides project specific settings that override the default settings used by the build system. The settings that are overridden are the directories that executables, libraries, and object files are built to; the directories that are searched for header files; and the names of the Visual Studio solution and Xcode project to generate respectively.

Different modules are initialized depending on the operating system that the build tool is being run on. If on Microsoft Windows the MinGW, Microsoft Visual C++, and Visual Studio modules are initialized. If on Apple MacOSX the LLVM-GCC and Xcode modules are initialized.

The buildfiles() function loads the dependency graph from several buildfiles spread through the project directory hierarchy. The buildfile() function takes a relative path to the buildfile to load as its only parameter. The function sets the working directory to the directory that contains the buildfile and then executes the buildfile as a Lua script. Changing the working directory has the effect of making any relative paths expressed in the buildfile relative to the directory that the buildfile is in.

The use of separate buildfiles in combination with the inheritance of settings described in the following section allows buildfiles to be reused in a modular way. For example the lua/lua.build buildfile above is used again in the Sweet Build project but with different settings inherited from the build.lua script in the Sweet Build project without any changes needing to be made to the lua/lua.build buildfile.

For more real world examples have a look at the Sweet Build source or any of the other libraries at http://www.sweetsoftware.co.nz/.

3.3. Settings

The build.initialize() function called from the initialize() function of a build.lua file allows the default settings used by the build system to be overridden on a per project basis.

The build.initialize() function takes as its sole parameter a table that is merged into a table that defines the default settings used by the build system. Merging combines the tables so that fields with table values end up having the values that are specified in both tables while fields with simple values use the value in the second table (the project settings in this case) in place of the value in the first table (the default settings in this case). This merged settings table becomes the default settings used by the build system for building that specific project.

More settings are loaded from the optional user_settings.lua file located in the current user’s home directory and then the local_settings.lua file in the root directory of the project and merged into a table that is then set to inherit from the default settings.

The user_settings.lua file provides a way to specify per-user and/or per-machine configuration that should apply to all projects. For example the install directory for common external libraries such as Boost or Qt might be specified in the user_settings.lua file.

The local_settings.lua file provides a place for settings autodetected by the build system to be stored and potentially edited by the user. For example the build system automatically detects the directory that Microsoft Visual C++ is installed in and writes it to local_settings.lua but only if local_settings.lua doesn’t already exist. The local_settings.lua file can then be edited and the autodetected values overridden.

This settings table then becomes the global settings that are applied to the top level executable and library targets specified for the project.

Targets are able to define their own settings table. When the build system loads those targets their settings tables are set to inherit from their parent’s settings table (for targets such as Source targets) or the global settings table (for top level targets such as Executable and Library targets). Targets that don’t define their own settings use the settings table of their parent or the global settings table. The inheritance allows settings to be conditionally overridden on a per target basis. Any settings that aren’t overridden are inherited from the global or parent settings tables.

For example the following buildfile is used to build the build tool executable. It overrides the settings for subsystem and stack size but inherits the values for all other settings from the global settings.

Executable {
    id = "build";

    settings = {
        subsystem = "CONSOLE";
        stack_size = 32768;
    };

    libraries = {
        "assert/assert",
        "build_tool/build_tool",
        "cmdline/cmdline",
        "debug/debug",
        "error/error",
        "lua/lua",
        "lua/lua_/liblua",
        "path/path",
        "persist/persist",
        "pointer/pointer",
        "process/process",
        "rtti/rtti",
        "thread/thread",
    };

    Source {
        pch = "stdafx.hpp";
        "Application.cpp",
        "main.cpp"
    };
}

Any of the following settings can be passed to the build.initialize() call in the build.lua file or as fields in the settings table of an Executable, Library, or Source target.

  • The bin setting sets the directory that executables and dynamic libraries are generated in.

  • The lib setting sets the directory that static libraries are generated in.

  • The obj setting sets the directory that object files and other intermediate files are generated in.

  • The include_directories setting sets the include directories to use when compiling C and C++ source files.

  • The library_directories setting sets the library directories to search when linking executables and dynamic libraries.

  • The platforms setting sets the platforms that are valid to build for.

  • The variants setting sets variants that are valid to build for and per variant settings.

  • The compile_as_c setting sets whether to compile source files as C or C++.

  • The debug setting sets whether or not to generate debug information.

  • The exceptions setting sets whether or not to enable C++ exceptions.

  • The generate_map_file setting sets whether or not to generate a map file.

  • The incremental_linking setting sets whether or not to use incremental linking.

  • The library_type setting sets the meaning of the Library target prototype to mean StaticLibrary if its value is "static" or DynamicLibrary is its value is "dynamic".

  • The link_time_code_generation setting sets whether or not to use link time code generation.

  • The minimal_rebuild setting sets whether or not to use minimal rebuild.

  • The optimization setting sets whether or not to optimize code.

  • The pre_compiled_headers setting sets whether or not to use precompiled headers.

  • The preprocess setting enables preprocessing source code rather than compiling it for debugging preprocessor errors.

  • The profiling setting enables or disables profiling in the build.

  • The run_time_checks setting enables run time checks.

  • The runtime_library setting controls which C runtime to link with. It’s value can be "static_debug" to link to the static debug runtime, "static" to link to the static runtime, "dynamic_debug" to link to the dynamic debug runtime or "dynamic" to link to the dynamic runtime.

  • The run_time_type_info settings enables and disables run time type information.

  • The stack_size setting sets the stack size of executables.

  • The string_pooling setting enables and disables string pooling.

  • The subsystem setting sets the value to pass as subsystem when linking (either "CONSOLE" or "WINDOWS").

  • The verbose_linking setting enables verbose linking for debugging linker errors.

3.4. Target Prototypes

The Executable target prototype defines an executable to be linked. Its identifier is the name of the executable to link less any platform and variant dependent suffix that the build system will add, e.g. "_msvc_debug.exe". It may contain a libraries key to specify the libraries that the executable depends on. It may contain a settings key to specify any target specific settings that override the global settings. Its contents are the Source and Parser targets that specify the source files to generate, compile, and link to create the executable.

The StaticLibrary target prototype defines a static library to be archived. Its identifier is the name of the library to archive less any platform and variant dependent suffix that the build system will add, e.g. "_msvc_debug.lib". It may contain a settings key to specify any target specific settings that override the global settings. Its contents are the Source and Parser targets that specify the source files to generate, compile, and archive to create the library. It may contain a libraries key that is ignored to allow for the case where a library specified by the Library target prototype alias is building a static library.

The DynamicLibrary target prototype defines a dynamic library to be linked. Its identifier is the name of the library to link less any platform and variant dependent suffix that the build system will add, e.g. "_msvc_debug.dll". It may contain a libraries key to specify the libraries that the library depends on. It may contain a settings key to specify any target specific settings that override the global settings. Its contents are the Source and Parser targets that specify the source files to generate, compile, and link to create the dynamic library.

The Library target prototype is an alias for either the StaticLibrary or DynamicLibrary target prototype depending on the value of the library_type setting.

The Source target prototype defines a group of C or C++ source files. Its identifier is ignored. It may contain a pch attribute to specify the pre-compiled header to use. It may contain a defines attribute to specify extra preprocessor macros to be defined when compiling. It may contain a settings key to specify any target specific settings that override the global settings or settings inherited from its containing target. It must appear within an Executable, StaticLibrary, DynamicLibrary, or Library target.

The Parser target prototype defines a group of grammar files to be processed into header files with the Sweet Parser tool. Its identifier is ignored. It must appear within an Executable, StaticLibrary, DynamicLibrary, or Library target.

The SourceFile target prototype defines a source file that must exist. It doesn’t generally appear in buildfiles but is used by the build system to expand other targets during the "load", "static_depend", and "depend" passes.

The HeaderFile target prototype defines a header file (or source file that doesn’t need to exist). It doesn’t generally appear in buildfiles but is used by the build system to expand other targets during the "load", "static_depend", and "depend" passes.

The File target prototype defines a generated file. It doesn’t generally appear in buildfiles but is used by the build system to expand other targets during the "load", "static_depend", and "depend" passes.

3.5. Preprocessor Macros

The following preprocessor macros are defined when compiling C and C++ source. They allow conditional compilation based on the platform, variant, and module being built and pass automatically generated version information to the build.

  • BUILD_PLATFORM_x is defined to indicate the platform being built where x is replaced by the uppercase platform name. For example BUILD_PLATFORM_MSVC is defined when building for the "msvc" platform.

  • BUILD_VARIANT_x is defined to indicate the variant that is being built where x is replaced by the uppercase variant name. For example BUILD_PLATFORM_RELEASE is defined when building the "release" variant.

  • BUILD_MODULE_x is defined to indicate the module being built where x is replaced by the uppercase executable or library identifier. For example BUILD_MODULE_LUA is defined when building the "lua" library.

  • BUILD_LIBRARY_TYPE_x is defined to indicate whether dynamic or static libraries are being built where x is replaced by STATIC or DYNAMIC for "static" or "dynamic" library types respectively.

  • BUILD_LIBRARY_SUFFIX is defined to be the suffix appended to library names to distinguish between libraries for different platforms and variants. For example BUILD_LIBRARY_SUFFIX is defined as "_msvc_debug.lib" when building for the "msvc" platform and "debug" variant.

  • BUILD_VERSION is defined to be the date, variant, and platform of the build.

The preprocessor macro BUILD_LIBRARY_TYPE_DYNAMIC is defined when libraries should be compiled and linked into dynamic libraries instead of static ones. This is used in conjunction with the BUILD_MODULE_x macro to create macros that evaluate to __declspec(dllexport), __declspec(dllimport), or nothing based on whether a dynamic library is being compiled, a dynamic library is being linked with, or a static library is being compiled or linked with.

The BUILD_VERSION macro is not defined when precompiled headers are created. This is because its value changes based on the time of the build and this causes the macro to be defined to different values for the precompiled header when the precompiled header is compiled at a different time from the source files that include it.

3.6. Preprocessor And Linker Debugging

It is possible to generate preprocessed files from source files rather than passing them to the the compiler. This is done by setting the settings field preprocess to true.

It is also possible to set the linker to generate verbose output about the libraries it is searching for symbols and why it is searching them. This is done by setting the settings field verbose_linking to true.

4. Reference

4.1. Targets

At its core Sweet Build is a dependency graph. The nodes in the graph represent the files generated by a build and the connections represent the dependencies between those files. The actions of the build system are carried out by making traversals over the dependency graph and carrying out different operations as each node in the graph is visited.

The nodes in the graph are referred to as targets. Targets are created by calling the target() function in a build script or buildfile.

Typically buildfiles call the functions that make up the domain specific language (Executable, Library, Source, etc) and these functions then create appropriate targets to populate the dependency graph.

Targets exist in a hierarchical namespace with semantics similar to operating system paths. Each target has an identifier, a single parent target, and zero or more child targets. A target path is then made up of target identifiers separated by "/" characters and using "." and ".." to refer to the current and parent targets respectively.

Existing targets can be found using the find_target() function which takes a target path and finds the target at that path; relative paths are considered relative to the current working directory.

The target() function, used to create targets, also takes a target path as its first parameter. Again relative paths are considered relative to the current working directory. Passing a path to target() that refers to a target outside of the current working directory will create a target that is also outside of the current working directory - just as it would behave in a typical filesystem operation. Targets are always created by treating their identifier as a path relative to their working directory to place them correctly in the hierarchy.

The working directory of a target is whatever the current working directory was when the target was created. Usually this is the directory that contains the buildfile that indirectly constructed the target (for targets constructed in a buildfile) or the current working directory at the time the target was created (for targets created at other times).

Targets that are created with an explicit identifier are implicitly added as dependencies of the current working directory. This allows all of the targets defined in a directory to be built simply by specifying that the target that represents that directory be built.

Targets may also be created anonymously by passing an empty string as their identifier. This creates a target using a uniquely generated identifier. Targets created this way aren’t added as dependencies of the working directory - they’re expected to be added as dependencies of the right target by the build system.

4.2. Target Prototypes

Although it is possible to create targets that have no specific target prototype the majority of explicitly created targets have a target prototype that defines their behaviour.

A target prototype is created by calling the TargetPrototype() function passing in the name and bind type of the target prototype and assigning the table that it returns to a variable. Functions are then defined on that variable to specify what happens when a target for that target prototype is visited during a traversal.

For example the Directory target prototype defines a target prototype whose targets bind as directories and that create a directory with a name matching their identifier when they are outdated.

DirectoryPrototype = TargetPrototype { "Directory", BIND_DIRECTORY };

function DirectoryPrototype.build( directory )
    if directory:is_outdated() then
        mkdir( directory:get_filename() );
    end
end

function Directory( directory )
  return target( directory, DirectoryPrototype );
end

Using target prototypes is the only way to associate functions with targets and have them saved and loaded from the dependency graph correctly. The serialization code is unable to save or load functions and so it ignores occurences of functions when processing target tables. The target prototype relationship is serialized correctly and so will be recovered or maintained across a save and load of the dependency graph.

4.3. Traversal

Once the initial dependency graph has been populated it is traversed several times to expand explicit and implicit dependencies, bind targets to their associated files in the filesystem, and carry out the actions required to bring the files in the build up to date.

The dependency relationships between targets form a directed graph that dictates the order in which targets are visited during a traversal. The two available traversals are preorder and postorder. A preorder traversal visits each target in the graph before it visits that target’s dependencies. A postorder traversal visits each target’s dependencies before it visits that target. Each traversal takes a visit function as a parameter and that function is called on each target to carry out the visit operation. Cyclic references are quietly ignored.

The visit function typically calls to a member function in the target being visited to do the processing for the traversal. For example most target prototypes define a build() function that is called during the "build" traversal to build targets for that target prototype. Targets that don’t provide the function are assumed to be successfully visited. The dependencies of a target are always visited regardless of whether or not the depending target provides a function for the pass. The visit is considered successful if no errors are raised during its execution.

Typical traversals made over a graph are a preorder traversal to propagate settings (the "load" pass), a preorder traversal to create explicit dependencies between targets (the "static_depend" pass), a preorder traversal to create implicit dependencies between targets (the "depend" pass), and then a postorder traversal to build any targets that are outdated (the "build" pass) or a postorder traversal to remove generated files (the "clean" pass).

The "load" pass is a preorder traversal that propagates settings through the targets created when buildfiles are loaded.

The "static_depend" pass is a preorder traversal that create dependencies based on the values specified in targets. These dependencies only change when the buildfiles themselves change and so this pass only needs to be run when the cache is outdated. This pass happens after the "load" pass because it relies on settings having been propagated.

The "depend" pass is a preorder traversal that creates implicit dependencies by scanning source files for regular expressions (e.g. recursively scanning C or C++ source files for the header files that they include).

The "build" pass is a postorder traversal in which targets are built if they are outdated. The postorder traversal provides the correct ordering to ensure that dependencies are built before the targets that depend on them.

The "clean" pass is a postorder traversal that removes files generated during the "build" pass.

4.4. Binding

The bind pass is a special case postorder traversal in which targets are bound to files and their dependencies. It is implemented in C++ to avoid having to make Lua calls to visit every target in the graph.

The bind type of a target prototype specifies how targets for that target prototype bind to files and how they are considered outdated with respect to their dependencies. Bind type can be BIND_PHONY, BIND_DIRECTORY, BIND_SOURCE_FILE, BIND_INTERMEDIATE_FILE, or BIND_GENERATED_FILE.

Targets that bind as source or intermediate files set their timestamp to the later of the timestamp of the file they are bound to or the latest timestamp of all their dependencies and are never outdated. Targets that bind as generated files set their timestamp to the timestamp of the file they are bound to and are outdated if any of their dependencies have a timestamp that is newer than theirs. Targets that bind themselves as phony targets set their timestamp to the latest timestamp of all of their dependencies and are considered to be outdated if any of their dependencies are outdated.

4.5. Scanning

Sweet Build is able to scan files for regular expressions and call Lua functions when those regular expression match. The value of each sub-expression in the regular expression is passed as a parameter in the call to the match function.

The scan() function takes a target representing the source file to scan and a Scanner that defines the regular expressions to match and the functions to call when the expressions are matched. Scanners are defined by calling the Scanner() function passing a table of regular expressions to scan for and and match functions to execute when matches are found.

For example the CcScanner definition below recursively scans C/C++ source and header files for dependencies implied by the use of the include directive:

CcScanner = Scanner {
    [ [[^#include "([^"\n\r]*)"]] ] = function( target, match )
        local header = HeaderFile( target:directory()..match );
        target:add_dependency( header );
        if header:get_bind_type() ~= BIND_INTERMEDIATE_FILE then
            scan( header, CcScanner );
        end
    end;

    [ [[^#include <([^>\n\r]*)>]] ] = function( target, match )
        local filename = root( "../"..match );
        if exists(filename) then
            local header = HeaderFile( filename );
            target:add_dependency( header );
            if header:get_bind_type() ~= BIND_INTERMEDIATE_FILE then
                scan( header, CcScanner );
            end
        end
    end;
}

The build system internally stores the last write time at the time a target was last scanned. If the target’s last write time hasn’t changed since the last time it was scanned then scanning silently does nothing. This is an optmization to avoid scanning files unnecessarily.

Once it has been determined that a target should be scanned a simple heuristic is used to determine how many lines of the file to scan. There is a maximum number of lines scanned at the start of the file without finding a match and a maximum number of lines scanned after a finding match without finding another match. If either of these maximums are exceeded then scanning stops. These values can be set using the functions Scanner:set_initial_lines() and Scanner:set_later_lines(). The default values for the initial and later lines to scan are both 0 which is interpreted as no limit - all lines in the source file will be scanned.

Scanning happens in parallel with the main thread processing. This means that match functions are executed asynchronously to the calling coroutine. The coroutine that called scan() continues execution immediately and matches are only actually processed when that coroutine finishes or the wait() function is called. This can lead to some unexpected results where matches put values into tables and the caller is expecting match results to be available immediately.

4.6. Execution

The execute() function optionally accepts a Scanner parameter. If a Scanner is passed to execute() then the output of the command is matched against the regular expressions in the Scanner. Any matches result in the corresponding match function being called asynchronously in a new coroutine. Lines that don’t match any regular expression are passed straight through to the output.

As in the scan() function the value of each sub-expression in the regular expression is passed as a parameter to the match function.

This can be used to filter the output of compilers or tools for better recognition by IDEs or for capturing the output of tools for use in the build system itself.

For example the following script executes the Subversion tool to query the properties of a version controlled directory and prints the output to the console. It could just as easily store the output in a table for later processing as part of the build:

local PropertyFilter = Scanner {
    [ [[([A-Za-z0-9_$/\.]+) ([A-Za-z0-9_$/\.]+)]] ] = function( repository_path, filesystem_path )
        print( "%s -> %s" % {repository_path, filesystem_path} );
    end;
};

local cmd = "C:/windows/system32/cmd.exe";
execute( cmd, "cmd /C svn propget svn:externals .", PropertyFilter );

Be aware that the execute() function suspends execution of the Lua coroutine that it is called on and that this can lead to race conditions in the Lua scripts in much the same way that it does in other environments.

If execute() is called as part of a postorder traversal to execute commands to build external files and the dependencies are expressed correctly then there should be no problems with race conditions. The ordering based on dependencies ensures that race conditions don’t occur.

If execute() is being called to generate data in a shared structure during a preorder pass be very aware that the preorder traversal will generate multiple coroutines and that those coroutines will be running at the same time and scheduled according to the sequence and timings of the execute() calls.

4.7. Loading and Saving

The dependency graph can be saved and loaded to and from binary or XML files. This is useful for caching implied dependencies to speed up incremental builds. For example it is quite time consuming to scan C++ source files and headers for extra dependencies implied by include directives but once it has been done that graph can be saved to a file and reloaded again to recreate the same dependencies. Only the C++ source files and headers that have changed since the graph was last saved need to be scanned again.

The boolean, numeric, string, and table values stored in Lua tables are also saved and reloaded as part of the cache. This includes correctly persisting cyclic relationships between tables.

Function and closure values are not saved. This is generally not a problem because functions and closures are defined in target prototypes and the target prototype relationship of each target is preserved across a save and a load.

4.8. Errors

Errors are reported by calling the error() function and passing a string that describes the error that has occured. This displays the error message on the console and causes the current processing to fail.

Errors that occur while visiting a target mark the target and any dependent targets as failing. The dependent targets are not visited (although an error message is displayed for each dependent target that is not visited). Processing of other targets that aren’t part of the same dependency chain continues.

4.9. Debugging

Debugging the build system is unfortunately only possible by adding prints at appropriate places. There are two functions, print_dependencies() and print_namespace(), that can be used to display the dependency graph and namespace respectively - this can be useful for tracking down why something is being built when it shouldn’t and vice versa.

5. API

5.1. Configuration Functions

set_maximum_parallel_jobs ( maximum_parallel_jobs )

Set the maximum number of commands allowed to execute at once. The minimum is 1 and the maximum is 64. The default is 1.

get_maximum_parallel_jobs ()

Get the maximum number of commands that can be executed at once.

set_stack_trace_enabled ( stack_trace_enabled )

Set whether or not stack traces are displayed when errors occur. The default is false.

is_stack_trace_enabled ()

Are stack traces displayed when errors occur? Returns true if stack traces are enabled otherwise false.

5.2. File System Functions

cp ( source, destination )

Copy a source file to a destination file.

cpdir ( source, destination )

Recursively copy source to destination. Recursively copies newer files from the source directory to the destination directory. Directories and files that start with a dot (.) are ignored.

exists ( path )

Check whether or not the file or directory at path exists. Returns true if path refers to an existing file or directory otherwise false.

find ( path )

Recursively list the contents of path and any descended directories. If path is relative then it is treated as being relative to the current working directory. The directory passed in is assumed to be a directory and its contents returned as an iterator. Glob patterns are not used - any filtering based on pattern matching must be done by the caller as each entry in the directory tree is returned.

is_directory ( path )

Check whether or not path is a directory. Returns true if path is a directory otherwise false.

is_file ( path )

Check whether path is a file. Returns true if path is a file otherwise false.

ls ( path )

List the contents of path (which is assumed to be a directory). If path is relative then it is treated as being relative to the current working directory. The directory passed in is assumed to be a directory and its contents returned as an iterator. Glob patterns are not used - any filtering based on pattern matching must be done by the caller as each entry in the directory is returned.

mkdir ( path )

Make the directory path. If path is relative it is treated as being relative to the current working directory. Any intermediate directories specified in the directory passed in that do not already exist are also created.

rm ( path )

Remove a file.

rmdir ( path )

Recursively remove a directory. Recursively removes a directory and all of its content. Be careful!

5.3. Path and String Functions

absolute ( path, [working_directory] )

Convert path into an absolute path by prepending working_directory or the current working directory if working_directory is not specified. If the path is already absolute then it is returned unchanged.

basename ( path )

Return the basename of path (everything except for the extension of which the dot "." is considered part, i.e. the dot "." is not returned as part of the basename).

branch ( path )

Return all but the last element of path.

extension ( path )

Return the extension of path (including the dot ".").

home ( [path] )

Convert path into a directory relative to the current user’s home directory. If path is omitted then the current user’s home directory is returned.

initial ( [path] )

Convert path into a directory relative to the directory that the build tool was invoked from. If path is omitted then the initial directory is returned.

is_absolute ( path )

Check whether or path is absolute. Returns true if path is absolute otherwise false.

is_relative ( path )

Check whether or path is relative. Returns true if path is relative otherwise false.

leaf ( path )

Return the last element of path.

lower ( value )

Convert value to lower case. Returns value converted to lower case letters.

native ( path )

Convert path into its native equivalent. Returns path converted into its native equivalent.

relative ( path, [working_directory] )

Convert path into a path relative to working_directory or relative to the current working directory if working_directory is not specified. If the path is already relative then it is returned unchanged.

root ( [path] )

Convert path into a path relative to the directory that the build.lua file was found in when searching up from the initial directory. If path is omitted then the root directory is returned.

upper ( value )

Convert value to upper case. Returns value converted to upper case letters.

5.4. Working Directory Functions

cd ( path )

Change the current working directory to path. If path is relative it is treated as being relative to the current working directory.

popd ()

Pop the current working directory and restore the working directory to the working directory saved by the most recent call to pushd(). If the current working directory is the only directory on the directory stack then this function silently does nothing.

pushd ( path )

Push the current working directory (so that it can be returned to later by calling popd()) and set the new current working directory to path. If path is relative it is treated as being relative to the current working directory.

pwd ()

Get the current (present) working directory. Returns the current working directory.

5.5. Environment Functions

getenv ( name )

Get the value of an environment attribute. If the environment attribute doesn’t exist then this function returns nil.

putenv ( attribute, value )

Set the value of an environment attribute. Setting an environment attribute to the empty string clears that environment attribute.

5.6. Operating System Functions

execute ( command, arguments, filter )

Executes command passing arguments as the command line and optionally using filter to process the output.

The command will be executed in a thread and processing of any jobs that can be performed in parallel continues. Returns the value returned by command when it exits. The filter parameter is optional and can be nil to pass the output through to the console unchanged.

hostname ()

Get the hostname of the computer that the build tool is running on. Returns the hostname.

operating_system ()

Return a string that identifies the operating system that the build tool is being run on - either "windows" or "macosx".

print ( text )

Print text to stdout.

sleep ( duration )

Do nothing for duration milliseconds.

whoami ()

Get the name of the user account that the build tool is running under. Returns the user name.

5.7. Graph Functions

bind ( [target] )

Bind the targets to files and set their outdated status and timestamps accordingly. Returns the number of targets that failed to bind because their files were expected to exist but didn’t (see Target:set_required_to_exist()).

buildfile ( filename )

Load a buildfile from filename.

find_target ( id )

Find a target in this Graph whose identifier matches id. If id is a relative path then it is treated as being relative to the current working directory.

load_binary ( filename )

Load a previously saved dependency graph from filename.

save_binary ( filename )

Save the current dependency graph to filename.

load_xml ( filename )

Load a previously saved dependency graph from filename.

save_xml ( filename )

Save the current dependency graph to filename.

postorder ( visitor, target )

Perform a postorder traversal of targets calling the visitor function for each target visited. Returns the number of visited targets that failed because they generated an error during their visitation.

preorder ( visitor, _target )

Perform a preorder traversal of targets calling the visitor function for each target visited. Returns the number of visited targets that failed because they generated an error during their visitation.

print_dependencies ( [target] )

Print the dependency tree of Targets in this Graph. If target is nil then dependencies from the entire Graph are printed otherwise dependencies of target are recursively printed.

print_namespace ()

Print the namespace of Targets in this Graph. If target is nil then the namespace of the entire Graph is printed otherwise only Targets that are descended from target are printed.

target ( [id], [target_prototype], [table] )

Create or return an existing target.

The id parameter specifies a target path that uniquely identifies the target to create or return. If id specifies a relative path then it is considered relative to the current working directory. If id is the empty string or nil then an anonymous target is created using a unique identifier to create a target in the current working directory.

The target_prototype parameter specifies the target prototype to use when creating the target. It is an error for the same target to be created with more than one target prototype. If the target_prototype parameter is omitted or nil then a target without any target prototype is created.

The table parameter specifies the table to use to represent the target in the Lua virtual machine. If the table parameter is omitted or nil then an empty table is created and used.

5.8. Scanners

Scanner { [regex = function]* }

Create a scanner that calls function for each match of regex in its input from a target file or from the output of an externally executed process.

scan ( target, scanner, )

Use this scanner to scan the file that has been bound to target. Any additional arguments are passed through as additional parameters to match functions that are called as a result of matching regular expressions in the scanned file.

Scanner:set_initial_lines ( initial_lines )

Set the maximum number of unmatched lines allowed at the start of a file before scanning is stopped.

Scanner:get_initial_lines ()

Get the maximum number of unmatched lines allowed at the start of a file before scanning is stopped.

Scanner:set_later_lines ( later_lines )

Set the maximum number of unmatched lines allowed after at least one matched line before scanning is stopped.

Scanner:get_later_lines ()

Get the maximum number of unmatched lines allowed after at least one matched line before scanning is stopped.

5.9. Targets

Target:id ()

Get the identifier of this target. Returns the identifier of this target.

Target:path ()

Get the full path of this target. Returns the full path of this target.

Target:directory ()

Get the directory part of the path of this target. Returns the directory part of the path of this target.

Target:parent ()

Get the parent of this target. Returns the parent of this target. This is the target’s parent in the hierarchical namespace of targets not the target’s parent in the dependency graph (of which there may be many).

Target:prototype ()

Get the target prototype for this target. Returns the target prototype for this target or nil if this target was implicitly created as a working directory for other targets or doesn’t have a target prototype.

Target:set_bind_type ( bind_type )

Set the bind type for this target. This overrides the bind type specified by a target’s TargetPrototype. See the TargetPrototype documentation for more details on the different bind types. The bind type passed to a target can also be BIND_NULL to set the target to use the bind type specified by its TargetPrototype.

Target:get_bind_type ()

Get the bind type for this target.

Target:set_required_to_exist ( required_to_exist )

Set whether or not this target is required to be bound to an existing file or not. The check for existence is done when targets are bound and an error is reported for any targets that have required to exist set to true but are bound to files that don’t exist.

Target:is_required_to_exist ()

Is this target required to be bound to an existing file? Returns true if this target is required to be bound to an existing file otherwise false.

Target:set_timestamp ( timestamp )

Set the timestamp of this target to timestamp.

The timestamp is the time that is used to determine whether generated targets are outdated with respect to their dependencies. targets that are bound as generated files that have any dependencies with a timestamp later than theirs are considered to be outdated and in need of update.

Target:get_timestamp ()

Get the timestamp of this target.

If this target isn’t bound to a file then the last write time is always the beginning of the epoch - January 1st, 1970, 00:00 GMT. Because this is the oldest possible timestamp this will leave unbound targets always needing to be updated.

Returns the timestamp of this target.

Target:set_outdated ( outdated )

Set whether or not this target is outdated.

Target:is_outdated ()

Is this target outdated? Returns true if this target is outdated otherwise false.

Target:set_filename ( filename )

Set the filename of this target to filename. This is the name of the file that this target will attempt to bind to during the bind traversal.

Target:get_filename ()

Get the filename of this target. Returns the filename of this target or an empty string if this target hasn’t been bound to a file.

Target:set_working_directory ( target )

Set the working directory of this target to target or to the root directory of its containing graph if target is nil.

The working directory is the target that specifies the directory that files specified in scripts for the target are relative to by default and that any commands executed by the target’s script functions are started in. If a target has no working directory then the root target is used instead.

Target:get_working_directory ()

Get the working directory of this target. Returns the working directory of this target or nil if this target doesn’t have a working directory.

Target:add_dependency ( dependency )

Add a dependency to this target. Cyclic dependencies are not valid but are not reported by this function. If a cyclic dependency is detected during a traversal then it is silently ignored. If dependency is nil then this function will silently do nothing.

Target:get_dependencies ()

Iterate over the dependencies of this target. Returns an iterator over all of the dependencies of this target.

5.10. Target Prototypes

TargetPrototype { name, bind_type [, functions*] }

Create a new target prototype that can be used to create new targets during a traversal.

The name parameter specifies the name that the target prototype is referred to in buildfiles. By convention this should be the same as the name of the Lua value that it is assigned to.

The bind_type parameter specifies how the targets for this target prototype bind to files. It can be any of BIND_PHONY to not bind to any file, BIND_DIRECTORY to bind to a directory, BIND_SOURCE_FILE to bind to a source file that must exist, BIND_INTERMEDIATE_FILE to bind to a source file that need not exist or a generated source file, or BIND_GENERATED_FILE to bind to a generated file.

A bind type of BIND_PHONY binds targets as phonies. They are targets that aren’t bound to any file or directory; are outdated if any of their dependencies are outdated; and have the timestamp of their newest dependency.

A bind type of BIND_DIRECTORY binds targets to directories. The target is bound to a directory; is outdated if the directory doesn’t exist; and has its timestamp set to the earliest possible time so that it won’t cause any of the targets that depend on the directory target to be outdated (as they simply require that the directory exists they don’t care if the directory is newer or not).

A bind type of BIND_SOURCE_FILE or BIND_INTERMEDIATE_FILE binds targets to intermediate or source files. The targets are bound to files; their timestamp is set to the latest timestamp of the file that they are bound to and all their dependencies; they are outdated if any of their dependencies are outdated or if any of their dependencies are newer than they are. Targets that are bound as source files must also exist and an error will be thrown if they aren’t found.

A bind type of BIND_GENERATED_FILE binds targets to generated files. The targets are bound to files; their timestamp is set to the last write time of the file that they are bound to; they are outdated if any of their dependencies are newer than they are.

The bind type can be overridden on a per target basis by calling Target:set_bind_type() for the targets that need to have bind types different to that specified by their rules.