The previous attempt to use PRI* macros to avoid compiler-specific
printf format specifiers didn't work because we are using raw
long long type, not the uintX_t types. On systems where long and
long long are the same size, uint64_t might be type "long" and PRId64
is just "ld".
Instead write new functions that convert [unsigned] long long to a
string and call those instead.
* src/makeint.h: Declare make_lltoa() and make_ulltoa().
* src/misc.c (make_lltoa): New function that writes a long long value
into a provided buffer. Return the buffer for ease-of-use.
(make_ulltoa): Ditto, for unsigned long long.
* src/function.c (func_wordlist): Call these new methods. Also
rework the error strings so we share the translated string.
* src/dir.c (print_dir_data_base): Call the new methods instead of
using MSVC macros.
We may change the global environ variable in the child; when using
vfork() this also sets it in the parent. Preserve the parent's
environ in child_execute_job() so it takes effect for all callers.
Reported by Denis Excoffier <bug-tar@Denis-Excoffier.org>
Root cause found by Martin Dorey <Martin.Dorey@hitachivantara.com>
* src/job.c (start_job_command): Remove save/restore of the parent
environment.
(child_execute_job): Add save/restore of the parent environment,
if we use vfork().
* tests/scripts/functions/shell: Add a test the crashes if we don't
reset environ after we run $(shell ...).
* src/w32/w32os.c (osync_get_mutex, osync_parse_mutex): Cast to
DWORD_PTR when converting integers to HANDLEs and vice versa.
* src/w32/pathstuff.c (w32ify): Pacify compiler warnings about
'strncpy'.
* src/makeint.h (PRId64) [!HAVE_INTTYPES_H]: Define if undefined.
* src/function.c (func_wordlist): Use PRId64 instead of %lld.
* src/implicit.c: (pattern_search):
* src/main.c: (main): Avoid compilation warnings for variables
only used when compiling a debug version with assertions.
Fail if a mandatory include file fails to be built even if it's
built as part of a grouped target where the other include file
is optional.
* src/main.c (main): If a makefile doesn't build set any_failed.
* tests/scripts/features/include: Add tests.
* tests/scripts/options/dash-k: Stop after include build failure.
If any of a set of grouped targets is missing or out of date, even
if make is not trying to build that target, rebuild them all.
Ensure this is true for explicit grouped targets as well as pattern
rule grouped targets.
Original patch by Jonathan Gravel <jo@stashed.dev>
* src/remake.c (update_file_1): After matching any pattern rules,
go through the also_make targets and set noexist as needed. Also
compute the oldest this_mtime.
* tests/scripts/features/grouped_targets: Add regression tests.
* tests/scripts/features/patternrules: Ditto.
* tests/features/vpath: Rewrite to use modern run_make_test().
Add a test that we check for VPATH before implicit rule search.
Move the tests in vpath2 and vpath3 into this suite.
* tests/features/vpathplus: Rewrite to use modern run_make_test().
The next version of the POSIX standard defines parallel execution
and requires the .WAIT special target as is implemented in some other
versions of make.
This implementation behaves similarly to others in that it does not
create a relationship between targets in the dependency graph, so
that the same two targets may be run in parallel if they appear as
prerequisites elsewhere without .WAIT between them.
Now that we support .WAIT it's trivial to also support prerequisites
of the .NOTPARALLEL special target, which forces the prerequisites of
those targets to be run serially (as if .WAIT was specified between
each one).
* NEWS: Announce the new .WAIT and .NOTPARALLEL support.
* doc/make.texi (Parallel Disable): A new section to discuss ways in
which parallel execution can be controlled. Modify cross-refs to
refer to this section.
* src/dep.h (struct dep): Add a new wait_here boolean.
(parse_file_seq): Add PARSEFS_WAIT to check for .WAIT dependencies.
* src/file.c (split_prereqs): Use PARSEFS_WAIT.
(snap_deps): If .NOTPARALLEL has prerequisites, set .WAIT between
each of _their_ prerequisites.
(print_prereqs): Add back in .WAIT when printing prerequisites.
* src/implicit.c (struct patdeps): Preserve wait_here.
(pattern_search): Ditto. Use PARSEFS_WAIT when parsing prereqs for
pattern rule expansion.
* src/read.c (check_specials): Don't give up early: remembering to
update these options is not worth the rare speedup.
(check_special_file): If .WAIT is given as a target show an error--
once--if it has prereqs or commands.
(record_files): Call check_special_file on each target.
(parse_file_seq): If PARSEFS_WAIT is given, look for .WAIT prereqs.
If we see one assume that we are building a struct dep chain and set
the wait_here option while not putting it into the list.
* src/remake.c (update_file_1): If wait_here is set and we are still
running, then stop trying to build this target's prerequisites.
* src/rule.c (get_rule_defn): Add .WAIT to the prerequisite list.
* src/shuffle.c (shuffle_deps): Don't shuffle the prerequisite list
if .WAIT appears anywhere in it.
* tests/scripts/targets/WAIT: Add a test suite for this feature.
If makefile rules do not update an unloaded shared object, load it
again. Avoid double loading of the same object if the setup function
returns -1.
* src/filedef.h (struct file): Add "unloaded" flag.
* src/makeint.h (load_file): Take struct file *.
(unload_file): Return int.
* src/main.c (main): Reload unloaded shared objects if they weren't
updated.
* src/commands.c (execute_file_commands): Set "unloaded" and reset
"loaded" when a shared object is unloaded.
* src/read.c (eval): Set "loaded" and reset "unloaded" when a shared
object is loaded. Add successfully loaded files to the db.
* src/load.c (load_file): Check "loaded" to avoid double loading the
same object. Fix a memory leak of string loaded. Return -1, rather
than 1, if the object is already loaded. This fixes double loading of
the same object when the setup routine returns -1.
(load_object): Add a log message.
(unload_file): Return an error on dlclose failure. Log a message.
* tests/scripts/features/loadapi: Add new tests.
Commit 07eea3aa4 `make --shuffle` prevented shuffling prerequisites
that use .SECONDEXPANSION, since shuffle happens before expansion.
This has two problems:
1. No shuffling happens for such prerequisites.
2. Use-after-free when outdated '->shuf' links are used.
Add a reshuffle into expansion phase right after dependency changes.
* src/file.c (expand_deps): Add reshuffle if dependencies change.
* src/shuffle.c (identity_shuffle_array): Fix comment typo.
* tests/scripts/options/shuffle: Add new SECONDEXPANSION test.
Allow build systems to choose an alternative location for make to
store its temporary files.
* NEWS: Announce the new environment variable.
* doc/make.texi (Temporary Files): Provide documentation.
* src/misc.c (get_tmpdir): Split into a new function. Compute the
temporary directory and store it in a static location.
* tests/scripts/features/jobserver: Add a test of MAKE_TMPDIR.
* src/load.c (load_file): Update .LOADED if setup returns non-0.
* tests/scripts/features/load: Change the return value based on an
environment variable. Ensure that returning -1 still adds to
.LOADED. Also add a test that verifies that make doesn't try to
rebuild the loaded file if -1 is returned.
Notes provided by Tzvetelin Katchov <katchov@gnu.org>.
* doc/make.texi: Remove obsolete @refill statements. Add some breaks
to overfull lines. Fix up some example text.
Notes provided by Tzvetelin Katchov <katchov@gnu.org>
* maintMakefile (git-very-clean): Add -x to git clean.
(GNULIBDIR): Use GNULIB_SRCDIR, or the local module if not set.
* tests/scripts/features/load: Add function prototypes.
* tests/scripts/features/temp_stdin: Only set user execute bits.
Some configurations don't allow changing the group bits...??
If we detect a recursive variable reference when constructing the
environment for the shell function, return the original value from the
caller's environment. Other options such as failing, returning the
empty string, or returning the unexpanded make variable value have
been shown to not behave well in real-world environments. If the
variable doesn't exist in the caller's environment, return the empty
string.
Found by Sergei Trofimovich <slyich@gmail.com> when testing older
versions of autoconf.
* NEWS: Clarify this behavior.
* doc/make.texi (Shell Function): Ditto. Also add info about !=.
* src/expand.c (recursively_expand_for_file): Search the caller's
environment if we detect a recursive variable expansion.
* tests/scripts/functions/shell: Add tests for this behavior.
The fix for SV 10593 caused recursive expansion errors when exporting
a variable that contains a $(shell ...) invocation. If we see this
type of recursion, ignore it and expand to the empty string rather
than failing.
* src/variable.h (env_recursion): New global variable.
* src/variable.c (target_environment): If creating the environment
for a $(shell ...) function increment env_recursion. Remove the
check for expansion in a shell function context.
* src/expand.c (recursively_expand_for_file): Check for recursive
expansion in a $(shell ...) environment context and if present,
show the verbose message and return the empty string.
* tests/scripts/functions/shell: Add a test for this situation.
For reproducibility ensure all files in the dist tar file are owned by
root, and sorted by name. Force the mode on all files to be writable
by the owner and not writable by group or user, by default.
Remove unnecessary execute permissions from build_w32.bat.
Recommended by Tzvetelin Katchov <katchov@gnu.org>.
* README.git: Document the changes.
* bootstrap.conf: Set GNULIB_REVISION to avoid random updates.
* configure.ac: Set the minimum autoconf version back to 2.69:
that's the most common version deployed today, and it still works.
* bootstrap: Copy the latest from gnulib.
* autogen.sh: Ditto.
* autopull.sh: Ditto.
* bootstrap-funclib.sh: Ditto.
* maintMakefile: Remove the template headers as prerequisites.
* tests/scripts/features/jobserver: Only test fifo if enabled.
* tests/scripts/variables/INCLUDE_DIRS: On MacOS none of the default
directories exist so .INCLUDE_DIRS is empty by default.
* tests/scripts/features/se_explicit: Fail via exit. cp will show
different error messages on different systems.
* tests/scripts/features/se_implicit: Ditto.
* tests/scripts/features/se_statpat: Ditto.
If users run 'make --version | head -n1' they expect make to exit
with a success (0) code. This works because the pipe forces the
default buffering on stdout to be fully buffered so all the output
is printed to the pipe in a single write(2) and won't fail. However
due to output sync we forcibly set stdout to line buffered, which
means if the reader closes the pipe fast enough make will exit with
an error code because the write to stdout failed.
Move the setup done in output_init() back into main() where it can
be done in a proper order. Rework the order of operations during
startup so that we check for help and version flags before we change
the buffering. Clean up the behavior of print_usage().
Original changes from Dmitry Goncharov <dgoncharov@users.sf.net>.
* src/main.c (switches): Don't send --version in the environment.
(print_usage): Add a blank line after the version before the usage.
Move the die() into this function since we always die() afterward.
Note the die will flush so no need to do it explicitly.
(print_version): The caller will fflush when appropriate.
(close_stdout): Move from output.c so it is installed early.
(decode_switches): Only call print_usage on error, not for --help.
(main): Install the close_stdout handler immediately after start.
Move the calls to print_usage() due to --help and --version to be
called immediately after we decode the switches. Move the buffer set
here from output_init(), immediately after we know we'll be running.
* src/output.c (output_init): Move buffer setting to main().
(close_stdout): Move to main().
Some POSIX systems do not allow locks to be taken on non-files, such
as pipes. This is a problem since very often make is invoked with
its stdout redirected to a pipe. Also, if stdout is redirected to a
file that already has a lock on it for some other reason (perhaps a
shared file such as /dev/null) it can cause a hang.
This means our previous method of locking stdout, although it had some
nice advantages, is not portable enough. Instead, use a temporary
file and take the lock on that. We pass the name of the file to child
make processes. On Windows we continue to use a shared mutex for
output sync.
Remove POSIX emulation functions like fcntl from Windows; instead
follow the lead of the jobserver and create an interface in os.h for
output sync, and move the OS-specific content to posixos.c and
w32os.c.
* NEWS: Add a note.
* src/makeint.h (ALL_SET): Check that all bits are set.
* src/os.h: Add bits for checking the state of stdin/stdout/stderr.
Add prototypes for OS-specific output sync methods.
* src/posixos.c (check_io_state): Determine the status of stdin,
stdout, stderr an return a suite of bits describing them.
(osync_enabled): If the global variable holding the FD of the lock
file (osync_handle) is valid return true.
(osync_setup): Create a temporary file and remember its name in a
global variable (osync_tmpfile), and set osync_handle.
(osync_get_mutex): If output sync is enabled, return the filename
of the lock file prefixed with "fnm:" to denote a filename.
(osync_parse_mutex): If the provided filename has the wrong format
disable output sync. Else open the lock file and set osync_handle.
(osync_clear): Close osync_handle. If we're the parent make, then
also unlink the temporary file.
(osync_acquire): Take a lock on the osync_handle descriptor.
(osync_release): Release the lock on the osync_handle descriptor.
(fd_set_append): Add APPEND mode to a file descriptor.
* src/w32/w32os.c: Perform the same actions as posixos.c, copying
the details from src/w32/compat/posixfcn.c. Use a mutex rather
than locking a temporary file.
* src/output.h: Remove all the OS-specific content.
* src/output.c: Remove all the OS-specific content.
(set_append_mode): Remove and replace with fd_set_append().
(sync_init): Remove and replace with check_io_state().
(acquire_semaphore): Remove and replace with osync_acquire().
(release_semaphore): Remove and replace with osync_release().
(setup_tmpfile): If the IO state is not obtained, get it. If stdout
and/or stderr are valid, set up a tempfile to capture them.
(output_init): Set io_state if not set already, and check it when
deciding whether to close stdout on exit.
* src/main.c (main): If we're syncing, set up the mutex using the
new osync_setup() / osync_parse_mutex() methods.
(prepare_mutex_handl_string): Replace with osync_parse_mutex().
(die): Call osync_clear().
* src/w32/compat/posixfcn.c: Remove implementations of fcntl(),
record_sync_mutex(), create_mutex(), and same_stream().
The output sync feature wants a file descriptor not a FILE*. We were
using tmpfile() but this returns FILE* which means we needed to dup()
the descriptor then fclose() the original, which is just unnecessary
overhead for every command we run.
Create a get_tmpfd() method that returns a file descriptor directly
by using mkstemp() if available, else do the best we can.
Also allow anonymous temp files if the filename pointer is NULL.
This causes the file to be unlinked. On Windows this requires a
special open so add an os_anontmp() method to handle this.
* src/makeint.h: Add prototype for get_tmpfd().
* src/misc.c (get_tmpfd): If we have mkstemp() use that, else just
open(2). If we don't want to keep the filename, unlink the file.
(get_tmpfile): Use get_tmpfd() if we have fdopen(), else use fopen().
* src/output.c (output_tmpfd): Call get_tmpfd() with NULL.
* src/os.h (os_anontmp): On Windows make this a function, else fails.
* src/w32/compat/posixcfn.c (tmpfile): Move to w32os.c:os_anontmp().
* src/w32/w32os.c (os_anontmp): Create a temp file that will be deleted
when the process exits, and return a file descriptor to it.
Reset the temp directory for every test to a local directory, then
after each test see if any new temp files were created and not
deleted: if they were then fail the test. Rather than delete the
temp files we leave them there and avoid reporting files that were
seen before, so the user can investigate them.
Rewrite the temp_stdin tests to rely on this built-in behavior
rather than implementing the checks directly.
* tests/test_driver.pl: Create a $TEMPDIR variable pointing to a
temporary directory outside the test temp directory.
(toplevel) Before starting any tests create a temp directory and set
the POSIX and Windows temp directory environment variables to use it.
(compare_output) Check the contents of the temp directory. If any
new files have appeared, fail the test.
* tests/scripts/features/temp_stdin: Remove check_tempfile() and
all users of it, plus setting of temp environment variables.
* README.git: Clarify that these methods are lightly tested.
* build_w32.bat: Don't support any config step: fail if not completed.
Move the config steps into bootstrap.bat. Don't print compile lines
by default and add a --verbose option to show them.
* bootstrap.bat: Ensure we have curl and sed before we do anything.
Pull the latest necessary files from gnulib. Create a convert.sed
script that can update the various template files, and update
Basic.mk, config.h.W32, and gmk-default.h.
* tests/run_make_tests.pl: Remove CRLF rather than using chop. If we
run perl in Git for Bash it seems to handle newlines differently.
* tests/scripts/features/temp_stdin: Remove the make copy and close
STDIN so we can delete the temp file on Windows.
* .gitignore: Ignore the convert.sed script.
Using anonymous pipes for jobserver support has some advantages:
for example there is nothing on disk that needs to be cleaned up.
However it has many obscure problems, related to the fact that in
order for it to work we need to ensure these resources are properly
passed through to child processes that want to use the jobserver.
At the same time we don't want to pass the pipe to process which
DON'T know about the jobserver.
Other processes can open file descriptors which we then think are
our jobserver, but aren't. And, we open the pipe file descriptors
in blocking mode which doesn't work for all users.
See issues such as SV 57178, SV 57242, and SV 62397
To avoid these issues, use named pipes (on systems where they are
available) instead of anonoymous pipes. This simplifies many things:
we never need to pass open file descriptors to our children; they
can open the jobserver named pipe. We don't need to worry about
recursive vs. non-recursive children. Users don't have to "pass
through" the resources if they are invoking sub-makes. Each child
can open its own file descriptor and set blocking as needed.
The downside is the named pipe exists on disk and so must be cleaned
up when the "top-level" make instance exits.
In order to allow make to continue to be used in build systems where
older versions of GNU make, or other tools that want to use the
jobserver, but don't understand named pipes, introduce a new option
--jobserver-style that allows the user to choose anonymous pipes.
* NEWS: Announce the change and the --jobserver-style option.
* doc/make.1: Add --jobserver-style documentation.
* doc/make.texi (Special Variables): Add missing items to .FEATURES.
(Options Summary): Add --jobserver-style.
(POSIX Jobserver): Named pipes, changes to --jobserver-auth, and the
--jobserver-style option.
(Windows Jobserver): Document --jobserver-style for Windows.
* configure.ac: Check for mkfifo.
* src/config.h-vms.template: Undefined HAVE_MKFIFO.
* src/config.h.W32.template: Ditto.
* src/main.c: Add jobserver-style as a new command line option.
(main): Add jobserver-fifo to .FEATURES if supported. Pass the style
option to jobserver_setup().
* src/os.h (jobserver_setup): Accept a style string option.
* src/posixos.c (enum js_type): Enumeration of the jobserver style.
(js_type): Which style we are currently using.
(fifo_name): The path to the named pipe (if in use).
(jobserver_setup): If no style is given, or "fifo" is given, set up a
named pipe: get a temporary file and use mkfifo() on it, then open it
for reading and writing. If something fails fall back to anonymous
pipes.
(jobserver_parse_auth): Parse jobserver-auth to determine the style.
If we are using a named pipe, open it. If we're using anonymous pipes
ensure they're valid as before.
(jobserver_get_invalid_auth): Don't invalidate the jobserver when
using named pipes.
(jobserver_clear): Clean up memory used for named pipes.
(jobserver_acquire_all): Unlink the named pipe when done.
* src/w32/w32os.c (jobserver_setup): Check the style argument.
* tests/scripts/features/jobserver: Use --jobserver-style to test
the anonymous pipe behavior, and also test named pipe/semaphore
behavior. Check invalid jobserver-style options.
* tests/scripts/functions/shell: Use --jobserver-style to test the
anonymous pipe behavior, and also test named pipe/semaphore
behavior.
Move all the logic on creating temporary files into misc.c, and add
a new function get_tmppath() that returns the pathname to a temporary
file without creating or opening it.
* src/makeint.h: Add a declaration for get_tmppath(). Remove the
template argument from get_tmpfile(): it will compute its own.
* src/main.c (main): Remove the logic for computing templates.
* src/vmsjobs.c (child_execute_job): Ditto.
* src/misc.c (get_tmptemplate): New function to return an allocated
template string for use with various mktemp-style functions.
(get_tmppath): Return an allocated path to a temporary file, but do
not create it. Generally this should be avoided due to TOCTOU issues.
(get_tmpfile): Use get_tmptemplate() to generate a template rather
than using one passed in. If we don't have mkstemp() then use
get_tmppath() to compute the path of a temp file.
Second-expand only the prerequisites of the targets being built.
Defer second-expanding the prerequisites of targets until we need
to decide if they should be built.
* NEWS: Mention the change in behavior.
* doc/make.texi (Secondary Expansion): Document the new behavior.
* src/filedef.h (struct file): Add flag snapped.
(expand_deps): Declare a function to second expand the
prerequisites of a target.
* src/file.c (rehash_file): Merge flag snapped.
(expand_deps): Remove qualifier static. Check flag snapped.
(snap_deps): Remove the loop which performed second expansion for all
targets.
* src/remake.c (update_file_1): Second expand the prerequisites of
the considered target.
* tests/scripts/features/se_explicit: Add tests.
* tests/scripts/features/se_implicit: Ditto.
* tests/scripts/features/se_statpat: Ditto.
Savannah issues such as SV 57242 and SV 62397 show how passing
references to closed file descriptors via the --jobserver-auth option
in MAKEFLAGS can lead to problematic outcomes.
When computing the child environment for a non-recursive shell, add
an extra option to MAKEFLAGS to disable the file descriptors for the
jobserver.
Unfortunately this doesn't modify the value of the make variable
MAKEFLAGS, it only modifies the value of the sub-shell environment
variable MAKEFLAGS. This can lead to confusion if the user is not
considering the distinction.
* src/makeint.h: Publish the jobserver-auth value. Add a global
definition of the name of the command line option.
* src/os.h (jobserver_get_invalid_auth): New function to return a
string invalidating the jobserver-auth option.
* src/w32/w32os.c (jobserver_get_invaid_auth): Implement it. On
Windows we use a semaphore so there's no need to invalidate.
* src/posixos.c (jobserver_parse_auth): If we parse the invalid
auth string, don't set up the jobserver.
(jobserver_get_invalid_auth): Return an invalid option.
* src/variable.h (target_environment): Specify if the target
environment is for a recursive shell or non-recursive shell.
* src/variable.c (target_environment): Move checking for MAKELEVEL
into the loop rather than doing it at the end.
Along with this, check for MAKEFLAGS and MFLAGS, and update them
based on whether we're invoking a recursive or non-recursive child,
and also on whether it's necessary to invalidate the jobserver.
* src/function.c (func_shell_base): Shell functions can never be
recursive to pass 0 to target_environment().
* src/job.c (start_job_command): Specify whether the child is
recursive when calling target_environment().
* src/main.c: Export jobserver_auth. sync_mutex doesn't need to
be exported. Use the global definition for the option name.
* tests/scripts/variables/MAKEFLAGS: Add tests for $MAKEFLAGS.
