Remake
 All Classes Files Functions Variables Typedefs Enumerations Enumerator Macros Groups Pages
remake.cpp
Go to the documentation of this file.
1 /**
2 @mainpage Remake, a build system that bridges the gap between make and redo.
3 
4 As with <b>make</b>, <b>remake</b> uses a centralized rule file, which is
5 named <b>Remakefile</b>. It contains rules with a <em>make</em>-like
6 syntax:
7 
8 @verbatim
9 target1 target2 ... : prerequisite1 prerequisite2 ...
10  shell script
11  that builds
12  the targets
13 @endverbatim
14 
15 A target is known to be up-to-date if all its prerequisites are. If it
16 has no known prerequisites yet the file already exits, it is assumed to
17 be up-to-date. Obsolete targets are rebuilt thanks to the shell script
18 provided by the rule.
19 
20 As with <b>redo</b>, <b>remake</b> supports dynamic dependencies in
21 addition to these static dependencies. Whenever a script executes
22 <tt>remake prerequisite4 prerequisite5 ...</tt>, these prerequisites are
23 rebuilt if they are obsolete. (So <b>remake</b> acts like
24 <b>redo-ifchange</b>.) Moreover, all the dependencies are stored in file
25 <b>.remake</b> so that they are remembered in subsequent runs. Note that
26 dynamic dependencies from previous runs are only used to decide whether a
27 target is obsolete; they are not automatically rebuilt when they are
28 obsolete yet a target depends on them. They will only be rebuilt once the
29 dynamic call to <b>remake</b> is executed.
30 
31 In other words, the following two rules have almost the same behavior.
32 
33 @verbatim
34 target1 target2 ... : prerequisite1 prerequisite2 ...
35  shell script
36 
37 target1 target2 ... :
38  remake prerequisite1 prerequisite2 ...
39  shell script
40 @endverbatim
41 
42 (There is a difference if the targets already exist, have never been
43 built before, and the prerequisites are either younger or obsolete, since
44 the targets will not be rebuilt in the second case.)
45 
46 The above usage of dynamic dependencies is hardly useful. Their strength
47 lies in the fact that they can be computed on the fly:
48 
49 @verbatim
50 %.o : %.c
51  gcc -MMD -MF $@.d -o $@ -c $<
52  remake -r < $@.d
53  rm $@.d
54 
55 %.cmo : %.ml
56  ocamldep $< | remake -r $@
57  ocamlc -c $<
58 
59 after.xml: before.xml rules.xsl
60  xsltproc --load-trace -o after.xml rules.xsl before.xml 2> deps
61  remake `sed -n -e "\\,//,! s,^.*URL=\"\\([^\"]*\\).*\$,\\1,p" deps`
62  rm deps
63 @endverbatim
64 
65 Note that the first rule fails if any of the header files included by
66 a C source file has to be automatically generated. In that case, one
67 should perform a first call to <b>remake</b> them before calling the
68 compiler. (Dependencies from several calls to <b>remake</b> are
69 cumulative, so they will all be remembered the next time.)
70 
71 \section sec-usage Usage
72 
73 Usage: <tt>remake <i>options</i> <i>targets</i></tt>
74 
75 Options:
76 
77 - <tt>-d</tt>: Echo script commands.
78 - <tt>-f FILE</tt>: Read <tt>FILE</tt> as <b>Remakefile</b>.
79 - <tt>-j[N]</tt>, <tt>--jobs=[N]</tt>: Allow <tt>N</tt> jobs at once;
80  infinite jobs with no argument.
81 - <tt>-k</tt>, <tt>--keep-going</tt>: Keep going when some targets cannot be made.
82 - <tt>-r</tt>: Look up targets from the dependencies on standard input.
83 - <tt>-s</tt>, <tt>--silent</tt>, <tt>--quiet</tt>: Do not echo targets.
84 
85 \section sec-syntax Syntax
86 
87 Lines starting with a space character or a tabulation are assumed to be rule
88 scripts. They are only allowed after a rule header.
89 
90 Lines starting with <tt>#</tt> are considered to be comments and are ignored.
91 They do interrupt rule scripts though.
92 
93 Any other line is either a rule header or a variable definition. If such a
94 line ends with a backslash, the following line break is ignored and the line
95 extends to the next one.
96 
97 Rule headers are a nonempty list of names, followed by a colon, followed by
98 another list of names, possibly empty. Variable definitions are a single
99 name followed by equal followed by a list of names, possibly empty. Basically,
100 the syntax of a rule is as follows:
101 
102 @verbatim
103 targets : prerequisites
104  shell script
105 @endverbatim
106 
107 List of names are space-separated sequences of names. If a name contains a
108 space character, it should be put into double quotes. Names can not be any
109 of the following special characters <tt>:$(),="</tt>. Again, quotation
110 should be used. Quotation marks can be escaped by a backslash inside
111 quoted names.
112 
113 \subsection sec-variables Variables
114 
115 Variables can be used to factor lists of targets or prerequisites. They are
116 expanded as they are encountered during <b>Remakefile</b> parsing.
117 
118 @verbatim
119 VAR2 = a
120 VAR1 = c d
121 VAR2 += $(VAR1) b
122 $(VAR2) e :
123 @endverbatim
124 
125 Variable assignments can appear instead of prerequisites inside non-generic
126 rules with no script. They are then expanded inside the corresponding
127 generic rule.
128 
129 @verbatim
130 foo.o: CFLAGS += -DBAR
131 
132 %.o : %.c
133  gcc $(CFLAGS) -MMD -MF $@.d -o $@ -c $<
134  remake -r < $@.d
135  rm $@.d
136 @endverbatim
137 
138 Note: contrarily to <b>make</b>, variable names have to be enclosed in
139 parentheses. For instance, <tt>$y</tt> is not a shorthand for <tt>\$(y)</tt> and
140 is left unexpanded.
141 
142 \subsection sec-autovars Automatic variables
143 
144 The following special symbols can appear inside scripts:
145 
146 - <tt>$&lt;</tt> expands to the first static prerequisite of the rule.
147 - <tt>$^</tt> expands to all the static prerequisites of the rule, including
148  duplicates if any.
149 - <tt>$\@</tt> expands to the first target of the rule.
150 - <tt>$*</tt> expands to the string that matched <tt>%</tt> in a generic rule.
151 - <tt>$$</tt> expands to a single dollar symbol.
152 
153 Note: contrarily to <b>make</b>, there are no corresponding variables. For
154 instance, <tt>$^</tt> is not a shorthand for <tt>$(^)</tt>. Another
155 difference is that <tt>$\@</tt> is always the first target, not the one that
156 triggered the rule.
157 
158 \subsection sec-functions Built-in functions
159 
160 <b>remake</b> also supports a few built-in functions inspired from <b>make</b>.
161 
162 - <tt>$(addprefix <i>prefix</i>, <i>list</i>)</tt> returns the list obtained
163  by prepending its first argument to each element of its second argument.
164 - <tt>$(addsuffix <i>suffix</i>, <i>list</i>)</tt> returns the list obtained
165  by appending its first argument to each element of its second argument.
166 
167 \section sec-semantics Semantics
168 
169 \subsection src-obsolete When are targets obsolete?
170 
171 A target is obsolete:
172 
173 - if there is no file corresponding to the target, or to one of its siblings
174  in a multi-target rule,
175 - if any of its dynamic prerequisites from a previous run or any of its static
176  prerequisites is obsolete,
177 - if the latest file corresponding to its siblings or itself is older than any
178  of its dynamic prerequisites or static prerequisites.
179 
180 In all the other cases, it is assumed to be up-to-date (and so are all its
181 siblings). Note that the last rule above says "latest" and not "earliest". While
182 it might cause some obsolete targets to go unnoticed in corner cases, it allows
183 for the following kind of rules:
184 
185 @verbatim
186 config.h stamp-config_h: config.h.in config.status
187  ./config.status config.h
188  touch stamp-config_h
189 @endverbatim
190 
191 A <tt>config.status</tt> file generally does not update header files (here
192 <tt>config.h</tt>) if they would not change. As a consequence, if not for the
193 <tt>stamp-config_h</tt> file above, a header would always be considered obsolete
194 once one of its prerequisites is modified. Note that touching <tt>config.h</tt>
195 rather than <tt>stamp-config_h</tt> would defeat the point of not updating it
196 in the first place, since the program files would need to be rebuilt.
197 
198 Once all the static prerequisites of a target have been rebuilt, <b>remake</b>
199 checks whether the target still needs to be built. If it was obsolete only
200 because its prerequisites needed to be rebuilt and none of them changed, the
201 target is assumed to be up-to-date.
202 
203 \subsection sec-rules How are targets (re)built?
204 
205 There are two kinds of rules. If any of the targets or prerequisites contains
206 a <tt>%</tt> character, the rule is said to be <em>generic</em>. All the
207 targets of the rule shall then contain a single <tt>%</tt> character. All the
208 other rules are said to be <em>specific</em>.
209 
210 A rule is said to <em>match</em> a given target:
211 
212 - if it is specific and the target appears inside its target list,
213 - if it is generic and there is a way to replace the <tt>%</tt> character
214  from one of its targets so that it matches the given target.
215 
216 When <b>remake</b> tries to build a given target, it looks for a specific rule
217 that matches it. If there is one and its script is nonempty, it uses it to
218 rebuild the target.
219 
220 Otherwise, it looks for a generic rule that match the target. If there are
221 several matching rules, it chooses the one with the shortest pattern (and if
222 there are several ones, the earliest one). <b>remake</b> then looks for
223 specific rules that match each target of the generic rule. All the
224 prerequisites of these specific rules are added to those of the generic rule.
225 The script of the generic rule is used to build the target.
226 
227 Example:
228 
229 @verbatim
230 t%1 t2%: p1 p%2
231  commands building t%1 and t2%
232 
233 t2z: p4
234  commands building t2z
235 
236 ty1: p3
237 
238 # t2x is built by the first rule (which also builds tx1) and its prerequisites are p1, px2
239 # t2y is built by the first rule (which also builds ty1) and its prerequisites are p1, py2, p3
240 # t2z is built by the second rule and its prerequisite is p4
241 @endverbatim
242 
243 The set of rules from <b>Remakefile</b> is ill-formed:
244 
245 - if any specific rule matching a target of the generic rule has a nonempty script,
246 - if any target of the generic rule is matched by a generic rule with a shorter pattern.
247 
248 \section sec-compilation Compilation
249 
250 - On Linux, MacOSX, and BSD: <tt>g++ -o remake remake.cpp</tt>
251 - On Windows: <tt>g++ -o remake.exe remake.cpp -lws2_32</tt>
252 
253 Installing <b>remake</b> is needed only if <b>Remakefile</b> does not
254 specify the path to the executable for its recursive calls. Thanks to its
255 single source file, <b>remake</b> can be shipped inside other packages and
256 built at configuration time.
257 
258 \section sec-differences Differences with other build systems
259 
260 Differences with <b>make</b>:
261 
262 - Dynamic dependencies are supported.
263 - For rules with multiple targets, the shell script is executed only once
264  and is assumed to build all the targets. There is no need for
265  convoluted rules that are robust enough for parallel builds. For generic
266  rules, this is similar to the behavior of pattern rules from <b>gmake</b>.
267 - As with <b>redo</b>, only one shell is run when executing a script,
268  rather than one per script line. Note that the shells are run with
269  option <tt>-e</tt>, thus causing them to exit as soon as an error is
270  encountered.
271 - The prerequisites of generic rules (known as implicit rules in make lingo)
272  are not used to decide between several of them. <b>remake</b> does not
273  select one for which it could satisfy the dependencies.
274 - Variables and built-in functions are expanded as they are encountered
275  during <b>Remakefile</b> parsing.
276 
277 Differences with <b>redo</b>:
278 
279 - As with <b>make</b>, it is possible to write the following kind of rules
280  in <b>remake</b>.
281 @verbatim
282 Remakefile: Remakefile.in ./config.status
283  ./config.status Remakefile
284 @endverbatim
285 - If a target is already built the first time <b>remake</b> runs, it still
286  uses the static prerequisites of rules mentioning it to check whether it
287  needs to be rebuilt. It does not assume it to be up-to-date. As with
288  <b>redo</b> though, if its obsolete status would be due to a dynamic
289  prerequisite, it will go unnoticed; it should be removed beforehand.
290 - Multiple targets are supported.
291 - <b>remake</b> has almost no features: no checksum-based dependencies, no
292  compatibility with job servers, etc.
293 
294 \section sec-limitations Limitations
295 
296 - When the user calls <b>remake</b>, the current working directory should be
297  the one containing <b>.remake</b>. Rules are understood relatively to this
298  directory. If a rule script calls <b>remake</b>, the current working
299  directory should be the same as the one from the original <b>remake</b>.
300 - Some cases of ill-formed rules are not caught by <b>remake</b> and can
301  thus lead to unpredictable behaviors.
302 
303 \section sec-links Links
304 
305 @see http://cr.yp.to/redo.html for the philosophy of <b>redo</b> and
306 https://github.com/apenwarr/redo for an implementation and some comprehensive documentation.
307 
308 \section sec-licensing Licensing
309 
310 @author Guillaume Melquiond
311 @version 0.9
312 @date 2012-2013
313 @copyright
314 This program is free software: you can redistribute it and/or modify
315 it under the terms of the GNU General Public License as published by
316 the Free Software Foundation, either version 3 of the License, or
317 (at your option) any later version.
318 \n
319 This program is distributed in the hope that it will be useful,
320 but WITHOUT ANY WARRANTY; without even the implied warranty of
321 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
322 GNU General Public License for more details.
323 
324 \section sec-internals Internals
325 
326 The parent <b>remake</b> process acts as a server. The other ones have a
327 REMAKE_SOCKET environment variable that tells them how to contact the
328 server. They send the content of the REMAKE_JOB_ID environment variable,
329 so that the server can associate the child targets to the jobs that
330 spawned them. They then wait for completion and exit with the status
331 returned by the server. This is handled by #client_mode.
332 
333 The server calls #load_dependencies and #save_dependencies to serialize
334 dynamic dependencies from <b>.remake</b>. It loads <b>Remakefile</b> with
335 #load_rules. It then runs #server_mode, which calls #server_loop.
336 
337 When building a target, the following sequence of events happens:
338 
339 - #start calls #find_rule (and #find_generic_rule) to get the rule.
340 - It then creates a pseudo-client if the rule has static dependencies, or
341  calls #run_script otherwise. In both cases, a new job is created and its
342  targets are put into #job_targets.
343 - #run_script creates a shell process and stores it in #job_pids. It
344  increases #running_jobs.
345 - The child process possibly calls <b>remake</b> with a list of targets.
346 - #accept_client receives a build request from a child process and adds
347  it to #clients. It also records the new dependencies of the job into
348  #dependencies. It increases #waiting_jobs.
349 - #handle_clients uses #get_status to look up the obsoleteness of the
350  targets.
351 - Once the targets of a request have been built or one of them has failed,
352  #handle_clients calls #complete_request and removes the request from
353  #clients.
354 - If the build targets come from a pseudo-client, #complete_request calls
355  #run_script. Otherwise it sends the reply to the corresponding child
356  process and decreases #waiting_jobs.
357 - When a child process ends, #server_loop calls #finalize_job, which
358  removes the process from #job_pids, decreases #running_jobs, and calls
359  #complete_job.
360 - #complete_job removes the job from #job_targets and calls #update_status
361  to change the status of the targets. It also removes the target files in
362  case of failure.
363 */
364 
365 #ifdef _WIN32
366 #define WIN32_LEAN_AND_MEAN
367 #define WINDOWS
368 #endif
369 
370 #include <fstream>
371 #include <iostream>
372 #include <list>
373 #include <map>
374 #include <set>
375 #include <sstream>
376 #include <string>
377 #include <vector>
378 #include <cassert>
379 #include <cstdlib>
380 #include <ctime>
381 #include <errno.h>
382 #include <fcntl.h>
383 #include <signal.h>
384 #include <unistd.h>
385 #include <sys/stat.h>
386 #include <sys/types.h>
387 
388 #ifdef __APPLE__
389 #define MACOSX
390 #endif
391 
392 #ifdef __linux__
393 #define LINUX
394 #endif
395 
396 #ifdef WINDOWS
397 #include <windows.h>
398 #include <winbase.h>
399 #include <winsock2.h>
400 #define pid_t HANDLE
401 typedef SOCKET socket_t;
402 #else
403 #include <sys/socket.h>
404 #include <sys/un.h>
405 #include <sys/wait.h>
406 typedef int socket_t;
407 enum { INVALID_SOCKET = -1 };
408 extern char **environ;
409 #endif
410 
411 #if defined(WINDOWS) || defined(MACOSX)
412 enum { MSG_NOSIGNAL = 0 };
413 #endif
414 
415 typedef std::list<std::string> string_list;
416 
417 typedef std::set<std::string> string_set;
418 
419 /**
420  * Reference-counted shared object.
421  * @note The default constructor delays the creation of the object until it
422  * is first dereferenced.
423  */
424 template<class T>
425 struct ref_ptr
426 {
427  struct content
428  {
429  size_t cnt;
430  T val;
431  content(): cnt(1) {}
432  content(T const &t): cnt(1), val(t) {}
433  };
434  mutable content *ptr;
435  ref_ptr(): ptr(NULL) {}
436  ref_ptr(T const &t): ptr(new content(t)) {}
437  ref_ptr(ref_ptr const &p): ptr(p.ptr) { if (ptr) ++ptr->cnt; }
438  ~ref_ptr() { if (ptr && --ptr->cnt == 0) delete ptr; }
440  {
441  if (ptr == p.ptr) return *this;
442  if (ptr && --ptr->cnt == 0) delete ptr;
443  ptr = p.ptr;
444  if (ptr) ++ptr->cnt;
445  return *this;
446  }
447  T &operator*() const
448  {
449  if (!ptr) ptr = new content;
450  return ptr->val;
451  }
452  T *operator->() const { return &**this; }
453 };
454 
456 {
459 };
460 
461 typedef std::map<std::string, ref_ptr<dependency_t> > dependency_map;
462 
463 typedef std::map<std::string, string_list> variable_map;
464 
465 /**
466  * Build status of a target.
467  */
469 {
470  Uptodate, ///< Target is up-to-date.
471  Todo, ///< Target is missing or obsolete.
472  Recheck, ///< Target has an obsolete dependency.
473  Running, ///< Target is being rebuilt.
474  Remade, ///< Target was successfully rebuilt.
475  Failed ///< Build failed for target.
476 };
477 
478 /**
479  * Build status of a target.
480  */
481 struct status_t
482 {
483  status_e status; ///< Actual status.
484  time_t last; ///< Last-modified date.
485 };
486 
487 typedef std::map<std::string, status_t> status_map;
488 
489 /**
490  * Delayed assignment to a variable.
491  */
492 struct assign_t
493 {
494  std::string name;
495  bool append;
497 };
498 
499 typedef std::list<assign_t> assign_list;
500 
501 /**
502  * A rule loaded from Remakefile.
503  */
504 struct rule_t
505 {
506  string_list targets; ///< Files produced by this rule.
507  string_list deps; ///< Files used for an implicit call to remake at the start of the script.
508  assign_list vars; ///< Values of variables.
509  std::string script; ///< Shell script for building the targets.
510  std::string stem; ///< String used to instantiate a generic rule.
511 };
512 
513 typedef std::list<rule_t> rule_list;
514 
515 typedef std::map<std::string, ref_ptr<rule_t> > rule_map;
516 
517 typedef std::map<int, string_list> job_targets_map;
518 
519 typedef std::map<pid_t, int> pid_job_map;
520 
521 /**
522  * Client waiting for a request to complete.
523  *
524  * There are two kinds of clients:
525  * - real clients, which are instances of remake created by built scripts,
526  * - pseudo clients, which are created by the server to build specific targets.
527  *
528  * Among pseudo clients, there are two categories:
529  * - original clients, which are created for the targets passed on the
530  * command line by the user or for the initial regeneration of the rule file,
531  * - dependency clients, which are created to handle rules that have
532  * explicit dependencies and thus to emulate a call to remake.
533  */
534 struct client_t
535 {
536  socket_t socket; ///< Socket used to reply to the client (invalid for pseudo clients).
537  int job_id; ///< Job for which the built script called remake and spawned the client (negative for original clients).
538  bool failed; ///< Whether some targets failed in mode -k.
539  string_list pending; ///< Targets not yet started.
540  string_set running; ///< Targets being built.
541  rule_t *delayed; ///< Rule that implicitly created a dependency client, and which script has to be started on request completion.
542  client_t(): socket(INVALID_SOCKET), job_id(-1), failed(false), delayed(NULL) {}
543 };
544 
545 typedef std::list<client_t> client_list;
546 
547 /**
548  * Map from variable names to their content.
549  */
551 
552 /**
553  * Map from targets to their known dependencies.
554  */
556 
557 /**
558  * Map from targets to their build status.
559  */
561 
562 /**
563  * Set of generic rules loaded from Remakefile.
564  */
566 
567 /**
568  * Map from targets to specific rules loaded from Remakefile.
569  */
571 
572 /**
573  * Map from jobs to targets being built.
574  */
576 
577 /**
578  * Map from jobs to shell pids.
579  */
581 
582 /**
583  * List of clients waiting for a request to complete.
584  * New clients are put to front, so that the build process is depth-first.
585  */
587 
588 /**
589  * Maximum number of parallel jobs (non-positive if unbounded).
590  * Can be modified by the -j option.
591  */
592 static int max_active_jobs = 1;
593 
594 /**
595  * Whether to keep building targets in case of failure.
596  * Can be modified by the -k option.
597  */
598 static bool keep_going = false;
599 
600 /**
601  * Number of jobs currently running:
602  * - it increases when a process is created in #run_script,
603  * - it decreases when a completion message is received in #finalize_job.
604  *
605  * @note There might be some jobs running while #clients is empty.
606  * Indeed, if a client requested two targets to be rebuilt, if they
607  * are running concurrently, if one of them fails, the client will
608  * get a failure notice and might terminate before the other target
609  * finishes.
610  */
611 static int running_jobs = 0;
612 
613 /**
614  * Number of jobs currently waiting for a build request to finish:
615  * - it increases when a build request is received in #accept_client
616  * (since the client is presumably waiting for the reply),
617  * - it decreases when a reply is sent in #complete_request.
618  */
619 static int waiting_jobs = 0;
620 
621 /**
622  * Global counter used to produce increasing job numbers.
623  * @see job_targets
624  */
625 static int job_counter = 0;
626 
627 /**
628  * Socket on which the server listens for client request.
629  */
631 
632 /**
633  * Whether the request of an original client failed.
634  */
635 static bool build_failure;
636 
637 #ifndef WINDOWS
638 /**
639  * Name of the server socket in the file system.
640  */
641 static char *socket_name;
642 #endif
643 
644 /**
645  * Name of the first target of the first specific rule, used for default run.
646  */
647 static std::string first_target;
648 
649 /**
650  * Whether a short message should be displayed for each target.
651  */
652 static bool show_targets = true;
653 
654 /**
655  * Whether script commands are echoed.
656  */
657 static bool echo_scripts = false;
658 
659 static time_t now = time(NULL);
660 
661 static std::string working_dir;
662 
663 #ifndef WINDOWS
664 static volatile sig_atomic_t got_SIGCHLD = 0;
665 
666 static void sigchld_handler(int)
667 {
668  got_SIGCHLD = 1;
669 }
670 
671 static void sigint_handler(int)
672 {
673  // Child processes will receive the signal too, so just prevent
674  // new jobs from starting and wait for the running jobs to fail.
675  keep_going = false;
676 }
677 #endif
678 
679 struct log
680 {
681  bool active, open;
682  int depth;
683  log(): active(false), open(false), depth(0)
684  {
685  }
686  std::ostream &operator()()
687  {
688  if (open) std::cerr << std::endl;
689  assert(depth >= 0);
690  std::cerr << std::string(depth * 2, ' ');
691  open = false;
692  return std::cerr;
693  }
694  std::ostream &operator()(bool o)
695  {
696  if (o && open) std::cerr << std::endl;
697  if (!o) --depth;
698  assert(depth >= 0);
699  if (o || !open) std::cerr << std::string(depth * 2, ' ');
700  if (o) ++depth;
701  open = o;
702  return std::cerr;
703  }
704 };
705 
707 
709 {
712  {
713  }
715  {
716  if (debug.active && still_open) debug(false) << "done\n";
717  }
718 };
719 
720 #define DEBUG if (debug.active) debug()
721 #define DEBUG_open log_auto_close auto_close; if (debug.active) debug(true)
722 #define DEBUG_close if ((auto_close.still_open = false), debug.active) debug(false)
723 
724 /**
725  * Strong typedef for strings that need escaping.
726  * @note The string is stored as a reference, so the constructed object is
727  * meant to be immediately consumed.
728  */
730 {
731  std::string const &input;
732  escape_string(std::string const &s): input(s) {}
733 };
734 
735 /**
736  * Write the string in @a se to @a out if it does not contain any special
737  * characters, a quoted and escaped string otherwise.
738  */
739 static std::ostream &operator<<(std::ostream &out, escape_string const &se)
740 {
741  std::string const &s = se.input;
742  char const *quoted_char = ",: '";
743  char const *escaped_char = "\"\\$!";
744  bool need_quotes = false;
745  char *buf = NULL;
746  size_t len = s.length(), last = 0, j = 0;
747  for (size_t i = 0; i < len; ++i)
748  {
749  if (strchr(escaped_char, s[i]))
750  {
751  need_quotes = true;
752  if (!buf) buf = new char[len * 2];
753  memcpy(&buf[j], &s[last], i - last);
754  j += i - last;
755  buf[j++] = '\\';
756  buf[j++] = s[i];
757  last = i + 1;
758  }
759  if (!need_quotes && strchr(quoted_char, s[i]))
760  need_quotes = true;
761  }
762  if (!need_quotes) return out << s;
763  out << '"';
764  if (!buf) return out << s << '"';
765  out.write(buf, j);
766  out.write(&s[last], len - last);
767  delete[] buf;
768  return out << '"';
769 }
770 
771 /**
772  * @defgroup paths Path helpers
773  *
774  * @{
775  */
776 
777 /**
778  * Initialize #working_dir.
779  */
781 {
782  char buf[1024];
783  char *res = getcwd(buf, sizeof(buf));
784  if (!res)
785  {
786  perror("Failed to get working directory");
787  exit(EXIT_FAILURE);
788  }
789  working_dir = buf;
790 }
791 
792 /**
793  * Normalize an absolute path with respect to the working directory.
794  * Paths outside the working subtree are left unchanged.
795  */
796 static std::string normalize_abs(std::string const &s)
797 {
798  size_t l = working_dir.length();
799  if (s.compare(0, l, working_dir)) return s;
800  size_t ll = s.length();
801  if (ll == l) return ".";
802  if (s[l] != '/')
803  {
804  size_t pos = s.rfind('/', l);
805  assert(pos != std::string::npos);
806  return s.substr(pos + 1);
807  }
808  if (ll == l + 1) return ".";
809  return s.substr(l + 1);
810 }
811 
812 /**
813  * Normalize a target name.
814  */
815 static std::string normalize(std::string const &s)
816 {
817 #ifdef WINDOWS
818  char const *delim = "/\\";
819 #else
820  char delim = '/';
821 #endif
822  size_t prev = 0, len = s.length();
823  size_t pos = s.find_first_of(delim);
824  if (pos == std::string::npos) return s;
825  bool absolute = pos == 0;
826  string_list l;
827  for (;;)
828  {
829  if (pos != prev)
830  {
831  std::string n = s.substr(prev, pos - prev);
832  if (n == "..")
833  {
834  if (!l.empty()) l.pop_back();
835  else if (!absolute)
836  return normalize(working_dir + '/' + s);
837  }
838  else if (n != ".")
839  l.push_back(n);
840  }
841  ++pos;
842  if (pos >= len) break;
843  prev = pos;
844  pos = s.find_first_of(delim, prev);
845  if (pos == std::string::npos) pos = len;
846  }
847  string_list::const_iterator i = l.begin(), i_end = l.end();
848  if (i == i_end) return absolute ? "/" : ".";
849  std::string n;
850  if (absolute) n.push_back('/');
851  n.append(*i);
852  for (++i; i != i_end; ++i)
853  {
854  n.push_back('/');
855  n.append(*i);
856  }
857  if (absolute) return normalize_abs(n);
858  return n;
859 }
860 
861 /**
862  * Normalize the content of a list of targets.
863  */
865 {
866  for (string_list::iterator i = l.begin(),
867  i_end = l.end(); i != i_end; ++i)
868  {
869  *i = normalize(*i);
870  }
871 }
872 
873 /** @} */
874 
875 /**
876  * @defgroup lexer Lexer
877  *
878  * @{
879  */
880 
881 /**
882  * Skip spaces.
883  */
884 static void skip_spaces(std::istream &in)
885 {
886  char c;
887  while (strchr(" \t", (c = in.get()))) {}
888  if (in.good()) in.putback(c);
889 }
890 
891 /**
892  * Skip empty lines.
893  */
894 static void skip_empty(std::istream &in)
895 {
896  char c;
897  while (strchr("\r\n", (c = in.get()))) {}
898  if (in.good()) in.putback(c);
899 }
900 
901 /**
902  * Skip end of line. If @a multi is true, skip the following empty lines too.
903  * @return true if there was a line to end.
904  */
905 static bool skip_eol(std::istream &in, bool multi = false)
906 {
907  char c = in.get();
908  if (c == '\r') c = in.get();
909  if (c != '\n' && in.good()) in.putback(c);
910  if (c != '\n' && !in.eof()) return false;
911  if (multi) skip_empty(in);
912  return true;
913 }
914 
915 enum
916 {
918  Word = 1 << 1,
919  Colon = 1 << 2,
920  Equal = 1 << 3,
921  Dollarpar = 1 << 4,
922  Rightpar = 1 << 5,
923  Comma = 1 << 6,
924  Plusequal = 1 << 7,
925 };
926 
927 /**
928  * Skip spaces and peek at the next token.
929  * If it is one of @a mask, skip it (if it is not Word) and return it.
930  * @note For composite tokens allowed by @a mask, input characters might
931  * have been eaten even for an Unexpected result.
932  */
933 static int expect_token(std::istream &in, int mask)
934 {
935  while (true)
936  {
937  skip_spaces(in);
938  char c = in.peek();
939  if (!in.good()) return Unexpected;
940  int tok;
941  switch (c)
942  {
943  case '\r':
944  case '\n': return Unexpected;
945  case ':': tok = Colon; break;
946  case ',': tok = Comma; break;
947  case '=': tok = Equal; break;
948  case ')': tok = Rightpar; break;
949  case '$':
950  if (!(mask & Dollarpar)) return Unexpected;
951  in.ignore(1);
952  tok = Dollarpar;
953  if (in.peek() != '(') return Unexpected;
954  break;
955  case '+':
956  if (!(mask & Plusequal)) return Unexpected;
957  in.ignore(1);
958  tok = Plusequal;
959  if (in.peek() != '=') return Unexpected;
960  break;
961  case '\\':
962  in.ignore(1);
963  if (skip_eol(in)) continue;
964  in.putback('\\');
965  return mask & Word ? Word : Unexpected;
966  default:
967  return mask & Word ? Word : Unexpected;
968  }
969  if (!(tok & mask)) return Unexpected;
970  in.ignore(1);
971  return tok;
972  }
973 }
974 
975 /**
976  * Read a (possibly quoted) word.
977  */
978 static std::string read_word(std::istream &in)
979 {
980  int c = in.get();
981  std::string res;
982  if (!in.good()) return res;
983  char const *separators = " \t\r\n:$(),=+\"";
984  bool quoted = c == '"';
985  if (!quoted)
986  {
987  if (strchr(separators, c))
988  {
989  in.putback(c);
990  return res;
991  }
992  res += c;
993  }
994  while (true)
995  {
996  c = in.get();
997  if (!in.good()) return res;
998  if (quoted)
999  {
1000  if (c == '\\')
1001  res += in.get();
1002  else if (c == '"')
1003  return res;
1004  else
1005  res += c;
1006  }
1007  else
1008  {
1009  if (strchr(separators, c))
1010  {
1011  in.putback(c);
1012  return res;
1013  }
1014  res += c;
1015  }
1016  }
1017 }
1018 
1019 /** @} */
1020 
1021 /**
1022  * @defgroup stream Token streams
1023  *
1024  * @{
1025  */
1026 
1027 /**
1028  * Possible results from word producers.
1029  */
1031 {
1035 };
1036 
1037 /**
1038  * Interface for word producers.
1039  */
1041 {
1042  virtual ~generator() {}
1043  virtual input_status next(std::string &) = 0;
1044 };
1045 
1046 /**
1047  * Generator for the words of a variable.
1048  */
1050 {
1051  std::string name;
1052  string_list::const_iterator cur1, end1;
1053  assign_list::const_iterator cur2, end2;
1054  variable_generator(std::string const &, assign_list const *);
1055  input_status next(std::string &);
1056 };
1057 
1059  assign_list const *local_variables): name(n)
1060 {
1061  bool append = true;
1062  if (local_variables)
1063  {
1064  // Set cur2 to the last variable overwriter, if any.
1065  cur2 = local_variables->begin();
1066  end2 = local_variables->end();
1067  for (assign_list::const_iterator i = cur2; i != end2; ++i)
1068  {
1069  if (i->name == name && !i->append)
1070  {
1071  append = false;
1072  cur2 = i;
1073  }
1074  }
1075  }
1076  else
1077  {
1078  static assign_list dummy;
1079  cur2 = dummy.begin();
1080  end2 = dummy.end();
1081  }
1082  static string_list dummy;
1083  cur1 = dummy.begin();
1084  end1 = dummy.end();
1085  if (append)
1086  {
1087  variable_map::const_iterator i = variables.find(name);
1088  if (i == variables.end()) return;
1089  cur1 = i->second.begin();
1090  end1 = i->second.end();
1091  }
1092 }
1093 
1095 {
1096  restart:
1097  if (cur1 != end1)
1098  {
1099  res = *cur1;
1100  ++cur1;
1101  return Success;
1102  }
1103  while (cur2 != end2)
1104  {
1105  if (cur2->name == name)
1106  {
1107  cur1 = cur2->value.begin();
1108  end1 = cur2->value.end();
1109  ++cur2;
1110  goto restart;
1111  }
1112  ++cur2;
1113  }
1114  return Eof;
1115 }
1116 
1117 /**
1118  * Generator for the words of an input stream.
1119  */
1121 {
1122  std::istream &in;
1126  input_generator(std::istream &i, assign_list const *lv, bool e = false)
1127  : in(i), nested(NULL), local_variables(lv), earliest_exit(e), done(false) {}
1128  input_status next(std::string &);
1129  ~input_generator() { assert(!nested); }
1130 };
1131 
1132 static generator *get_function(input_generator const &, std::string const &);
1133 
1135 {
1136  if (nested)
1137  {
1138  restart:
1139  input_status s = nested->next(res);
1140  if (s == Success) return Success;
1141  delete nested;
1142  nested = NULL;
1143  if (s == SyntaxError) return SyntaxError;
1144  }
1145  if (done) return Eof;
1146  if (earliest_exit) done = true;
1147  switch (expect_token(in, Word | Dollarpar))
1148  {
1149  case Word:
1150  res = read_word(in);
1151  return Success;
1152  case Dollarpar:
1153  {
1154  std::string name = read_word(in);
1155  if (name.empty()) return SyntaxError;
1156  if (expect_token(in, Rightpar))
1158  else
1159  {
1160  nested = get_function(*this, name);
1161  if (!nested) return SyntaxError;
1162  }
1163  goto restart;
1164  }
1165  default:
1166  return Eof;
1167  }
1168 }
1169 
1170 /**
1171  * Read a list of words from an input generator.
1172  * @return false if a syntax error was encountered.
1173  */
1175 {
1176  while (true)
1177  {
1178  res.push_back(std::string());
1179  input_status s = in.next(res.back());
1180  if (s == Success) continue;
1181  res.pop_back();
1182  return s == Eof;
1183  }
1184 }
1185 
1186 static bool read_words(std::istream &in, string_list &res)
1187 {
1188  input_generator gen(in, NULL);
1189  return read_words(gen, res);
1190 }
1191 
1192 /**
1193  * Generator for the result of function addprefix.
1194  */
1196 {
1199  string_list::const_iterator prei;
1200  size_t prej, prel;
1201  std::string suf;
1202  addprefix_generator(input_generator const &, bool &);
1203  input_status next(std::string &);
1204 };
1205 
1207  : gen(top.in, top.local_variables)
1208 {
1209  if (!read_words(gen, pre)) return;
1210  if (!expect_token(gen.in, Comma)) return;
1211  prej = 0;
1212  prel = pre.size();
1213  ok = true;
1214 }
1215 
1217 {
1218  if (prej)
1219  {
1220  produce:
1221  if (prej == prel)
1222  {
1223  res = *prei + suf;
1224  prej = 0;
1225  }
1226  else
1227  {
1228  res = *prei++;
1229  ++prej;
1230  }
1231  return Success;
1232  }
1233  switch (gen.next(res))
1234  {
1235  case Success:
1236  if (!prel) return Success;
1237  prei = pre.begin();
1238  prej = 1;
1239  suf = res;
1240  goto produce;
1241  case Eof:
1242  return expect_token(gen.in, Rightpar) ? Eof : SyntaxError;
1243  default:
1244  return SyntaxError;
1245  }
1246 }
1247 
1248 /**
1249  * Generator for the result of function addsuffix.
1250  */
1252 {
1255  string_list::const_iterator sufi;
1256  size_t sufj, sufl;
1257  std::string pre;
1258  addsuffix_generator(input_generator const &, bool &);
1259  input_status next(std::string &);
1260 };
1261 
1263  : gen(top.in, top.local_variables)
1264 {
1265  if (!read_words(gen, suf)) return;
1266  if (!expect_token(gen.in, Comma)) return;
1267  sufj = 0;
1268  sufl = suf.size();
1269  ok = true;
1270 }
1271 
1273 {
1274  if (sufj)
1275  {
1276  if (sufj != sufl)
1277  {
1278  res = *sufi++;
1279  ++sufj;
1280  return Success;
1281  }
1282  sufj = 0;
1283  }
1284  switch (gen.next(res))
1285  {
1286  case Success:
1287  if (!sufl) return Success;
1288  sufi = suf.begin();
1289  sufj = 1;
1290  res += *sufi++;
1291  return Success;
1292  case Eof:
1293  return expect_token(gen.in, Rightpar) ? Eof : SyntaxError;
1294  default:
1295  return SyntaxError;
1296  }
1297 }
1298 
1299 /**
1300  * Return a generator for function @a name.
1301  */
1302 generator *get_function(input_generator const &in, std::string const &name)
1303 {
1304  skip_spaces(in.in);
1305  generator *g = NULL;
1306  bool ok = false;
1307  if (name == "addprefix") g = new addprefix_generator(in, ok);
1308  else if (name == "addsuffix") g = new addsuffix_generator(in, ok);
1309  if (!g || ok) return g;
1310  delete g;
1311  return NULL;
1312 }
1313 
1314 /** @} */
1315 
1316 /**
1317  * @defgroup database Dependency database
1318  *
1319  * @{
1320  */
1321 
1322 /**
1323  * Load dependencies from @a in.
1324  */
1325 static void load_dependencies(std::istream &in)
1326 {
1327  if (false)
1328  {
1329  error:
1330  std::cerr << "Failed to load database" << std::endl;
1331  exit(EXIT_FAILURE);
1332  }
1333 
1334  while (!in.eof())
1335  {
1336  string_list targets;
1337  if (!read_words(in, targets)) goto error;
1338  if (in.eof()) return;
1339  if (targets.empty()) goto error;
1340  DEBUG << "reading dependencies of target " << targets.front() << std::endl;
1341  if (in.get() != ':') goto error;
1343  dep->targets = targets;
1344  string_list deps;
1345  if (!read_words(in, deps)) goto error;
1346  dep->deps.insert(deps.begin(), deps.end());
1347  for (string_list::const_iterator i = targets.begin(),
1348  i_end = targets.end(); i != i_end; ++i)
1349  {
1350  dependencies[*i] = dep;
1351  }
1352  skip_empty(in);
1353  }
1354 }
1355 
1356 /**
1357  * Load known dependencies from file <tt>.remake</tt>.
1358  */
1359 static void load_dependencies()
1360 {
1361  DEBUG_open << "Loading database... ";
1362  std::ifstream in(".remake");
1363  if (!in.good())
1364  {
1365  DEBUG_close << "not found\n";
1366  return;
1367  }
1368  load_dependencies(in);
1369 }
1370 
1371 
1372 /**
1373  * Save all the dependencies in file <tt>.remake</tt>.
1374  */
1375 static void save_dependencies()
1376 {
1377  DEBUG_open << "Saving database... ";
1378  std::ofstream db(".remake");
1379  while (!dependencies.empty())
1380  {
1381  ref_ptr<dependency_t> dep = dependencies.begin()->second;
1382  for (string_list::const_iterator i = dep->targets.begin(),
1383  i_end = dep->targets.end(); i != i_end; ++i)
1384  {
1385  db << escape_string(*i) << ' ';
1386  dependencies.erase(*i);
1387  }
1388  db << ':';
1389  for (string_set::const_iterator i = dep->deps.begin(),
1390  i_end = dep->deps.end(); i != i_end; ++i)
1391  {
1392  db << ' ' << escape_string(*i);
1393  }
1394  db << std::endl;
1395  }
1396 }
1397 
1398 /** @} */
1399 
1400 /**
1401  * @defgroup parser Rule parser
1402  *
1403  * @{
1404  */
1405 
1406 /**
1407  * Register a specific rule with an empty script:
1408  *
1409  * - Check that none of the targets already has an associated rule with a
1410  * nonempty script.
1411  * - Create a new rule with a single target for each target, if needed.
1412  * - Add the prerequisites of @a rule to all these associated rules.
1413  */
1414 static void register_transparent_rule(rule_t const &rule, string_list const &targets)
1415 {
1416  assert(rule.script.empty());
1417  for (string_list::const_iterator i = targets.begin(),
1418  i_end = targets.end(); i != i_end; ++i)
1419  {
1420  std::pair<rule_map::iterator, bool> j =
1421  specific_rules.insert(std::make_pair(*i, ref_ptr<rule_t>()));
1422  ref_ptr<rule_t> &r = j.first->second;
1423  if (j.second)
1424  {
1425  r = ref_ptr<rule_t>(rule);
1426  r->targets = string_list(1, *i);
1427  continue;
1428  }
1429  if (!r->script.empty())
1430  {
1431  std::cerr << "Failed to load rules: " << *i
1432  << " cannot be the target of several rules" << std::endl;
1433  exit(EXIT_FAILURE);
1434  }
1435  assert(r->targets.size() == 1 && r->targets.front() == *i);
1436  r->deps.insert(r->deps.end(), rule.deps.begin(), rule.deps.end());
1437  r->vars.insert(r->vars.end(), rule.vars.begin(), rule.vars.end());
1438  }
1439 
1440  for (string_list::const_iterator i = targets.begin(),
1441  i_end = targets.end(); i != i_end; ++i)
1442  {
1444  if (dep->targets.empty()) dep->targets.push_back(*i);
1445  dep->deps.insert(rule.deps.begin(), rule.deps.end());
1446  }
1447 }
1448 
1449 /**
1450  * Register a specific rule with a nonempty script:
1451  *
1452  * - Check that none of the targets already has an associated rule.
1453  * - Create a single shared rule and associate it to all the targets.
1454  * - Merge the prerequisites of all the targets into a single set and
1455  * add the prerequisites of the rule to it. (The preexisting
1456  * prerequisites, if any, come from a previous run.)
1457  */
1458 static void register_scripted_rule(rule_t const &rule)
1459 {
1460  ref_ptr<rule_t> r(rule);
1461  for (string_list::const_iterator i = rule.targets.begin(),
1462  i_end = rule.targets.end(); i != i_end; ++i)
1463  {
1464  std::pair<rule_map::iterator, bool> j =
1465  specific_rules.insert(std::make_pair(*i, r));
1466  if (j.second) continue;
1467  std::cerr << "Failed to load rules: " << *i
1468  << " cannot be the target of several rules" << std::endl;
1469  exit(EXIT_FAILURE);
1470  }
1471 
1473  dep->targets = rule.targets;
1474  dep->deps.insert(rule.deps.begin(), rule.deps.end());
1475  for (string_list::const_iterator i = rule.targets.begin(),
1476  i_end = rule.targets.end(); i != i_end; ++i)
1477  {
1479  dep->deps.insert(d->deps.begin(), d->deps.end());
1480  d = dep;
1481  }
1482 }
1483 
1484 /**
1485  * Read a rule starting with target @a first, if nonempty.
1486  * Store into #generic_rules or #specific_rules depending on its genericity.
1487  */
1488 static void load_rule(std::istream &in, std::string const &first)
1489 {
1490  DEBUG_open << "Reading rule for target " << first << "... ";
1491  if (false)
1492  {
1493  error:
1494  DEBUG_close << "failed\n";
1495  std::cerr << "Failed to load rules: syntax error" << std::endl;
1496  exit(EXIT_FAILURE);
1497  }
1498  rule_t rule;
1499 
1500  // Read targets and check genericity.
1501  string_list targets;
1502  if (!read_words(in, targets)) goto error;
1503  if (!first.empty()) targets.push_front(first);
1504  else if (targets.empty()) goto error;
1505  else DEBUG << "actual target: " << targets.front() << std::endl;
1506  bool generic = false;
1507  normalize_list(targets);
1508  for (string_list::const_iterator i = targets.begin(),
1509  i_end = targets.end(); i != i_end; ++i)
1510  {
1511  if (i->empty()) goto error;
1512  if ((i->find('%') != std::string::npos) != generic)
1513  {
1514  if (i == targets.begin()) generic = true;
1515  else goto error;
1516  }
1517  }
1518  std::swap(rule.targets, targets);
1519  skip_spaces(in);
1520  if (in.get() != ':') goto error;
1521 
1522  bool assignment = false;
1523 
1524  // Read dependencies.
1525  if (expect_token(in, Word))
1526  {
1527  std::string d = read_word(in);
1528  if (int tok = expect_token(in, Equal | Plusequal))
1529  {
1530  rule.vars.push_back(assign_t());
1531  string_list v;
1532  if (!read_words(in, v)) goto error;
1533  assign_t &a = rule.vars.back();
1534  a.name = d;
1535  a.append = tok == Plusequal;
1536  a.value.swap(v);
1537  assignment = true;
1538  }
1539  else
1540  {
1541  string_list v;
1542  if (!read_words(in, v)) goto error;
1543  v.push_front(d);
1544  normalize_list(v);
1545  rule.deps.swap(v);
1546  }
1547  }
1548  else
1549  {
1550  string_list v;
1551  if (!read_words(in, v)) goto error;
1552  normalize_list(v);
1553  rule.deps.swap(v);
1554  }
1555  skip_spaces(in);
1556  if (!skip_eol(in, true)) goto error;
1557 
1558  // Read script.
1559  std::ostringstream buf;
1560  while (true)
1561  {
1562  char c = in.get();
1563  if (!in.good()) break;
1564  if (c == '\t' || c == ' ')
1565  {
1566  in.get(*buf.rdbuf());
1567  if (in.fail() && !in.eof()) in.clear();
1568  }
1569  else if (c == '\r' || c == '\n')
1570  buf << c;
1571  else
1572  {
1573  in.putback(c);
1574  break;
1575  }
1576  }
1577  rule.script = buf.str();
1578 
1579  // Add generic rules to the correct set.
1580  if (generic)
1581  {
1582  if (assignment) goto error;
1583  generic_rules.push_back(rule);
1584  return;
1585  }
1586 
1587  if (!rule.script.empty())
1588  {
1589  if (assignment) goto error;
1590  register_scripted_rule(rule);
1591  }
1592  else
1593  {
1594  // Swap away the targets to avoid costly copies when registering.
1595  string_list targets;
1596  std::swap(rule.targets, targets);
1597  register_transparent_rule(rule, targets);
1598  std::swap(rule.targets, targets);
1599  }
1600 
1601  // If there is no default target yet, mark it as such.
1602  if (first_target.empty())
1603  first_target = rule.targets.front();
1604 }
1605 
1606 /**
1607  * Load rules from @a remakefile.
1608  * If some rules have dependencies and non-generic targets, add these
1609  * dependencies to the targets.
1610  */
1611 static void load_rules(std::string const &remakefile)
1612 {
1613  DEBUG_open << "Loading rules... ";
1614  if (false)
1615  {
1616  error:
1617  std::cerr << "Failed to load rules: syntax error" << std::endl;
1618  exit(EXIT_FAILURE);
1619  }
1620  std::ifstream in(remakefile.c_str());
1621  if (!in.good())
1622  {
1623  std::cerr << "Failed to load rules: no Remakefile found" << std::endl;
1624  exit(EXIT_FAILURE);
1625  }
1626  skip_empty(in);
1627 
1628  // Read rules
1629  while (in.good())
1630  {
1631  char c = in.peek();
1632  if (c == '#')
1633  {
1634  while (in.get() != '\n') {}
1635  skip_empty(in);
1636  continue;
1637  }
1638  if (c == ' ' || c == '\t') goto error;
1639  if (expect_token(in, Word))
1640  {
1641  std::string name = read_word(in);
1642  if (name.empty()) goto error;
1643  if (int tok = expect_token(in, Equal | Plusequal))
1644  {
1645  DEBUG << "Assignment to variable " << name << std::endl;
1646  string_list value;
1647  if (!read_words(in, value)) goto error;
1648  string_list &dest = variables[name];
1649  if (tok == Equal) dest.swap(value);
1650  else dest.splice(dest.end(), value);
1651  if (!skip_eol(in, true)) goto error;
1652  }
1653  else load_rule(in, name);
1654  }
1655  else load_rule(in, std::string());
1656  }
1657 }
1658 
1659 /** @} */
1660 
1661 /**
1662  * @defgroup rules Rule resolution
1663  *
1664  * @{
1665  */
1666 
1667 /**
1668  * Substitute a pattern into a list of strings.
1669  */
1670 static void substitute_pattern(std::string const &pat, string_list const &src, string_list &dst)
1671 {
1672  for (string_list::const_iterator i = src.begin(),
1673  i_end = src.end(); i != i_end; ++i)
1674  {
1675  size_t pos = i->find('%');
1676  if (pos == std::string::npos)dst.push_back(*i);
1677  else dst.push_back(i->substr(0, pos) + pat + i->substr(pos + 1));
1678  }
1679 }
1680 
1681 /**
1682  * Find a generic rule matching @a target:
1683  * - the one leading to shorter matches has priority,
1684  * - among equivalent rules, the earliest one has priority.
1685  */
1686 static rule_t find_generic_rule(std::string const &target)
1687 {
1688  size_t tlen = target.length(), plen = tlen + 1;
1689  rule_t rule;
1690  for (rule_list::const_iterator i = generic_rules.begin(),
1691  i_end = generic_rules.end(); i != i_end; ++i)
1692  {
1693  for (string_list::const_iterator j = i->targets.begin(),
1694  j_end = i->targets.end(); j != j_end; ++j)
1695  {
1696  size_t len = j->length();
1697  if (tlen < len) continue;
1698  if (plen <= tlen - (len - 1)) continue;
1699  size_t pos = j->find('%');
1700  if (pos == std::string::npos) continue;
1701  size_t len2 = len - (pos + 1);
1702  if (j->compare(0, pos, target, 0, pos) ||
1703  j->compare(pos + 1, len2, target, tlen - len2, len2))
1704  continue;
1705  plen = tlen - (len - 1);
1706  rule = rule_t();
1707  rule.stem = target.substr(pos, plen);
1708  rule.script = i->script;
1709  substitute_pattern(rule.stem, i->targets, rule.targets);
1710  substitute_pattern(rule.stem, i->deps, rule.deps);
1711  break;
1712  }
1713  }
1714  return rule;
1715 }
1716 
1717 /**
1718  * Find a specific rule matching @a target. Return a generic one otherwise.
1719  * If there is both a specific rule with an empty script and a generic rule, the
1720  * generic one is returned after adding the dependencies of the specific one.
1721  */
1722 static rule_t find_rule(std::string const &target)
1723 {
1724  rule_map::const_iterator i = specific_rules.find(target),
1725  i_end = specific_rules.end();
1726  // If there is a specific rule with a script, return it.
1727  if (i != i_end && !i->second->script.empty()) return *i->second;
1728  rule_t grule = find_generic_rule(target);
1729  // If there is no generic rule, return the specific rule (no script), if any.
1730  if (grule.targets.empty())
1731  {
1732  if (i != i_end) return *i->second;
1733  return grule;
1734  }
1735  // Optimize the lookup when there is only one target (already looked up).
1736  if (grule.targets.size() == 1)
1737  {
1738  if (i == i_end) return grule;
1739  grule.deps.insert(grule.deps.end(),
1740  i->second->deps.begin(), i->second->deps.end());
1741  grule.vars.insert(grule.vars.end(),
1742  i->second->vars.begin(), i->second->vars.end());
1743  return grule;
1744  }
1745  // Add the dependencies of the specific rules of every target to the
1746  // generic rule. If any of those rules has a nonempty script, error out.
1747  for (string_list::const_iterator j = grule.targets.begin(),
1748  j_end = grule.targets.end(); j != j_end; ++j)
1749  {
1750  i = specific_rules.find(*j);
1751  if (i == i_end) continue;
1752  if (!i->second->script.empty()) return rule_t();
1753  grule.deps.insert(grule.deps.end(),
1754  i->second->deps.begin(), i->second->deps.end());
1755  grule.vars.insert(grule.vars.end(),
1756  i->second->vars.begin(), i->second->vars.end());
1757  }
1758  return grule;
1759 }
1760 
1761 /** @} */
1762 
1763 /**
1764  * @defgroup status Target status
1765  *
1766  * @{
1767  */
1768 
1769 /**
1770  * Compute and memoize the status of @a target:
1771  * - if the file does not exist, the target is obsolete,
1772  * - if any dependency is obsolete or younger than the file, it is obsolete,
1773  * - otherwise it is up-to-date.
1774  *
1775  * @note For rules with multiple targets, all the targets share the same
1776  * status. (If one is obsolete, they all are.) The second rule above
1777  * is modified in that case: the latest target is chosen, not the oldest!
1778  */
1779 static status_t const &get_status(std::string const &target)
1780 {
1781  std::pair<status_map::iterator,bool> i =
1782  status.insert(std::make_pair(target, status_t()));
1783  status_t &ts = i.first->second;
1784  if (!i.second) return ts;
1785  DEBUG_open << "Checking status of " << target << "... ";
1786  dependency_map::const_iterator j = dependencies.find(target);
1787  if (j == dependencies.end())
1788  {
1789  struct stat s;
1790  if (stat(target.c_str(), &s) != 0)
1791  {
1792  DEBUG_close << "missing\n";
1793  ts.status = Todo;
1794  ts.last = 0;
1795  return ts;
1796  }
1797  DEBUG_close << "up-to-date\n";
1798  ts.status = Uptodate;
1799  ts.last = s.st_mtime;
1800  return ts;
1801  }
1802  dependency_t const &dep = *j->second;
1803  status_e st = Uptodate;
1804  time_t latest = 0;
1805  for (string_list::const_iterator k = dep.targets.begin(),
1806  k_end = dep.targets.end(); k != k_end; ++k)
1807  {
1808  struct stat s;
1809  if (stat(k->c_str(), &s) != 0)
1810  {
1811  if (st == Uptodate) DEBUG_close << *k << " missing\n";
1812  s.st_mtime = 0;
1813  st = Todo;
1814  }
1815  status[*k].last = s.st_mtime;
1816  if (s.st_mtime > latest) latest = s.st_mtime;
1817  }
1818  if (st == Todo) goto update;
1819  for (string_set::const_iterator k = dep.deps.begin(),
1820  k_end = dep.deps.end(); k != k_end; ++k)
1821  {
1822  status_t const &ts_ = get_status(*k);
1823  if (latest < ts_.last)
1824  {
1825  DEBUG_close << "older than " << *k << std::endl;
1826  st = Todo;
1827  goto update;
1828  }
1829  if (ts_.status == Uptodate) continue;
1830  if (st == Uptodate)
1831  DEBUG << "obsolete dependency " << *k << std::endl;
1832  st = Recheck;
1833  }
1834  if (st == Uptodate) DEBUG_close << "all siblings up-to-date\n";
1835  update:
1836  for (string_list::const_iterator k = dep.targets.begin(),
1837  k_end = dep.targets.end(); k != k_end; ++k)
1838  {
1839  status[*k].status = st;
1840  }
1841  return ts;
1842 }
1843 
1844 /**
1845  * Change the status of @a target to #Remade or #Uptodate depending on whether
1846  * its modification time changed.
1847  */
1848 static void update_status(std::string const &target)
1849 {
1850  DEBUG_open << "Rechecking status of " << target << "... ";
1851  status_map::iterator i = status.find(target);
1852  assert(i != status.end());
1853  status_t &ts = i->second;
1854  ts.status = Remade;
1855  if (ts.last >= now)
1856  {
1857  DEBUG_close << "possibly remade\n";
1858  return;
1859  }
1860  struct stat s;
1861  if (stat(target.c_str(), &s) != 0)
1862  {
1863  DEBUG_close << "missing\n";
1864  ts.last = 0;
1865  }
1866  else if (s.st_mtime != ts.last)
1867  {
1868  DEBUG_close << "remade\n";
1869  ts.last = s.st_mtime;
1870  }
1871  else
1872  {
1873  DEBUG_close << "unchanged\n";
1874  ts.status = Uptodate;
1875  }
1876 }
1877 
1878 /**
1879  * Check whether all the prerequisites of @a target ended being up-to-date.
1880  */
1881 static bool still_need_rebuild(std::string const &target)
1882 {
1883  DEBUG_open << "Rechecking obsoleteness of " << target << "... ";
1884  status_map::const_iterator i = status.find(target);
1885  assert(i != status.end());
1886  if (i->second.status != Recheck) return true;
1887  dependency_map::const_iterator j = dependencies.find(target);
1888  assert(j != dependencies.end());
1889  dependency_t const &dep = *j->second;
1890  for (string_set::const_iterator k = dep.deps.begin(),
1891  k_end = dep.deps.end(); k != k_end; ++k)
1892  {
1893  if (status[*k].status != Uptodate) return true;
1894  }
1895  for (string_list::const_iterator k = dep.targets.begin(),
1896  k_end = dep.targets.end(); k != k_end; ++k)
1897  {
1898  status[*k].status = Uptodate;
1899  }
1900  DEBUG_close << "no longer obsolete\n";
1901  return false;
1902 }
1903 
1904 /** @} */
1905 
1906 /**
1907  * @defgroup server Server
1908  *
1909  * @{
1910  */
1911 
1912 /**
1913  * Handle job completion.
1914  */
1915 static void complete_job(int job_id, bool success)
1916 {
1917  DEBUG_open << "Completing job " << job_id << "... ";
1918  job_targets_map::iterator i = job_targets.find(job_id);
1919  assert(i != job_targets.end());
1920  string_list const &targets = i->second;
1921  if (success)
1922  {
1923  for (string_list::const_iterator j = targets.begin(),
1924  j_end = targets.end(); j != j_end; ++j)
1925  {
1926  update_status(*j);
1927  }
1928  }
1929  else
1930  {
1931  DEBUG_close << "failed\n";
1932  std::cerr << "Failed to build";
1933  for (string_list::const_iterator j = targets.begin(),
1934  j_end = targets.end(); j != j_end; ++j)
1935  {
1936  status[*j].status = Failed;
1937  std::cerr << ' ' << *j;
1938  remove(j->c_str());
1939  }
1940  std::cerr << std::endl;
1941  }
1942  job_targets.erase(i);
1943 }
1944 
1945 /**
1946  * Return the script obtained by substituting variables.
1947  */
1948 static std::string prepare_script(rule_t const &rule)
1949 {
1950  std::string const &s = rule.script;
1951  std::istringstream in(s);
1952  std::ostringstream out;
1953  size_t len = s.size();
1954 
1955  while (!in.eof())
1956  {
1957  size_t pos = in.tellg(), p = s.find('$', pos);
1958  if (p == std::string::npos || p == len - 1) p = len;
1959  out.write(&s[pos], p - pos);
1960  if (p == len) break;
1961  ++p;
1962  switch (s[p])
1963  {
1964  case '$':
1965  out << '$';
1966  in.seekg(p + 1);
1967  break;
1968  case '<':
1969  if (!rule.deps.empty())
1970  out << rule.deps.front();
1971  in.seekg(p + 1);
1972  break;
1973  case '^':
1974  {
1975  bool first = true;
1976  for (string_list::const_iterator i = rule.deps.begin(),
1977  i_end = rule.deps.end(); i != i_end; ++i)
1978  {
1979  if (first) first = false;
1980  else out << ' ';
1981  out << *i;
1982  }
1983  in.seekg(p + 1);
1984  break;
1985  }
1986  case '@':
1987  assert(!rule.targets.empty());
1988  out << rule.targets.front();
1989  in.seekg(p + 1);
1990  break;
1991  case '*':
1992  out << rule.stem;
1993  in.seekg(p + 1);
1994  break;
1995  case '(':
1996  {
1997  in.seekg(p - 1);
1998  bool first = true;
1999  input_generator gen(in, &rule.vars, true);
2000  while (true)
2001  {
2002  std::string w;
2003  input_status s = gen.next(w);
2004  if (s == SyntaxError)
2005  {
2006  // TODO
2007  return "false";
2008  }
2009  if (s == Eof) break;
2010  if (first) first = false;
2011  else out << ' ';
2012  out << w;
2013  }
2014  break;
2015  }
2016  default:
2017  // Let dollars followed by an unrecognized character
2018  // go through. This differs from Make, which would
2019  // use a one-letter variable.
2020  out << '$';
2021  in.seekg(p);
2022  }
2023  }
2024 
2025  return out.str();
2026 }
2027 
2028 /**
2029  * Execute the script from @a rule.
2030  */
2031 static bool run_script(int job_id, rule_t const &rule)
2032 {
2033  if (show_targets)
2034  {
2035  std::cout << "Building";
2036  for (string_list::const_iterator i = rule.targets.begin(),
2037  i_end = rule.targets.end(); i != i_end; ++i)
2038  {
2039  std::cout << ' ' << *i;
2040  }
2041  std::cout << std::endl;
2042  }
2043 
2045  dep->targets = rule.targets;
2046  dep->deps.insert(rule.deps.begin(), rule.deps.end());
2047  for (string_list::const_iterator i = rule.targets.begin(),
2048  i_end = rule.targets.end(); i != i_end; ++i)
2049  {
2050  dependencies[*i] = dep;
2051  }
2052 
2053  std::string script = prepare_script(rule);
2054 
2055  std::ostringstream job_id_buf;
2056  job_id_buf << job_id;
2057  std::string job_id_ = job_id_buf.str();
2058 
2059  DEBUG_open << "Starting script for job " << job_id << "... ";
2060  if (false)
2061  {
2062  error:
2063  DEBUG_close << "failed\n";
2064  complete_job(job_id, false);
2065  return false;
2066  }
2067 
2068 #ifdef WINDOWS
2069  HANDLE pfd[2];
2070  if (false)
2071  {
2072  error2:
2073  CloseHandle(pfd[0]);
2074  CloseHandle(pfd[1]);
2075  goto error;
2076  }
2077  if (!CreatePipe(&pfd[0], &pfd[1], NULL, 0))
2078  goto error;
2079  if (!SetHandleInformation(pfd[0], HANDLE_FLAG_INHERIT, HANDLE_FLAG_INHERIT))
2080  goto error2;
2081  STARTUPINFO si;
2082  ZeroMemory(&si, sizeof(STARTUPINFO));
2083  si.cb = sizeof(STARTUPINFO);
2084  si.hStdError = GetStdHandle(STD_ERROR_HANDLE);
2085  si.hStdOutput = GetStdHandle(STD_OUTPUT_HANDLE);
2086  si.hStdInput = pfd[0];
2087  si.dwFlags |= STARTF_USESTDHANDLES;
2088  PROCESS_INFORMATION pi;
2089  ZeroMemory(&pi, sizeof(PROCESS_INFORMATION));
2090  if (!SetEnvironmentVariable("REMAKE_JOB_ID", job_id_.c_str()))
2091  goto error2;
2092  char const *argv = echo_scripts ? "SH.EXE -e -s -v" : "SH.EXE -e -s";
2093  if (!CreateProcess(NULL, (char *)argv, NULL, NULL,
2094  true, 0, NULL, NULL, &si, &pi))
2095  {
2096  goto error2;
2097  }
2098  CloseHandle(pi.hThread);
2099  DWORD len = script.length(), wlen;
2100  if (!WriteFile(pfd[1], script.c_str(), len, &wlen, NULL) || wlen < len)
2101  std::cerr << "Unexpected failure while sending script to shell" << std::endl;
2102  CloseHandle(pfd[0]);
2103  CloseHandle(pfd[1]);
2104  ++running_jobs;
2105  job_pids[pi.hProcess] = job_id;
2106  return true;
2107 #else
2108  int pfd[2];
2109  if (false)
2110  {
2111  error2:
2112  close(pfd[0]);
2113  close(pfd[1]);
2114  goto error;
2115  }
2116  if (pipe(pfd) == -1)
2117  goto error;
2118  if (setenv("REMAKE_JOB_ID", job_id_.c_str(), 1))
2119  goto error2;
2120  if (pid_t pid = vfork())
2121  {
2122  if (pid == -1) goto error2;
2123  ssize_t len = script.length();
2124  if (write(pfd[1], script.c_str(), len) < len)
2125  std::cerr << "Unexpected failure while sending script to shell" << std::endl;
2126  close(pfd[0]);
2127  close(pfd[1]);
2128  ++running_jobs;
2129  job_pids[pid] = job_id;
2130  return true;
2131  }
2132  // Child process starts here. Notice the use of vfork above.
2133  char const *argv[5] = { "sh", "-e", "-s", NULL, NULL };
2134  if (echo_scripts) argv[3] = "-v";
2135  if (pfd[0] != 0)
2136  {
2137  dup2(pfd[0], 0);
2138  close(pfd[0]);
2139  }
2140  close(pfd[1]);
2141  execve("/bin/sh", (char **)argv, environ);
2142  _exit(EXIT_FAILURE);
2143 #endif
2144 }
2145 
2146 /**
2147  * Create a job for @a target according to the loaded rules.
2148  * Mark all the targets from the rule as running and reset their dependencies.
2149  * If the rule has dependencies, create a new client to build them just
2150  * before @a current, and change @a current so that it points to it.
2151  */
2152 static bool start(std::string const &target, client_list::iterator &current)
2153 {
2154  DEBUG_open << "Starting job " << job_counter << " for " << target << "... ";
2155  rule_t rule = find_rule(target);
2156  if (rule.targets.empty())
2157  {
2158  status[target].status = Failed;
2159  DEBUG_close << "failed\n";
2160  std::cerr << "No rule for building " << target << std::endl;
2161  return false;
2162  }
2163  for (string_list::const_iterator i = rule.targets.begin(),
2164  i_end = rule.targets.end(); i != i_end; ++i)
2165  {
2166  status[*i].status = Running;
2167  }
2168  int job_id = job_counter++;
2169  job_targets[job_id] = rule.targets;
2170  if (!rule.deps.empty())
2171  {
2172  current = clients.insert(current, client_t());
2173  current->job_id = job_id;
2174  current->pending = rule.deps;
2175  current->delayed = new rule_t(rule);
2176  return true;
2177  }
2178  return run_script(job_id, rule);
2179 }
2180 
2181 /**
2182  * Send a reply to a client then remove it.
2183  * If the client was a dependency client, start the actual script.
2184  */
2185 static void complete_request(client_t &client, bool success)
2186 {
2187  DEBUG_open << "Completing request from client of job " << client.job_id << "... ";
2188  if (client.delayed)
2189  {
2190  assert(client.socket == INVALID_SOCKET);
2191  if (success)
2192  {
2193  if (still_need_rebuild(client.delayed->targets.front()))
2194  run_script(client.job_id, *client.delayed);
2195  else complete_job(client.job_id, true);
2196  }
2197  else complete_job(client.job_id, false);
2198  delete client.delayed;
2199  }
2200  else if (client.socket != INVALID_SOCKET)
2201  {
2202  char res = success ? 1 : 0;
2203  send(client.socket, &res, 1, MSG_NOSIGNAL);
2204  #ifdef WINDOWS
2205  closesocket(client.socket);
2206  #else
2207  close(client.socket);
2208  #endif
2209  --waiting_jobs;
2210  }
2211 
2212  if (client.job_id < 0 && !success) build_failure = true;
2213 }
2214 
2215 /**
2216  * Return whether there are slots for starting new jobs.
2217  */
2218 static bool has_free_slots()
2219 {
2220  if (max_active_jobs <= 0) return true;
2222 }
2223 
2224 /**
2225  * Handle client requests:
2226  * - check for running targets that have finished,
2227  * - start as many pending targets as allowed,
2228  * - complete the request if there are neither running nor pending targets
2229  * left or if any of them failed.
2230  *
2231  * @return true if some child processes are still running.
2232  *
2233  * @post If there are pending requests, at least one child process is running.
2234  */
2235 static bool handle_clients()
2236 {
2237  DEBUG_open << "Handling client requests... ";
2238  restart:
2239 
2240  for (client_list::iterator i = clients.begin(), i_next = i,
2241  i_end = clients.end(); i != i_end && has_free_slots(); i = i_next)
2242  {
2243  ++i_next;
2244  DEBUG_open << "Handling client from job " << i->job_id << "... ";
2245  if (false)
2246  {
2247  failed:
2248  complete_request(*i, false);
2249  clients.erase(i);
2250  DEBUG_close << "failed\n";
2251  continue;
2252  }
2253 
2254  // Remove running targets that have finished.
2255  for (string_set::iterator j = i->running.begin(), j_next = j,
2256  j_end = i->running.end(); j != j_end; j = j_next)
2257  {
2258  ++j_next;
2259  status_map::const_iterator k = status.find(*j);
2260  assert(k != status.end());
2261  switch (k->second.status)
2262  {
2263  case Running:
2264  break;
2265  case Failed:
2266  if (!keep_going) goto failed;
2267  i->failed = true;
2268  // no break
2269  case Uptodate:
2270  case Remade:
2271  i->running.erase(j);
2272  break;
2273  case Recheck:
2274  case Todo:
2275  assert(false);
2276  }
2277  }
2278 
2279  // Start pending targets.
2280  while (!i->pending.empty())
2281  {
2282  std::string target = i->pending.front();
2283  i->pending.pop_front();
2284  switch (get_status(target).status)
2285  {
2286  case Running:
2287  i->running.insert(target);
2288  break;
2289  case Failed:
2290  pending_failed:
2291  if (!keep_going) goto failed;
2292  i->failed = true;
2293  // no break
2294  case Uptodate:
2295  case Remade:
2296  break;
2297  case Recheck:
2298  case Todo:
2299  client_list::iterator j = i;
2300  if (!start(target, i)) goto pending_failed;
2301  j->running.insert(target);
2302  if (!has_free_slots()) return true;
2303  // Job start might insert a dependency client.
2304  i_next = i;
2305  ++i_next;
2306  break;
2307  }
2308  }
2309 
2310  // Try to complete the request.
2311  // (This might start a new job if it was a dependency client.)
2312  if (i->running.empty())
2313  {
2314  if (i->failed) goto failed;
2315  complete_request(*i, true);
2316  clients.erase(i);
2317  DEBUG_close << "finished\n";
2318  }
2319  }
2320 
2321  if (running_jobs != waiting_jobs) return true;
2322  if (running_jobs == 0 && clients.empty()) return false;
2323 
2324  // There is a circular dependency.
2325  // Try to break it by completing one of the requests.
2326  assert(!clients.empty());
2327  std::cerr << "Circular dependency detected" << std::endl;
2328  client_list::iterator i = clients.begin();
2329  complete_request(*i, false);
2330  clients.erase(i);
2331  goto restart;
2332 }
2333 
2334 /**
2335  * Create a named unix socket that listens for build requests. Also set
2336  * the REMAKE_SOCKET environment variable that will be inherited by all
2337  * the job scripts.
2338  */
2339 static void create_server()
2340 {
2341  if (false)
2342  {
2343  error:
2344  perror("Failed to create server");
2345 #ifndef WINDOWS
2346  error2:
2347 #endif
2348  exit(EXIT_FAILURE);
2349  }
2350  DEBUG_open << "Creating server... ";
2351 
2352 #ifdef WINDOWS
2353  // Prepare a windows socket.
2354  struct sockaddr_in socket_addr;
2355  socket_addr.sin_family = AF_INET;
2356  socket_addr.sin_addr.s_addr = inet_addr("127.0.0.1");
2357  socket_addr.sin_port = 0;
2358 
2359  // Create and listen to the socket.
2360  socket_fd = socket(AF_INET, SOCK_STREAM, 0);
2361  if (socket_fd == INVALID_SOCKET) goto error;
2362  if (!SetHandleInformation((HANDLE)socket_fd, HANDLE_FLAG_INHERIT, 0))
2363  goto error;
2364  if (bind(socket_fd, (struct sockaddr *)&socket_addr, sizeof(sockaddr_in)))
2365  goto error;
2366  int len = sizeof(sockaddr_in);
2367  if (getsockname(socket_fd, (struct sockaddr *)&socket_addr, &len))
2368  goto error;
2369  std::ostringstream buf;
2370  buf << socket_addr.sin_port;
2371  if (!SetEnvironmentVariable("REMAKE_SOCKET", buf.str().c_str()))
2372  goto error;
2373  if (listen(socket_fd, 1000)) goto error;
2374 #else
2375  // Set signal handlers for SIGCHLD and SIGINT.
2376  // Block SIGCHLD (unblocked during select).
2377  sigset_t sigmask;
2378  sigemptyset(&sigmask);
2379  sigaddset(&sigmask, SIGCHLD);
2380  if (sigprocmask(SIG_BLOCK, &sigmask, NULL) == -1) goto error;
2381  struct sigaction sa;
2382  sa.sa_flags = 0;
2383  sigemptyset(&sa.sa_mask);
2384  sa.sa_handler = &sigchld_handler;
2385  if (sigaction(SIGCHLD, &sa, NULL) == -1) goto error;
2386  sa.sa_handler = &sigint_handler;
2387  if (sigaction(SIGINT, &sa, NULL) == -1) goto error;
2388 
2389  // Prepare a named unix socket in temporary directory.
2390  socket_name = tempnam(NULL, "rmk-");
2391  if (!socket_name) goto error2;
2392  struct sockaddr_un socket_addr;
2393  size_t len = strlen(socket_name);
2394  if (len >= sizeof(socket_addr.sun_path) - 1) goto error2;
2395  socket_addr.sun_family = AF_UNIX;
2396  strcpy(socket_addr.sun_path, socket_name);
2397  len += sizeof(socket_addr.sun_family);
2398  if (setenv("REMAKE_SOCKET", socket_name, 1)) goto error;
2399 
2400  // Create and listen to the socket.
2401 #ifdef LINUX
2402  socket_fd = socket(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0);
2403  if (socket_fd == INVALID_SOCKET) goto error;
2404 #else
2405  socket_fd = socket(AF_UNIX, SOCK_STREAM, 0);
2406  if (socket_fd == INVALID_SOCKET) goto error;
2407  if (fcntl(socket_fd, F_SETFD, FD_CLOEXEC) < 0) goto error;
2408 #endif
2409  if (bind(socket_fd, (struct sockaddr *)&socket_addr, len))
2410  goto error;
2411  if (listen(socket_fd, 1000)) goto error;
2412 #endif
2413 }
2414 
2415 /**
2416  * Accept a connection from a client, get the job it spawned from,
2417  * get the targets, and mark them as dependencies of the job targets.
2418  */
2420 {
2421  DEBUG_open << "Handling client request... ";
2422 
2423  // Accept connection.
2424 #ifdef WINDOWS
2425  socket_t fd = accept(socket_fd, NULL, NULL);
2426  if (fd == INVALID_SOCKET) return;
2427  if (!SetHandleInformation((HANDLE)fd, HANDLE_FLAG_INHERIT, 0))
2428  {
2429  error2:
2430  std::cerr << "Unexpected failure while setting connection with client" << std::endl;
2431  closesocket(fd);
2432  return;
2433  }
2434  // WSAEventSelect puts sockets into nonblocking mode, so disable it here.
2435  u_long nbio = 0;
2436  if (ioctlsocket(fd, FIONBIO, &nbio)) goto error2;
2437 #elif defined(LINUX)
2438  int fd = accept4(socket_fd, NULL, NULL, SOCK_CLOEXEC);
2439  if (fd < 0) return;
2440 #else
2441  int fd = accept(socket_fd, NULL, NULL);
2442  if (fd < 0) return;
2443  if (fcntl(fd, F_SETFD, FD_CLOEXEC) < 0) return;
2444 #endif
2445  clients.push_front(client_t());
2446  client_list::iterator proc = clients.begin();
2447 
2448  if (false)
2449  {
2450  error:
2451  DEBUG_close << "failed\n";
2452  std::cerr << "Received an ill-formed client message" << std::endl;
2453  #ifdef WINDOWS
2454  closesocket(fd);
2455  #else
2456  close(fd);
2457  #endif
2458  clients.erase(proc);
2459  return;
2460  }
2461 
2462  // Receive message. Stop when encountering two nuls in a row.
2463  std::vector<char> buf;
2464  size_t len = 0;
2465  while (len < sizeof(int) + 2 || buf[len - 1] || buf[len - 2])
2466  {
2467  buf.resize(len + 1024);
2468  ssize_t l = recv(fd, &buf[0] + len, 1024, 0);
2469  if (l <= 0) goto error;
2470  len += l;
2471  }
2472 
2473  // Parse job that spawned the client.
2474  int job_id;
2475  memcpy(&job_id, &buf[0], sizeof(int));
2476  proc->socket = fd;
2477  proc->job_id = job_id;
2478  job_targets_map::const_iterator i = job_targets.find(job_id);
2479  if (i == job_targets.end()) goto error;
2480  DEBUG << "receiving request from job " << job_id << std::endl;
2481 
2482  // Parse the targets and mark them as dependencies from the job targets.
2483  dependency_t &dep = *dependencies[job_targets[job_id].front()];
2484  char const *p = &buf[0] + sizeof(int);
2485  while (true)
2486  {
2487  len = strlen(p);
2488  if (len == 0)
2489  {
2490  ++waiting_jobs;
2491  return;
2492  }
2493  std::string target(p, p + len);
2494  DEBUG << "adding dependency " << target << " to job\n";
2495  proc->pending.push_back(target);
2496  dep.deps.insert(target);
2497  p += len + 1;
2498  }
2499 }
2500 
2501 /**
2502  * Handle child process exit status.
2503  */
2504 void finalize_job(pid_t pid, bool res)
2505 {
2506  pid_job_map::iterator i = job_pids.find(pid);
2507  assert(i != job_pids.end());
2508  int job_id = i->second;
2509  job_pids.erase(i);
2510  --running_jobs;
2511  complete_job(job_id, res);
2512 }
2513 
2514 /**
2515  * Loop until all the jobs have finished.
2516  *
2517  * @post There are no client requests left, not even virtual ones.
2518  */
2520 {
2521  while (handle_clients())
2522  {
2523  DEBUG_open << "Handling events... ";
2524  #ifdef WINDOWS
2525  size_t len = job_pids.size() + 1;
2526  HANDLE h[len];
2527  int num = 0;
2528  for (pid_job_map::const_iterator i = job_pids.begin(),
2529  i_end = job_pids.end(); i != i_end; ++i, ++num)
2530  {
2531  h[num] = i->first;
2532  }
2533  WSAEVENT aev = WSACreateEvent();
2534  h[num] = aev;
2535  WSAEventSelect(socket_fd, aev, FD_ACCEPT);
2536  DWORD w = WaitForMultipleObjects(len, h, false, INFINITE);
2537  WSAEventSelect(socket_fd, aev, 0);
2538  WSACloseEvent(aev);
2539  if (len <= w)
2540  continue;
2541  if (w == len - 1)
2542  {
2543  accept_client();
2544  continue;
2545  }
2546  pid_t pid = h[w];
2547  DWORD s = 0;
2548  bool res = GetExitCodeProcess(pid, &s) && s == 0;
2549  CloseHandle(pid);
2550  finalize_job(pid, res);
2551  #else
2552  sigset_t emptymask;
2553  sigemptyset(&emptymask);
2554  fd_set fdset;
2555  FD_ZERO(&fdset);
2556  FD_SET(socket_fd, &fdset);
2557  int ret = pselect(socket_fd + 1, &fdset, NULL, NULL, NULL, &emptymask);
2558  if (ret > 0 /* && FD_ISSET(socket_fd, &fdset)*/) accept_client();
2559  if (!got_SIGCHLD) continue;
2560  got_SIGCHLD = 0;
2561  pid_t pid;
2562  int status;
2563  while ((pid = waitpid(-1, &status, WNOHANG)) > 0)
2564  {
2565  bool res = WIFEXITED(status) && WEXITSTATUS(status) == 0;
2566  finalize_job(pid, res);
2567  }
2568  #endif
2569  }
2570 
2571  assert(clients.empty());
2572 }
2573 
2574 /**
2575  * Load dependencies and rules, listen to client requests, and loop until
2576  * all the requests have completed.
2577  * If Remakefile is obsolete, perform a first run with it only, then reload
2578  * the rules, and perform a second with the original clients.
2579  */
2580 void server_mode(std::string const &remakefile, string_list const &targets)
2581 {
2583  load_rules(remakefile);
2584  create_server();
2585  if (get_status(remakefile).status != Uptodate)
2586  {
2587  clients.push_back(client_t());
2588  clients.back().pending.push_back(remakefile);
2589  server_loop();
2590  if (build_failure) goto early_exit;
2591  variables.clear();
2592  specific_rules.clear();
2593  generic_rules.clear();
2594  first_target.clear();
2595  load_rules(remakefile);
2596  }
2597  clients.push_back(client_t());
2598  if (!targets.empty()) clients.back().pending = targets;
2599  else if (!first_target.empty())
2600  clients.back().pending.push_back(first_target);
2601  server_loop();
2602  early_exit:
2603  close(socket_fd);
2604 #ifndef WINDOWS
2605  remove(socket_name);
2606  free(socket_name);
2607 #endif
2609  exit(build_failure ? EXIT_FAILURE : EXIT_SUCCESS);
2610 }
2611 
2612 /** @} */
2613 
2614 /**
2615  * @defgroup client Client
2616  *
2617  * @{
2618  */
2619 
2620 /**
2621  * Connect to the server @a socket_name, send a build request for @a targets,
2622  * and exit with the status returned by the server.
2623  */
2624 void client_mode(char *socket_name, string_list const &targets)
2625 {
2626  if (false)
2627  {
2628  error:
2629  perror("Failed to send targets to server");
2630  exit(EXIT_FAILURE);
2631  }
2632  if (targets.empty()) exit(EXIT_SUCCESS);
2633  DEBUG_open << "Connecting to server... ";
2634 
2635  // Connect to server.
2636 #ifdef WINDOWS
2637  struct sockaddr_in socket_addr;
2638  socket_fd = socket(AF_INET, SOCK_STREAM, 0);
2639  if (socket_fd == INVALID_SOCKET) goto error;
2640  socket_addr.sin_family = AF_INET;
2641  socket_addr.sin_addr.s_addr = inet_addr("127.0.0.1");
2642  socket_addr.sin_port = atoi(socket_name);
2643  if (connect(socket_fd, (struct sockaddr *)&socket_addr, sizeof(sockaddr_in)))
2644  goto error;
2645 #else
2646  struct sockaddr_un socket_addr;
2647  size_t len = strlen(socket_name);
2648  if (len >= sizeof(socket_addr.sun_path) - 1) exit(EXIT_FAILURE);
2649  socket_fd = socket(AF_UNIX, SOCK_STREAM, 0);
2650  if (socket_fd == INVALID_SOCKET) goto error;
2651  socket_addr.sun_family = AF_UNIX;
2652  strcpy(socket_addr.sun_path, socket_name);
2653  if (connect(socket_fd, (struct sockaddr *)&socket_addr, sizeof(socket_addr.sun_family) + len))
2654  goto error;
2655 #ifdef MACOSX
2656  int set_option = 1;
2657  if (setsockopt(socket_fd, SOL_SOCKET, SO_NOSIGPIPE, &set_option, sizeof(set_option)))
2658  goto error;
2659 #endif
2660 #endif
2661 
2662  // Send current job id.
2663  char *id = getenv("REMAKE_JOB_ID");
2664  int job_id = id ? atoi(id) : -1;
2665  if (send(socket_fd, (char *)&job_id, sizeof(job_id), MSG_NOSIGNAL) != sizeof(job_id))
2666  goto error;
2667 
2668  // Send targets.
2669  for (string_list::const_iterator i = targets.begin(),
2670  i_end = targets.end(); i != i_end; ++i)
2671  {
2672  DEBUG_open << "Sending " << *i << "... ";
2673  ssize_t len = i->length() + 1;
2674  if (send(socket_fd, i->c_str(), len, MSG_NOSIGNAL) != len)
2675  goto error;
2676  }
2677 
2678  // Send terminating nul and wait for reply.
2679  char result = 0;
2680  if (send(socket_fd, &result, 1, MSG_NOSIGNAL) != 1) goto error;
2681  if (recv(socket_fd, &result, 1, 0) != 1) exit(EXIT_FAILURE);
2682  exit(result ? EXIT_SUCCESS : EXIT_FAILURE);
2683 }
2684 
2685 /** @} */
2686 
2687 /**
2688  * @defgroup ui User interface
2689  *
2690  * @{
2691  */
2692 
2693 /**
2694  * Display usage and exit with @a exit_status.
2695  */
2696 void usage(int exit_status)
2697 {
2698  std::cerr << "Usage: remake [options] [target] ...\n"
2699  "Options\n"
2700  " -d Echo script commands.\n"
2701  " -d -d Print lots of debugging information.\n"
2702  " -f FILE Read FILE as Remakefile.\n"
2703  " -h, --help Print this message and exit.\n"
2704  " -j[N], --jobs=[N] Allow N jobs at once; infinite jobs with no arg.\n"
2705  " -k Keep going when some targets cannot be made.\n"
2706  " -r Look up targets from the dependencies on stdin.\n"
2707  " -s, --silent, --quiet Do not echo targets.\n";
2708  exit(exit_status);
2709 }
2710 
2711 /**
2712  * This program behaves in two different ways.
2713  *
2714  * - If the environment contains the REMAKE_SOCKET variable, the client
2715  * connects to this socket and sends to the server its build targets.
2716  * It exits once it receives the server reply.
2717  *
2718  * - Otherwise, it creates a server that waits for build requests. It
2719  * also creates a pseudo-client that requests the targets passed on the
2720  * command line.
2721  */
2722 int main(int argc, char *argv[])
2723 {
2724  init_working_dir();
2725 
2726  std::string remakefile = "Remakefile";
2727  string_list targets;
2728  bool indirect_targets = false;
2729 
2730  // Parse command-line arguments.
2731  for (int i = 1; i < argc; ++i)
2732  {
2733  std::string arg = argv[i];
2734  if (arg.empty()) usage(EXIT_FAILURE);
2735  if (arg == "-h" || arg == "--help") usage(EXIT_SUCCESS);
2736  if (arg == "-d")
2737  if (echo_scripts) debug.active = true;
2738  else echo_scripts = true;
2739  else if (arg == "-k" || arg =="--keep-going")
2740  keep_going = true;
2741  else if (arg == "-s" || arg == "--silent" || arg == "--quiet")
2742  show_targets = false;
2743  else if (arg == "-r")
2744  indirect_targets = true;
2745  else if (arg == "-f")
2746  {
2747  if (++i == argc) usage(EXIT_FAILURE);
2748  remakefile = argv[i];
2749  }
2750  else if (arg.compare(0, 2, "-j") == 0)
2751  max_active_jobs = atoi(arg.c_str() + 2);
2752  else if (arg.compare(0, 7, "--jobs=") == 0)
2753  max_active_jobs = atoi(arg.c_str() + 7);
2754  else
2755  {
2756  if (arg[0] == '-') usage(EXIT_FAILURE);
2757  targets.push_back(normalize(arg));
2758  DEBUG << "New target: " << arg << '\n';
2759  }
2760  }
2761 
2762  if (indirect_targets)
2763  {
2764  load_dependencies(std::cin);
2765  string_list l;
2766  targets.swap(l);
2767  if (l.empty() && !dependencies.empty())
2768  {
2769  l.push_back(dependencies.begin()->second->targets.front());
2770  }
2771  for (string_list::const_iterator i = l.begin(),
2772  i_end = l.end(); i != i_end; ++i)
2773  {
2774  dependency_map::const_iterator j = dependencies.find(*i);
2775  if (j == dependencies.end()) continue;
2776  dependency_t const &dep = *j->second;
2777  for (string_set::const_iterator k = dep.deps.begin(),
2778  k_end = dep.deps.end(); k != k_end; ++k)
2779  {
2780  targets.push_back(normalize(*k));
2781  }
2782  }
2783  dependencies.clear();
2784  }
2785 
2786 #ifdef WINDOWS
2787  WSADATA wsaData;
2788  if (WSAStartup(MAKEWORD(2,2), &wsaData))
2789  {
2790  std::cerr << "Unexpected failure while initializing Windows Socket" << std::endl;
2791  return 1;
2792  }
2793 #endif
2794 
2795  // Run as client if REMAKE_SOCKET is present in the environment.
2796  if (char *sn = getenv("REMAKE_SOCKET")) client_mode(sn, targets);
2797 
2798  // Otherwise run as server.
2799  server_mode(remakefile, targets);
2800 }
2801 
2802 /** @} */