DriverInternals.rst source code [clang_source_code/docs/DriverInternals.rst]

1	=========================
2	Driver Design & Internals
3	=========================
4
5	.. contents::
6	:local:
7
8	Introduction
9	============
10
11	This document describes the Clang driver. The purpose of this document
12	is to describe both the motivation and design goals for the driver, as
13	well as details of the internal implementation.
14
15	Features and Goals
16	==================
17
18	The Clang driver is intended to be a production quality compiler driver
19	providing access to the Clang compiler and tools, with a command line
20	interface which is compatible with the gcc driver.
21
22	Although the driver is part of and driven by the Clang project, it is
23	logically a separate tool which shares many of the same goals as Clang:
24
25	.. contents:: Features
26	:local:
27
28	GCC Compatibility
29	-----------------
30
31	The number one goal of the driver is to ease the adoption of Clang by
32	allowing users to drop Clang into a build system which was designed to
33	call GCC. Although this makes the driver much more complicated than
34	might otherwise be necessary, we decided that being very compatible with
35	the gcc command line interface was worth it in order to allow users to
36	quickly test clang on their projects.
37
38	Flexible
39	--------
40
41	The driver was designed to be flexible and easily accommodate new uses
42	as we grow the clang and LLVM infrastructure. As one example, the driver
43	can easily support the introduction of tools which have an integrated
44	assembler; something we hope to add to LLVM in the future.
45
46	Similarly, most of the driver functionality is kept in a library which
47	can be used to build other tools which want to implement or accept a gcc
48	like interface.
49
50	Low Overhead
51	------------
52
53	The driver should have as little overhead as possible. In practice, we
54	found that the gcc driver by itself incurred a small but meaningful
55	overhead when compiling many small files. The driver doesn't do much
56	work compared to a compilation, but we have tried to keep it as
57	efficient as possible by following a few simple principles:
58
59	- Avoid memory allocation and string copying when possible.
60	- Don't parse arguments more than once.
61	- Provide a few simple interfaces for efficiently searching arguments.
62
63	Simple
64	------
65
66	Finally, the driver was designed to be "as simple as possible", given
67	the other goals. Notably, trying to be completely compatible with the
68	gcc driver adds a significant amount of complexity. However, the design
69	of the driver attempts to mitigate this complexity by dividing the
70	process into a number of independent stages instead of a single
71	monolithic task.
72
73	Internal Design and Implementation
74	==================================
75
76	.. contents::
77	:local:
78	:depth: 1
79
80	Internals Introduction
81	----------------------
82
83	In order to satisfy the stated goals, the driver was designed to
84	completely subsume the functionality of the gcc executable; that is, the
85	driver should not need to delegate to gcc to perform subtasks. On
86	Darwin, this implies that the Clang driver also subsumes the gcc
87	driver-driver, which is used to implement support for building universal
88	images (binaries and object files). This also implies that the driver
89	should be able to call the language specific compilers (e.g. cc1)
90	directly, which means that it must have enough information to forward
91	command line arguments to child processes correctly.
92
93	Design Overview
94	---------------
95
96	The diagram below shows the significant components of the driver
97	architecture and how they relate to one another. The orange components
98	represent concrete data structures built by the driver, the green
99	components indicate conceptually distinct stages which manipulate these
100	data structures, and the blue components are important helper classes.
101
102	.. image:: DriverArchitecture.png
103	:align: center
104	:alt: Driver Architecture Diagram
105
106	Driver Stages
107	-------------
108
109	The driver functionality is conceptually divided into five stages:
110
111	#. Parse: Option Parsing
112
113	The command line argument strings are decomposed into arguments
114	(``Arg`` instances). The driver expects to understand all available
115	options, although there is some facility for just passing certain
116	classes of options through (like ``-Wl,``).
117
118	Each argument corresponds to exactly one abstract ``Option``
119	definition, which describes how the option is parsed along with some
120	additional metadata. The Arg instances themselves are lightweight and
121	merely contain enough information for clients to determine which
122	option they correspond to and their values (if they have additional
123	parameters).
124
125	For example, a command line like "-Ifoo -I foo" would parse to two
126	Arg instances (a JoinedArg and a SeparateArg instance), but each
127	would refer to the same Option.
128
129	Options are lazily created in order to avoid populating all Option
130	classes when the driver is loaded. Most of the driver code only needs
131	to deal with options by their unique ID (e.g., ``options::OPT_I``),
132
133	Arg instances themselves do not generally store the values of
134	parameters. In many cases, this would simply result in creating
135	unnecessary string copies. Instead, Arg instances are always embedded
136	inside an ArgList structure, which contains the original vector of
137	argument strings. Each Arg itself only needs to contain an index into
138	this vector instead of storing its values directly.
139
140	The clang driver can dump the results of this stage using the
141	``-###`` flag (which must precede any actual command
142	line arguments). For example:
143
144	.. code-block:: console
145
146	$ clang -### -Xarch_i386 -fomit-frame-pointer -Wa,-fast -Ifoo -I foo t.c
147	Option 0 - Name: "-Xarch_", Values: {"i386", "-fomit-frame-pointer"}
148	Option 1 - Name: "-Wa,", Values: {"-fast"}
149	Option 2 - Name: "-I", Values: {"foo"}
150	Option 3 - Name: "-I", Values: {"foo"}
151	Option 4 - Name: "<input>", Values: {"t.c"}
152
153	After this stage is complete the command line should be broken down
154	into well defined option objects with their appropriate parameters.
155	Subsequent stages should rarely, if ever, need to do any string
156	processing.
157
158	#. Pipeline: Compilation Action Construction
159
160	Once the arguments are parsed, the tree of subprocess jobs needed for
161	the desired compilation sequence are constructed. This involves
162	determining the input files and their types, what work is to be done
163	on them (preprocess, compile, assemble, link, etc.), and constructing
164	a list of Action instances for each task. The result is a list of one
165	or more top-level actions, each of which generally corresponds to a
166	single output (for example, an object or linked executable).
167
168	The majority of Actions correspond to actual tasks, however there are
169	two special Actions. The first is InputAction, which simply serves to
170	adapt an input argument for use as an input to other Actions. The
171	second is BindArchAction, which conceptually alters the architecture
172	to be used for all of its input Actions.
173
174	The clang driver can dump the results of this stage using the
175	``-ccc-print-phases`` flag. For example:
176
177	.. code-block:: console
178
179	$ clang -ccc-print-phases -x c t.c -x assembler t.s
180	0: input, "t.c", c
181	1: preprocessor, {0}, cpp-output
182	2: compiler, {1}, assembler
183	3: assembler, {2}, object
184	4: input, "t.s", assembler
185	5: assembler, {4}, object
186	6: linker, {3, 5}, image
187
188	Here the driver is constructing seven distinct actions, four to
189	compile the "t.c" input into an object file, two to assemble the
190	"t.s" input, and one to link them together.
191
192	A rather different compilation pipeline is shown here; in this
193	example there are two top level actions to compile the input files
194	into two separate object files, where each object file is built using
195	``lipo`` to merge results built for two separate architectures.
196
197	.. code-block:: console
198
199	$ clang -ccc-print-phases -c -arch i386 -arch x86_64 t0.c t1.c
200	0: input, "t0.c", c
201	1: preprocessor, {0}, cpp-output
202	2: compiler, {1}, assembler
203	3: assembler, {2}, object
204	4: bind-arch, "i386", {3}, object
205	5: bind-arch, "x86_64", {3}, object
206	6: lipo, {4, 5}, object
207	7: input, "t1.c", c
208	8: preprocessor, {7}, cpp-output
209	9: compiler, {8}, assembler
210	10: assembler, {9}, object
211	11: bind-arch, "i386", {10}, object
212	12: bind-arch, "x86_64", {10}, object
213	13: lipo, {11, 12}, object
214
215	After this stage is complete the compilation process is divided into
216	a simple set of actions which need to be performed to produce
217	intermediate or final outputs (in some cases, like ``-fsyntax-only``,
218	there is no "real" final output). Phases are well known compilation
219	steps, such as "preprocess", "compile", "assemble", "link", etc.
220
221	#. Bind: Tool & Filename Selection
222
223	This stage (in conjunction with the Translate stage) turns the tree
224	of Actions into a list of actual subprocess to run. Conceptually, the
225	driver performs a top down matching to assign Action(s) to Tools. The
226	ToolChain is responsible for selecting the tool to perform a
227	particular action; once selected the driver interacts with the tool
228	to see if it can match additional actions (for example, by having an
229	integrated preprocessor).
230
231	Once Tools have been selected for all actions, the driver determines
232	how the tools should be connected (for example, using an inprocess
233	module, pipes, temporary files, or user provided filenames). If an
234	output file is required, the driver also computes the appropriate
235	file name (the suffix and file location depend on the input types and
236	options such as ``-save-temps``).
237
238	The driver interacts with a ToolChain to perform the Tool bindings.
239	Each ToolChain contains information about all the tools needed for
240	compilation for a particular architecture, platform, and operating
241	system. A single driver invocation may query multiple ToolChains
242	during one compilation in order to interact with tools for separate
243	architectures.
244
245	The results of this stage are not computed directly, but the driver
246	can print the results via the ``-ccc-print-bindings`` option. For
247	example:
248
249	.. code-block:: console
250
251	$ clang -ccc-print-bindings -arch i386 -arch ppc t0.c
252	# "i386-apple-darwin9" - "clang", inputs: ["t0.c"], output: "/tmp/cc-Sn4RKF.s"
253	# "i386-apple-darwin9" - "darwin::Assemble", inputs: ["/tmp/cc-Sn4RKF.s"], output: "/tmp/cc-gvSnbS.o"
254	# "i386-apple-darwin9" - "darwin::Link", inputs: ["/tmp/cc-gvSnbS.o"], output: "/tmp/cc-jgHQxi.out"
255	# "ppc-apple-darwin9" - "gcc::Compile", inputs: ["t0.c"], output: "/tmp/cc-Q0bTox.s"
256	# "ppc-apple-darwin9" - "gcc::Assemble", inputs: ["/tmp/cc-Q0bTox.s"], output: "/tmp/cc-WCdicw.o"
257	# "ppc-apple-darwin9" - "gcc::Link", inputs: ["/tmp/cc-WCdicw.o"], output: "/tmp/cc-HHBEBh.out"
258	# "i386-apple-darwin9" - "darwin::Lipo", inputs: ["/tmp/cc-jgHQxi.out", "/tmp/cc-HHBEBh.out"], output: "a.out"
259
260	This shows the tool chain, tool, inputs and outputs which have been
261	bound for this compilation sequence. Here clang is being used to
262	compile t0.c on the i386 architecture and darwin specific versions of
263	the tools are being used to assemble and link the result, but generic
264	gcc versions of the tools are being used on PowerPC.
265
266	#. Translate: Tool Specific Argument Translation
267
268	Once a Tool has been selected to perform a particular Action, the
269	Tool must construct concrete Commands which will be executed during
270	compilation. The main work is in translating from the gcc style
271	command line options to whatever options the subprocess expects.
272
273	Some tools, such as the assembler, only interact with a handful of
274	arguments and just determine the path of the executable to call and
275	pass on their input and output arguments. Others, like the compiler
276	or the linker, may translate a large number of arguments in addition.
277
278	The ArgList class provides a number of simple helper methods to
279	assist with translating arguments; for example, to pass on only the
280	last of arguments corresponding to some option, or all arguments for
281	an option.
282
283	The result of this stage is a list of Commands (executable paths and
284	argument strings) to execute.
285
286	#. Execute
287
288	Finally, the compilation pipeline is executed. This is mostly
289	straightforward, although there is some interaction with options like
290	``-pipe``, ``-pass-exit-codes`` and ``-time``.
291
292	Additional Notes
293	----------------
294
295	The Compilation Object
296	^^^^^^^^^^^^^^^^^^^^^^
297
298	The driver constructs a Compilation object for each set of command line
299	arguments. The Driver itself is intended to be invariant during
300	construction of a Compilation; an IDE should be able to construct a
301	single long lived driver instance to use for an entire build, for
302	example.
303
304	The Compilation object holds information that is particular to each
305	compilation sequence. For example, the list of used temporary files
306	(which must be removed once compilation is finished) and result files
307	(which should be removed if compilation fails).
308
309	Unified Parsing & Pipelining
310	^^^^^^^^^^^^^^^^^^^^^^^^^^^^
311
312	Parsing and pipelining both occur without reference to a Compilation
313	instance. This is by design; the driver expects that both of these
314	phases are platform neutral, with a few very well defined exceptions
315	such as whether the platform uses a driver driver.
316
317	ToolChain Argument Translation
318	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
319
320	In order to match gcc very closely, the clang driver currently allows
321	tool chains to perform their own translation of the argument list (into
322	a new ArgList data structure). Although this allows the clang driver to
323	match gcc easily, it also makes the driver operation much harder to
324	understand (since the Tools stop seeing some arguments the user
325	provided, and see new ones instead).
326
327	For example, on Darwin ``-gfull`` gets translated into two separate
328	arguments, ``-g`` and ``-fno-eliminate-unused-debug-symbols``. Trying to
329	write Tool logic to do something with ``-gfull`` will not work, because
330	Tool argument translation is done after the arguments have been
331	translated.
332
333	A long term goal is to remove this tool chain specific translation, and
334	instead force each tool to change its own logic to do the right thing on
335	the untranslated original arguments.
336
337	Unused Argument Warnings
338	^^^^^^^^^^^^^^^^^^^^^^^^
339
340	The driver operates by parsing all arguments but giving Tools the
341	opportunity to choose which arguments to pass on. One downside of this
342	infrastructure is that if the user misspells some option, or is confused
343	about which options to use, some command line arguments the user really
344	cared about may go unused. This problem is particularly important when
345	using clang as a compiler, since the clang compiler does not support
346	anywhere near all the options that gcc does, and we want to make sure
347	users know which ones are being used.
348
349	To support this, the driver maintains a bit associated with each
350	argument of whether it has been used (at all) during the compilation.
351	This bit usually doesn't need to be set by hand, as the key ArgList
352	accessors will set it automatically.
353
354	When a compilation is successful (there are no errors), the driver
355	checks the bit and emits an "unused argument" warning for any arguments
356	which were never accessed. This is conservative (the argument may not
357	have been used to do what the user wanted) but still catches the most
358	obvious cases.
359
360	Relation to GCC Driver Concepts
361	-------------------------------
362
363	For those familiar with the gcc driver, this section provides a brief
364	overview of how things from the gcc driver map to the clang driver.
365
366	- Driver Driver
367
368	The driver driver is fully integrated into the clang driver. The
369	driver simply constructs additional Actions to bind the architecture
370	during the Pipeline phase. The tool chain specific argument
371	translation is responsible for handling ``-Xarch_``.
372
373	The one caveat is that this approach requires ``-Xarch_`` not be used
374	to alter the compilation itself (for example, one cannot provide
375	``-S`` as an ``-Xarch_`` argument). The driver attempts to reject
376	such invocations, and overall there isn't a good reason to abuse
377	``-Xarch_`` to that end in practice.
378
379	The upside is that the clang driver is more efficient and does little
380	extra work to support universal builds. It also provides better error
381	reporting and UI consistency.
382
383	- Specs
384
385	The clang driver has no direct correspondent for "specs". The
386	majority of the functionality that is embedded in specs is in the
387	Tool specific argument translation routines. The parts of specs which
388	control the compilation pipeline are generally part of the Pipeline
389	stage.
390
391	- Toolchains
392
393	The gcc driver has no direct understanding of tool chains. Each gcc
394	binary roughly corresponds to the information which is embedded
395	inside a single ToolChain.
396
397	The clang driver is intended to be portable and support complex
398	compilation environments. All platform and tool chain specific code
399	should be protected behind either abstract or well defined interfaces
400	(such as whether the platform supports use as a driver driver).
401

Clang Project