3 # Author: Masahiro Yamada <yamada.masahiro@socionext.com>
5 # SPDX-License-Identifier: GPL-2.0+
9 Move config options from headers to defconfig files.
11 Since Kconfig was introduced to U-Boot, we have worked on moving
12 config options from headers to Kconfig (defconfig).
14 This tool intends to help this tremendous work.
20 First, you must edit the Kconfig to add the menu entries for the configs
23 And then run this tool giving CONFIG names you want to move.
24 For example, if you want to move CONFIG_CMD_USB and CONFIG_SYS_TEXT_BASE,
25 simply type as follows:
27 $ tools/moveconfig.py CONFIG_CMD_USB CONFIG_SYS_TEXT_BASE
29 The tool walks through all the defconfig files and move the given CONFIGs.
31 The log is also displayed on the terminal.
33 The log is printed for each defconfig as follows:
41 <defconfig_name> is the name of the defconfig.
43 <action*> shows what the tool did for that defconfig.
44 It looks like one of the following:
47 This config option was moved to the defconfig
49 - CONFIG_... is not defined in Kconfig. Do nothing.
50 The entry for this CONFIG was not found in Kconfig. The option is not
51 defined in the config header, either. So, this case can be just skipped.
53 - CONFIG_... is not defined in Kconfig (suspicious). Do nothing.
54 This option is defined in the config header, but its entry was not found
56 There are two common cases:
57 - You forgot to create an entry for the CONFIG before running
58 this tool, or made a typo in a CONFIG passed to this tool.
59 - The entry was hidden due to unmet 'depends on'.
60 The tool does not know if the result is reasonable, so please check it
63 - 'CONFIG_...' is the same as the define in Kconfig. Do nothing.
64 The define in the config header matched the one in Kconfig.
65 We do not need to touch it.
67 - Compiler is missing. Do nothing.
68 The compiler specified for this architecture was not found
69 in your PATH environment.
70 (If -e option is passed, the tool exits immediately.)
73 An error occurred during processing this defconfig. Skipped.
74 (If -e option is passed, the tool exits immediately on error.)
76 Finally, you will be asked, Clean up headers? [y/n]:
78 If you say 'y' here, the unnecessary config defines are removed
79 from the config headers (include/configs/*.h).
80 It just uses the regex method, so you should not rely on it.
81 Just in case, please do 'git diff' to see what happened.
87 This tool runs configuration and builds include/autoconf.mk for every
88 defconfig. The config options defined in Kconfig appear in the .config
89 file (unless they are hidden because of unmet dependency.)
90 On the other hand, the config options defined by board headers are seen
91 in include/autoconf.mk. The tool looks for the specified options in both
92 of them to decide the appropriate action for the options. If the given
93 config option is found in the .config, but its value does not match the
94 one from the board header, the config option in the .config is replaced
95 with the define in the board header. Then, the .config is synced by
96 "make savedefconfig" and the defconfig is updated with it.
98 For faster processing, this tool handles multi-threading. It creates
99 separate build directories where the out-of-tree build is run. The
100 temporary build directories are automatically created and deleted as
101 needed. The number of threads are chosen based on the number of the CPU
102 cores of your system although you can change it via -j (--jobs) option.
108 Appropriate toolchain are necessary to generate include/autoconf.mk
109 for all the architectures supported by U-Boot. Most of them are available
110 at the kernel.org site, some are not provided by kernel.org.
112 The default per-arch CROSS_COMPILE used by this tool is specified by
113 the list below, CROSS_COMPILE. You may wish to update the list to
114 use your own. Instead of modifying the list directly, you can give
115 them via environments.
122 Surround each portion of the log with escape sequences to display it
123 in color on the terminal.
126 Specify a file containing a list of defconfigs to move
129 Perform a trial run that does not make any changes. It is useful to
130 see what is going to happen before one actually runs it.
133 Exit immediately if Make exits with a non-zero status while processing
137 Do "make savedefconfig" forcibly for all the defconfig files.
138 If not specified, "make savedefconfig" only occurs for cases
139 where at least one CONFIG was moved.
142 Only cleanup the headers; skip the defconfig processing
145 Specify the number of threads to run simultaneously. If not specified,
146 the number of threads is the same as the number of CPU cores.
149 Specify the git ref to clone for building the autoconf.mk. If unspecified
150 use the CWD. This is useful for when changes to the Kconfig affect the
151 default values and you want to capture the state of the defconfig from
152 before that change was in effect. If in doubt, specify a ref pre-Kconfig
153 changes (use HEAD if Kconfig changes are not committed). Worst case it will
154 take a bit longer to run, but will always do the right thing.
157 Show any build errors as boards are built
159 To see the complete list of supported options, run
161 $ tools/moveconfig.py -h
169 import multiprocessing
179 SHOW_GNU_MAKE = 'scripts/show-gnu-make'
182 # Here is the list of cross-tools I use.
183 # Most of them are available at kernel.org
184 # (https://www.kernel.org/pub/tools/crosstool/files/bin/), except the following:
185 # arc: https://github.com/foss-for-synopsys-dwc-arc-processors/toolchain/releases
186 # blackfin: http://sourceforge.net/projects/adi-toolchain/files/
187 # nds32: http://osdk.andestech.com/packages/nds32le-linux-glibc-v1.tgz
188 # nios2: https://sourcery.mentor.com/GNUToolchain/subscription42545
189 # sh: http://sourcery.mentor.com/public/gnu_toolchain/sh-linux-gnu
191 # openrisc kernel.org toolchain is out of date, download latest one from
192 # http://opencores.org/or1k/OpenRISC_GNU_tool_chain#Prebuilt_versions
195 'aarch64': 'aarch64-linux-',
196 'arm': 'arm-unknown-linux-gnueabi-',
197 'avr32': 'avr32-linux-',
198 'blackfin': 'bfin-elf-',
199 'm68k': 'm68k-linux-',
200 'microblaze': 'microblaze-linux-',
201 'mips': 'mips-linux-',
202 'nds32': 'nds32le-linux-',
203 'nios2': 'nios2-linux-gnu-',
204 'openrisc': 'or1k-elf-',
205 'powerpc': 'powerpc-linux-',
206 'sh': 'sh-linux-gnu-',
207 'sparc': 'sparc-linux-',
208 'x86': 'i386-linux-',
209 'xtensa': 'xtensa-linux-'
215 STATE_SAVEDEFCONFIG = 3
219 ACTION_NO_ENTRY_WARN = 2
227 COLOR_PURPLE = '0;35'
229 COLOR_LIGHT_GRAY = '0;37'
230 COLOR_DARK_GRAY = '1;30'
231 COLOR_LIGHT_RED = '1;31'
232 COLOR_LIGHT_GREEN = '1;32'
233 COLOR_YELLOW = '1;33'
234 COLOR_LIGHT_BLUE = '1;34'
235 COLOR_LIGHT_PURPLE = '1;35'
236 COLOR_LIGHT_CYAN = '1;36'
239 ### helper functions ###
241 """Get the file object of '/dev/null' device."""
243 devnull = subprocess.DEVNULL # py3k
244 except AttributeError:
245 devnull = open(os.devnull, 'wb')
248 def check_top_directory():
249 """Exit if we are not at the top of source directory."""
250 for f in ('README', 'Licenses'):
251 if not os.path.exists(f):
252 sys.exit('Please run at the top of source directory.')
254 def check_clean_directory():
255 """Exit if the source tree is not clean."""
256 for f in ('.config', 'include/config'):
257 if os.path.exists(f):
258 sys.exit("source tree is not clean, please run 'make mrproper'")
261 """Get the command name of GNU Make.
263 U-Boot needs GNU Make for building, but the command name is not
264 necessarily "make". (for example, "gmake" on FreeBSD).
265 Returns the most appropriate command name on your system.
267 process = subprocess.Popen([SHOW_GNU_MAKE], stdout=subprocess.PIPE)
268 ret = process.communicate()
269 if process.returncode:
270 sys.exit('GNU Make not found')
271 return ret[0].rstrip()
273 def get_all_defconfigs():
274 """Get all the defconfig files under the configs/ directory."""
276 for (dirpath, dirnames, filenames) in os.walk('configs'):
277 dirpath = dirpath[len('configs') + 1:]
278 for filename in fnmatch.filter(filenames, '*_defconfig'):
279 defconfigs.append(os.path.join(dirpath, filename))
283 def color_text(color_enabled, color, string):
284 """Return colored string."""
286 # LF should not be surrounded by the escape sequence.
287 # Otherwise, additional whitespace or line-feed might be printed.
288 return '\n'.join([ '\033[' + color + 'm' + s + '\033[0m' if s else ''
289 for s in string.split('\n') ])
293 def show_diff(a, b, file_path, color_enabled):
294 """Show unidified diff.
297 a: A list of lines (before)
298 b: A list of lines (after)
299 file_path: Path to the file
300 color_enabled: Display the diff in color
303 diff = difflib.unified_diff(a, b,
304 fromfile=os.path.join('a', file_path),
305 tofile=os.path.join('b', file_path))
308 if line[0] == '-' and line[1] != '-':
309 print color_text(color_enabled, COLOR_RED, line),
310 elif line[0] == '+' and line[1] != '+':
311 print color_text(color_enabled, COLOR_GREEN, line),
315 def update_cross_compile(color_enabled):
316 """Update per-arch CROSS_COMPILE via environment variables
318 The default CROSS_COMPILE values are available
319 in the CROSS_COMPILE list above.
321 You can override them via environment variables
322 CROSS_COMPILE_{ARCH}.
324 For example, if you want to override toolchain prefixes
325 for ARM and PowerPC, you can do as follows in your shell:
327 export CROSS_COMPILE_ARM=...
328 export CROSS_COMPILE_POWERPC=...
330 Then, this function checks if specified compilers really exist in your
335 for arch in os.listdir('arch'):
336 if os.path.exists(os.path.join('arch', arch, 'Makefile')):
339 # arm64 is a special case
340 archs.append('aarch64')
343 env = 'CROSS_COMPILE_' + arch.upper()
344 cross_compile = os.environ.get(env)
345 if not cross_compile:
346 cross_compile = CROSS_COMPILE.get(arch, '')
348 for path in os.environ["PATH"].split(os.pathsep):
349 gcc_path = os.path.join(path, cross_compile + 'gcc')
350 if os.path.isfile(gcc_path) and os.access(gcc_path, os.X_OK):
353 print >> sys.stderr, color_text(color_enabled, COLOR_YELLOW,
354 'warning: %sgcc: not found in PATH. %s architecture boards will be skipped'
355 % (cross_compile, arch))
358 CROSS_COMPILE[arch] = cross_compile
360 def extend_matched_lines(lines, matched, pre_patterns, post_patterns, extend_pre,
362 """Extend matched lines if desired patterns are found before/after already
366 lines: A list of lines handled.
367 matched: A list of line numbers that have been already matched.
368 (will be updated by this function)
369 pre_patterns: A list of regular expression that should be matched as
371 post_patterns: A list of regular expression that should be matched as
373 extend_pre: Add the line number of matched preamble to the matched list.
374 extend_post: Add the line number of matched postamble to the matched list.
376 extended_matched = []
389 for p in pre_patterns:
390 if p.search(lines[i - 1]):
396 for p in post_patterns:
397 if p.search(lines[j]):
404 extended_matched.append(i - 1)
406 extended_matched.append(j)
408 matched += extended_matched
411 def cleanup_one_header(header_path, patterns, options):
412 """Clean regex-matched lines away from a file.
415 header_path: path to the cleaned file.
416 patterns: list of regex patterns. Any lines matching to these
417 patterns are deleted.
418 options: option flags.
420 with open(header_path) as f:
421 lines = f.readlines()
424 for i, line in enumerate(lines):
425 if i - 1 in matched and lines[i - 1][-2:] == '\\\n':
428 for pattern in patterns:
429 if pattern.search(line):
436 # remove empty #ifdef ... #endif, successive blank lines
437 pattern_if = re.compile(r'#\s*if(def|ndef)?\W') # #if, #ifdef, #ifndef
438 pattern_elif = re.compile(r'#\s*el(if|se)\W') # #elif, #else
439 pattern_endif = re.compile(r'#\s*endif\W') # #endif
440 pattern_blank = re.compile(r'^\s*$') # empty line
443 old_matched = copy.copy(matched)
444 extend_matched_lines(lines, matched, [pattern_if],
445 [pattern_endif], True, True)
446 extend_matched_lines(lines, matched, [pattern_elif],
447 [pattern_elif, pattern_endif], True, False)
448 extend_matched_lines(lines, matched, [pattern_if, pattern_elif],
449 [pattern_blank], False, True)
450 extend_matched_lines(lines, matched, [pattern_blank],
451 [pattern_elif, pattern_endif], True, False)
452 extend_matched_lines(lines, matched, [pattern_blank],
453 [pattern_blank], True, False)
454 if matched == old_matched:
457 tolines = copy.copy(lines)
459 for i in reversed(matched):
462 show_diff(lines, tolines, header_path, options.color)
467 with open(header_path, 'w') as f:
471 def cleanup_headers(configs, options):
472 """Delete config defines from board headers.
475 configs: A list of CONFIGs to remove.
476 options: option flags.
479 choice = raw_input('Clean up headers? [y/n]: ').lower()
481 if choice == 'y' or choice == 'n':
488 for config in configs:
489 patterns.append(re.compile(r'#\s*define\s+%s\W' % config))
490 patterns.append(re.compile(r'#\s*undef\s+%s\W' % config))
492 for dir in 'include', 'arch', 'board':
493 for (dirpath, dirnames, filenames) in os.walk(dir):
494 if dirpath == os.path.join('include', 'generated'):
496 for filename in filenames:
497 if not fnmatch.fnmatch(filename, '*~'):
498 cleanup_one_header(os.path.join(dirpath, filename),
501 def cleanup_one_extra_option(defconfig_path, configs, options):
502 """Delete config defines in CONFIG_SYS_EXTRA_OPTIONS in one defconfig file.
505 defconfig_path: path to the cleaned defconfig file.
506 configs: A list of CONFIGs to remove.
507 options: option flags.
510 start = 'CONFIG_SYS_EXTRA_OPTIONS="'
513 with open(defconfig_path) as f:
514 lines = f.readlines()
516 for i, line in enumerate(lines):
517 if line.startswith(start) and line.endswith(end):
520 # CONFIG_SYS_EXTRA_OPTIONS was not found in this defconfig
523 old_tokens = line[len(start):-len(end)].split(',')
526 for token in old_tokens:
527 pos = token.find('=')
528 if not (token[:pos] if pos >= 0 else token) in configs:
529 new_tokens.append(token)
531 if new_tokens == old_tokens:
534 tolines = copy.copy(lines)
537 tolines[i] = start + ','.join(new_tokens) + end
541 show_diff(lines, tolines, defconfig_path, options.color)
546 with open(defconfig_path, 'w') as f:
550 def cleanup_extra_options(configs, options):
551 """Delete config defines in CONFIG_SYS_EXTRA_OPTIONS in defconfig files.
554 configs: A list of CONFIGs to remove.
555 options: option flags.
558 choice = raw_input('Clean up CONFIG_SYS_EXTRA_OPTIONS? [y/n]: ').lower()
560 if choice == 'y' or choice == 'n':
566 configs = [ config[len('CONFIG_'):] for config in configs ]
568 defconfigs = get_all_defconfigs()
570 for defconfig in defconfigs:
571 cleanup_one_extra_option(os.path.join('configs', defconfig), configs,
577 """Progress Indicator"""
579 def __init__(self, total):
580 """Create a new progress indicator.
583 total: A number of defconfig files to process.
589 """Increment the number of processed defconfig files."""
594 """Display the progress."""
595 print ' %d defconfigs out of %d\r' % (self.current, self.total),
600 """A parser of .config and include/autoconf.mk."""
602 re_arch = re.compile(r'CONFIG_SYS_ARCH="(.*)"')
603 re_cpu = re.compile(r'CONFIG_SYS_CPU="(.*)"')
605 def __init__(self, configs, options, build_dir):
606 """Create a new parser.
609 configs: A list of CONFIGs to move.
610 options: option flags.
611 build_dir: Build directory.
613 self.configs = configs
614 self.options = options
615 self.dotconfig = os.path.join(build_dir, '.config')
616 self.autoconf = os.path.join(build_dir, 'include', 'autoconf.mk')
617 self.config_autoconf = os.path.join(build_dir, 'include', 'config',
619 self.defconfig = os.path.join(build_dir, 'defconfig')
621 def get_cross_compile(self):
622 """Parse .config file and return CROSS_COMPILE.
625 A string storing the compiler prefix for the architecture.
626 Return a NULL string for architectures that do not require
627 compiler prefix (Sandbox and native build is the case).
628 Return None if the specified compiler is missing in your PATH.
629 Caller should distinguish '' and None.
633 for line in open(self.dotconfig):
634 m = self.re_arch.match(line)
638 m = self.re_cpu.match(line)
646 if arch == 'arm' and cpu == 'armv8':
649 return CROSS_COMPILE.get(arch, None)
651 def parse_one_config(self, config, dotconfig_lines, autoconf_lines):
652 """Parse .config, defconfig, include/autoconf.mk for one config.
654 This function looks for the config options in the lines from
655 defconfig, .config, and include/autoconf.mk in order to decide
656 which action should be taken for this defconfig.
659 config: CONFIG name to parse.
660 dotconfig_lines: lines from the .config file.
661 autoconf_lines: lines from the include/autoconf.mk file.
664 A tupple of the action for this defconfig and the line
665 matched for the config.
667 not_set = '# %s is not set' % config
669 for line in autoconf_lines:
671 if line.startswith(config + '='):
677 for line in dotconfig_lines:
679 if line.startswith(config + '=') or line == not_set:
683 if new_val == not_set:
684 return (ACTION_NO_ENTRY, config)
686 return (ACTION_NO_ENTRY_WARN, config)
688 # If this CONFIG is neither bool nor trisate
689 if old_val[-2:] != '=y' and old_val[-2:] != '=m' and old_val != not_set:
690 # tools/scripts/define2mk.sed changes '1' to 'y'.
691 # This is a problem if the CONFIG is int type.
692 # Check the type in Kconfig and handle it correctly.
693 if new_val[-2:] == '=y':
694 new_val = new_val[:-1] + '1'
696 return (ACTION_NO_CHANGE if old_val == new_val else ACTION_MOVE,
699 def update_dotconfig(self):
700 """Parse files for the config options and update the .config.
702 This function parses the generated .config and include/autoconf.mk
703 searching the target options.
704 Move the config option(s) to the .config as needed.
707 defconfig: defconfig name.
710 Return a tuple of (updated flag, log string).
711 The "updated flag" is True if the .config was updated, False
712 otherwise. The "log string" shows what happend to the .config.
719 with open(self.dotconfig) as f:
720 dotconfig_lines = f.readlines()
722 with open(self.autoconf) as f:
723 autoconf_lines = f.readlines()
725 for config in self.configs:
726 result = self.parse_one_config(config, dotconfig_lines,
728 results.append(result)
732 for (action, value) in results:
733 if action == ACTION_MOVE:
734 actlog = "Move '%s'" % value
735 log_color = COLOR_LIGHT_GREEN
736 elif action == ACTION_NO_ENTRY:
737 actlog = "%s is not defined in Kconfig. Do nothing." % value
738 log_color = COLOR_LIGHT_BLUE
739 elif action == ACTION_NO_ENTRY_WARN:
740 actlog = "%s is not defined in Kconfig (suspicious). Do nothing." % value
741 log_color = COLOR_YELLOW
743 elif action == ACTION_NO_CHANGE:
744 actlog = "'%s' is the same as the define in Kconfig. Do nothing." \
746 log_color = COLOR_LIGHT_PURPLE
748 sys.exit("Internal Error. This should not happen.")
750 log += color_text(self.options.color, log_color, actlog) + '\n'
752 with open(self.dotconfig, 'a') as f:
753 for (action, value) in results:
754 if action == ACTION_MOVE:
755 f.write(value + '\n')
758 self.results = results
759 os.remove(self.config_autoconf)
760 os.remove(self.autoconf)
762 return (updated, suspicious, log)
764 def check_defconfig(self):
765 """Check the defconfig after savedefconfig
768 Return additional log if moved CONFIGs were removed again by
769 'make savedefconfig'.
774 with open(self.defconfig) as f:
775 defconfig_lines = f.readlines()
777 for (action, value) in self.results:
778 if action != ACTION_MOVE:
780 if not value + '\n' in defconfig_lines:
781 log += color_text(self.options.color, COLOR_YELLOW,
782 "'%s' was removed by savedefconfig.\n" %
789 """A slot to store a subprocess.
791 Each instance of this class handles one subprocess.
792 This class is useful to control multiple threads
793 for faster processing.
796 def __init__(self, configs, options, progress, devnull, make_cmd, reference_src_dir):
797 """Create a new process slot.
800 configs: A list of CONFIGs to move.
801 options: option flags.
802 progress: A progress indicator.
803 devnull: A file object of '/dev/null'.
804 make_cmd: command name of GNU Make.
805 reference_src_dir: Determine the true starting config state from this
808 self.options = options
809 self.progress = progress
810 self.build_dir = tempfile.mkdtemp()
811 self.devnull = devnull
812 self.make_cmd = (make_cmd, 'O=' + self.build_dir)
813 self.reference_src_dir = reference_src_dir
814 self.parser = KconfigParser(configs, options, self.build_dir)
815 self.state = STATE_IDLE
816 self.failed_boards = set()
817 self.suspicious_boards = set()
820 """Delete the working directory
822 This function makes sure the temporary directory is cleaned away
823 even if Python suddenly dies due to error. It should be done in here
824 because it is guaranteed the destructor is always invoked when the
825 instance of the class gets unreferenced.
827 If the subprocess is still running, wait until it finishes.
829 if self.state != STATE_IDLE:
830 while self.ps.poll() == None:
832 shutil.rmtree(self.build_dir)
834 def add(self, defconfig):
835 """Assign a new subprocess for defconfig and add it to the slot.
837 If the slot is vacant, create a new subprocess for processing the
838 given defconfig and add it to the slot. Just returns False if
839 the slot is occupied (i.e. the current subprocess is still running).
842 defconfig: defconfig name.
845 Return True on success or False on failure
847 if self.state != STATE_IDLE:
850 self.defconfig = defconfig
852 self.current_src_dir = self.reference_src_dir
857 """Check the status of the subprocess and handle it as needed.
859 Returns True if the slot is vacant (i.e. in idle state).
860 If the configuration is successfully finished, assign a new
861 subprocess to build include/autoconf.mk.
862 If include/autoconf.mk is generated, invoke the parser to
863 parse the .config and the include/autoconf.mk, moving
864 config options to the .config as needed.
865 If the .config was updated, run "make savedefconfig" to sync
866 it, update the original defconfig, and then set the slot back
870 Return True if the subprocess is terminated, False otherwise
872 if self.state == STATE_IDLE:
875 if self.ps.poll() == None:
878 if self.ps.poll() != 0:
880 elif self.state == STATE_DEFCONFIG:
881 if self.reference_src_dir and not self.current_src_dir:
882 self.do_savedefconfig()
885 elif self.state == STATE_AUTOCONF:
886 if self.current_src_dir:
887 self.current_src_dir = None
890 self.do_savedefconfig()
891 elif self.state == STATE_SAVEDEFCONFIG:
892 self.update_defconfig()
894 sys.exit("Internal Error. This should not happen.")
896 return True if self.state == STATE_IDLE else False
898 def handle_error(self):
899 """Handle error cases."""
901 self.log += color_text(self.options.color, COLOR_LIGHT_RED,
902 "Failed to process.\n")
903 if self.options.verbose:
904 self.log += color_text(self.options.color, COLOR_LIGHT_CYAN,
905 self.ps.stderr.read())
908 def do_defconfig(self):
909 """Run 'make <board>_defconfig' to create the .config file."""
911 cmd = list(self.make_cmd)
912 cmd.append(self.defconfig)
913 self.ps = subprocess.Popen(cmd, stdout=self.devnull,
914 stderr=subprocess.PIPE,
915 cwd=self.current_src_dir)
916 self.state = STATE_DEFCONFIG
918 def do_autoconf(self):
919 """Run 'make include/config/auto.conf'."""
921 self.cross_compile = self.parser.get_cross_compile()
922 if self.cross_compile is None:
923 self.log += color_text(self.options.color, COLOR_YELLOW,
924 "Compiler is missing. Do nothing.\n")
928 cmd = list(self.make_cmd)
929 if self.cross_compile:
930 cmd.append('CROSS_COMPILE=%s' % self.cross_compile)
931 cmd.append('KCONFIG_IGNORE_DUPLICATES=1')
932 cmd.append('include/config/auto.conf')
933 self.ps = subprocess.Popen(cmd, stdout=self.devnull,
934 stderr=subprocess.PIPE,
935 cwd=self.current_src_dir)
936 self.state = STATE_AUTOCONF
938 def do_savedefconfig(self):
939 """Update the .config and run 'make savedefconfig'."""
941 (updated, suspicious, log) = self.parser.update_dotconfig()
943 self.suspicious_boards.add(self.defconfig)
946 if not self.options.force_sync and not updated:
950 self.log += color_text(self.options.color, COLOR_LIGHT_GREEN,
951 "Syncing by savedefconfig...\n")
953 self.log += "Syncing by savedefconfig (forced by option)...\n"
955 cmd = list(self.make_cmd)
956 cmd.append('savedefconfig')
957 self.ps = subprocess.Popen(cmd, stdout=self.devnull,
958 stderr=subprocess.PIPE)
959 self.state = STATE_SAVEDEFCONFIG
961 def update_defconfig(self):
962 """Update the input defconfig and go back to the idle state."""
964 log = self.parser.check_defconfig()
966 self.suspicious_boards.add(self.defconfig)
968 orig_defconfig = os.path.join('configs', self.defconfig)
969 new_defconfig = os.path.join(self.build_dir, 'defconfig')
970 updated = not filecmp.cmp(orig_defconfig, new_defconfig)
973 self.log += color_text(self.options.color, COLOR_LIGHT_BLUE,
974 "defconfig was updated.\n")
976 if not self.options.dry_run and updated:
977 shutil.move(new_defconfig, orig_defconfig)
980 def finish(self, success):
981 """Display log along with progress and go to the idle state.
984 success: Should be True when the defconfig was processed
985 successfully, or False when it fails.
987 # output at least 30 characters to hide the "* defconfigs out of *".
988 log = self.defconfig.ljust(30) + '\n'
990 log += '\n'.join([ ' ' + s for s in self.log.split('\n') ])
991 # Some threads are running in parallel.
992 # Print log atomically to not mix up logs from different threads.
993 print >> (sys.stdout if success else sys.stderr), log
996 if self.options.exit_on_error:
997 sys.exit("Exit on error.")
998 # If --exit-on-error flag is not set, skip this board and continue.
999 # Record the failed board.
1000 self.failed_boards.add(self.defconfig)
1003 self.progress.show()
1004 self.state = STATE_IDLE
1006 def get_failed_boards(self):
1007 """Returns a set of failed boards (defconfigs) in this slot.
1009 return self.failed_boards
1011 def get_suspicious_boards(self):
1012 """Returns a set of boards (defconfigs) with possible misconversion.
1014 return self.suspicious_boards - self.failed_boards
1018 """Controller of the array of subprocess slots."""
1020 def __init__(self, configs, options, progress, reference_src_dir):
1021 """Create a new slots controller.
1024 configs: A list of CONFIGs to move.
1025 options: option flags.
1026 progress: A progress indicator.
1027 reference_src_dir: Determine the true starting config state from this
1030 self.options = options
1032 devnull = get_devnull()
1033 make_cmd = get_make_cmd()
1034 for i in range(options.jobs):
1035 self.slots.append(Slot(configs, options, progress, devnull,
1036 make_cmd, reference_src_dir))
1038 def add(self, defconfig):
1039 """Add a new subprocess if a vacant slot is found.
1042 defconfig: defconfig name to be put into.
1045 Return True on success or False on failure
1047 for slot in self.slots:
1048 if slot.add(defconfig):
1052 def available(self):
1053 """Check if there is a vacant slot.
1056 Return True if at lease one vacant slot is found, False otherwise.
1058 for slot in self.slots:
1064 """Check if all slots are vacant.
1067 Return True if all the slots are vacant, False otherwise.
1070 for slot in self.slots:
1075 def show_failed_boards(self):
1076 """Display all of the failed boards (defconfigs)."""
1078 output_file = 'moveconfig.failed'
1080 for slot in self.slots:
1081 boards |= slot.get_failed_boards()
1084 boards = '\n'.join(boards) + '\n'
1085 msg = "The following boards were not processed due to error:\n"
1087 msg += "(the list has been saved in %s)\n" % output_file
1088 print >> sys.stderr, color_text(self.options.color, COLOR_LIGHT_RED,
1091 with open(output_file, 'w') as f:
1094 def show_suspicious_boards(self):
1095 """Display all boards (defconfigs) with possible misconversion."""
1097 output_file = 'moveconfig.suspicious'
1099 for slot in self.slots:
1100 boards |= slot.get_suspicious_boards()
1103 boards = '\n'.join(boards) + '\n'
1104 msg = "The following boards might have been converted incorrectly.\n"
1105 msg += "It is highly recommended to check them manually:\n"
1107 msg += "(the list has been saved in %s)\n" % output_file
1108 print >> sys.stderr, color_text(self.options.color, COLOR_YELLOW,
1111 with open(output_file, 'w') as f:
1114 class ReferenceSource:
1116 """Reference source against which original configs should be parsed."""
1118 def __init__(self, commit):
1119 """Create a reference source directory based on a specified commit.
1122 commit: commit to git-clone
1124 self.src_dir = tempfile.mkdtemp()
1125 print "Cloning git repo to a separate work directory..."
1126 subprocess.check_output(['git', 'clone', os.getcwd(), '.'],
1128 print "Checkout '%s' to build the original autoconf.mk." % \
1129 subprocess.check_output(['git', 'rev-parse', '--short', commit]).strip()
1130 subprocess.check_output(['git', 'checkout', commit],
1131 stderr=subprocess.STDOUT, cwd=self.src_dir)
1134 """Delete the reference source directory
1136 This function makes sure the temporary directory is cleaned away
1137 even if Python suddenly dies due to error. It should be done in here
1138 because it is guaranteed the destructor is always invoked when the
1139 instance of the class gets unreferenced.
1141 shutil.rmtree(self.src_dir)
1144 """Return the absolute path to the reference source directory."""
1148 def move_config(configs, options):
1149 """Move config options to defconfig files.
1152 configs: A list of CONFIGs to move.
1153 options: option flags
1155 if len(configs) == 0:
1156 if options.force_sync:
1157 print 'No CONFIG is specified. You are probably syncing defconfigs.',
1159 print 'Neither CONFIG nor --force-sync is specified. Nothing will happen.',
1161 print 'Move ' + ', '.join(configs),
1162 print '(jobs: %d)\n' % options.jobs
1165 reference_src = ReferenceSource(options.git_ref)
1166 reference_src_dir = reference_src.get_dir()
1168 reference_src_dir = None
1170 if options.defconfigs:
1171 defconfigs = [line.strip() for line in open(options.defconfigs)]
1172 for i, defconfig in enumerate(defconfigs):
1173 if not defconfig.endswith('_defconfig'):
1174 defconfigs[i] = defconfig + '_defconfig'
1175 if not os.path.exists(os.path.join('configs', defconfigs[i])):
1176 sys.exit('%s - defconfig does not exist. Stopping.' %
1179 defconfigs = get_all_defconfigs()
1181 progress = Progress(len(defconfigs))
1182 slots = Slots(configs, options, progress, reference_src_dir)
1184 # Main loop to process defconfig files:
1185 # Add a new subprocess into a vacant slot.
1186 # Sleep if there is no available slot.
1187 for defconfig in defconfigs:
1188 while not slots.add(defconfig):
1189 while not slots.available():
1190 # No available slot: sleep for a while
1191 time.sleep(SLEEP_TIME)
1193 # wait until all the subprocesses finish
1194 while not slots.empty():
1195 time.sleep(SLEEP_TIME)
1198 slots.show_failed_boards()
1199 slots.show_suspicious_boards()
1203 cpu_count = multiprocessing.cpu_count()
1204 except NotImplementedError:
1207 parser = optparse.OptionParser()
1209 parser.add_option('-c', '--color', action='store_true', default=False,
1210 help='display the log in color')
1211 parser.add_option('-d', '--defconfigs', type='string',
1212 help='a file containing a list of defconfigs to move')
1213 parser.add_option('-n', '--dry-run', action='store_true', default=False,
1214 help='perform a trial run (show log with no changes)')
1215 parser.add_option('-e', '--exit-on-error', action='store_true',
1217 help='exit immediately on any error')
1218 parser.add_option('-s', '--force-sync', action='store_true', default=False,
1219 help='force sync by savedefconfig')
1220 parser.add_option('-H', '--headers-only', dest='cleanup_headers_only',
1221 action='store_true', default=False,
1222 help='only cleanup the headers')
1223 parser.add_option('-j', '--jobs', type='int', default=cpu_count,
1224 help='the number of jobs to run simultaneously')
1225 parser.add_option('-r', '--git-ref', type='string',
1226 help='the git ref to clone for building the autoconf.mk')
1227 parser.add_option('-v', '--verbose', action='store_true', default=False,
1228 help='show any build errors as boards are built')
1229 parser.usage += ' CONFIG ...'
1231 (options, configs) = parser.parse_args()
1233 if len(configs) == 0 and not options.force_sync:
1234 parser.print_usage()
1237 # prefix the option name with CONFIG_ if missing
1238 configs = [ config if config.startswith('CONFIG_') else 'CONFIG_' + config
1239 for config in configs ]
1241 check_top_directory()
1243 if not options.cleanup_headers_only:
1244 check_clean_directory()
1245 update_cross_compile(options.color)
1246 move_config(configs, options)
1249 cleanup_headers(configs, options)
1250 cleanup_extra_options(configs, options)
1252 if __name__ == '__main__':