Yikes, I did a quick "Bing" and I see that nmake does not support parallel build. And it looks like the Cygwin version of make does not have good performance. So it is only make on OS X and Linux that would benefit from parallelizing the makefiles. Thus the least common denominator is to do it in Python.
So in the new scheme what is going to happen to the top level makefile? I notice there is some hard coded cruft in them like the run target that points to NT32. Also why is clean so slow? It looks like clean parses a bunch of files, before it throws all that info away and deletes directories.
I'd like to ask that there be an option to build to construct all the auto-gen and makefiles, but not call the makefiles. This would let us play around with having an option of making parallel makefiles. Basically I think the list of objects to link are a list of dependencies so we should be able to work backwards from their.
On Mar 4, 2012, at 5:38 PM, Kinney, Michael D wrote:
Top level makefiles have two major components today.
build_libraries with a call to make for each lib
build_modules with a call to make for each module.
This is not optimal for both full and incremental builds. There are many scenarios where we could build a module earlier if its libs have been satisfied. Always waiting for all the libs is not ideal. Also, some libs are bigger than others. In order to get optimal build performance on multiple threads, you want to schedule the building of libs and modules so all threads will complete around the same time. There is more context in the Python code to understand relative complexity of each lib and module, and potentially profile build times and save the info to help scheduling for future builds. These are all concepts that cannot be done in a makefile alone.
Given the makefiles are auto-generated they should have full knowledge of all the dependencies. If they don't have full knowledge then that seems to imply they are not optimal?
It seems the DSC and FDF files have to parsed first and everything is gated on that. Then the INF files get parsed and the makefiles for the components get generated, and if needed the AutoGen.* files get generated. At that point I guess you can call the makefile. Is that the basic issue?
Could you give a high level outline on what happens in parallel (today and with the new proposal). I wonder if it is possible to build a parallel top level makefile the runs in parallel with the Python? Given the AutoGen is not triggered by the makefiles this may not be possible. I can try and track down a make expert and ask some questions.
So in terms of optimizing the makefiles does the following work from a clean build?
build -n 5 genmake
build -n 5 genc
make -j 5 GNUmakeifle
So to help me understand does the build today basically perform these steps, and what you want to change is adding that ability for the Python thread to call a module makefile when the dependencies are met (makefile generated and AutoGen.* created)?
I think it might be useful to optimize the makefile structure to see how much time is spent in make with the optimized makefiles vs. calling from Python. It would be a good way to set the performance goal for the Python code.
On Mar 3, 2012, at 10:59 AM, Kinney, Michael D wrote:
I think there are two types of optimization to consider here.
1)Proper use of makefiles for best performance.
2)Performance when the build n THREADNUMBER switch is used for > 1 THREADNUMBER to do parallel builds. We think Python is a better place to evaluate all the components that need to be build and how to choose what tasks get performed and when they get performed to optimize the build performance for multiple threads.
If you have suggestions on better ways to produce makefiles in EDK II for individual Library and Module builds, then please provide detailed feedback for the autogen changes.
This sounds more like an issue in the recipes in the makefile. I don't see how the conclusion can be do more work in Python, if the makefiles have not even been optimized. Seems like the Python is the slowest part of the build. I would imagine an enormous amount of resources have already been expended optimizing make/nmake for build system much bigger and more complicated then ours. Thus we should expend some energy optimizing the makefiles.
As a data point we found the makefiles generated in the EDK horribly inefficient. A co-worker of mine ported the EDK build system to make and was able to optimize the generated makefiles to get a massive increase in build performance.
On Mar 2, 2012, at 9:57 PM, Amaya, Elmer A wrote:
As we develop new methods and algorithms that can speed up the time it takes to build modules, in this case, being able to start building drivers after a set of libraries have been built, we propose having the python build tool call nmake/make on the individual module Makefiles directly. Currently, all of the libraries have to be built prior to building the driver modules the python build tool calls nmake/make on the top level Makefile, this Makefile calls nmake/make on the architectural Makefile (the Makefile under IA32/X64, etc.). The architectural Makefiles calls nmake/make on the individual module Makefiles.
Were proposing removing the top level and architectural level Makefiles from the Build tree and integrating the work that those makefiles perform into the BUILD.EXE tool itself.
I would like to get your feedback/comments no later than end of day March 15, 2012. If proposal is accepted, I will follow up with detail proposal and patch implementation for your review.