Given the makefiles are auto-generated they should have full knowledge of all the dependencies. If they don't have full knowledge then that seems to imply they are not optimal?
It seems the DSC and FDF files have to parsed first and everything is gated on that. Then the INF files get parsed and the makefiles for the components get generated, and if needed the AutoGen.* files get generated. At that point I guess you can call the makefile. Is that the basic issue?
Could you give a high level outline on what happens in parallel (today and with the new proposal). I wonder if it is possible to build a parallel top level makefile the runs in parallel with the Python? Given the AutoGen is not triggered by the makefiles this may not be possible. I can try and track down a make expert and ask some questions.
So in terms of optimizing the makefiles does the following work from a clean build?
build -n 5 genmake
build -n 5 genc
make -j 5 GNUmakeifle
So to help me understand does the build today basically perform these steps, and what you want to change is adding that ability for the Python thread to call a module makefile when the dependencies are met (makefile generated and AutoGen.* created)?
I think it might be useful to optimize the makefile structure to see how much time is spent in make with the optimized makefiles vs. calling from Python. It would be a good way to set the performance goal for the Python code.
On Mar 3, 2012, at 10:59 AM, Kinney, Michael D wrote:
I think there are two types of optimization to consider here. 1) Proper use of makefiles for best performance. 2) Performance when the build n THREADNUMBER switch is used for > 1 THREADNUMBER to do parallel builds. We think Python is a better place to evaluate all the components that need to be build and how to choose what tasks get performed and when they get performed to optimize the build performance for multiple threads. If you have suggestions on better ways to produce makefiles in EDK II for individual Library and Module builds, then please provide detailed feedback for the autogen changes.
This sounds more like an issue in the recipes in the makefile. I don't see how the conclusion can be do more work in Python, if the makefiles have not even been optimized. Seems like the Python is the slowest part of the build. I would imagine an enormous amount of resources have already been expended optimizing make/nmake for build system much bigger and more complicated then ours. Thus we should expend some energy optimizing the makefiles.
As a data point we found the makefiles generated in the EDK horribly inefficient. A co-worker of mine ported the EDK build system to make and was able to optimize the generated makefiles to get a massive increase in build performance.
On Mar 2, 2012, at 9:57 PM, Amaya, Elmer A wrote:
As we develop new methods and algorithms that can speed up the time it takes to build modules, in this case, being able to start building drivers after a set of libraries have been built, we propose having the python build tool call nmake/make on the individual module Makefiles directly. Currently, all of the libraries have to be built prior to building the driver modules the python build tool calls nmake/make on the top level Makefile, this Makefile calls nmake/make on the architectural Makefile (the Makefile under IA32/X64, etc.). The architectural Makefiles calls nmake/make on the individual module Makefiles. Were proposing removing the top level and architectural level Makefiles from the Build tree and integrating the work that those makefiles perform into the BUILD.EXE tool itself. I would like to get your feedback/comments no later than end of day March 15, 2012. If proposal is accepted, I will follow up with detail proposal and patch implementation for your review. Sr. Technical Marketing Engineer
Virtualization & Cloud Management Using Capacity Planning
Cloud computing makes use of virtualization - but cloud computing
also focuses on allowing computing to be delivered as a service.http://www.accelacomm.com/jaw/sfnl/114/51521223/_______________________________________________
edk2-devel mailing firstname.lastname@example.org://lists.sourceforge.net/lists/listinfo/edk2-devel