Re: [edk2] [edk2-buildtools] Proposal: Remove top level and architectural level Makefiles from the Build tree

Subject: Re: [edk2] [edk2-buildtools] Proposal: Remove top level and architectural level Makefiles from the Build tree

From: Andrew Fish <afish@apple.com>

To: Burt Silverman <burtms@gmail.com>

Date: 2012-03-05 17:16:33


On Mar 5, 2012, at 6:03 AM, Burt Silverman wrote:

What Mike is saying makes much sense to me: you do not want to break perfectly good Makefiles, but rather, you want to figure out the optimal sequence of calling those makefiles from the layer above, for the specific purpose at hand.


The current top level makefile is a template embedded in the Python code, but the lower level makefiles are actually generated by Python code to to match the build. So in either case Python code is changing. 

Andrew says:
>>t looks like the GNU make from Cygwin is clunky and does not get the speedup.

I would be quite surprised if GNU make has been modified for Cygwin. The problem with Cygwin is more general. Sometimes turning off anti-virus helps, and I am not sure if that is the entire issue. But something related to starting shells is very very slow with Cygwin. Now let me report the most bizarre behavior I see with Cygwin. The scenario is mixed use: in one Windows Command Prompt window, I am running build. At the same time, in a Cygwin terminal, I am running lots of commands like find . -name "*.c"| xargs grep somethingOrOther. Also between builds I may "rm -rf ." somewhere in the build tree as a way of doing a super clean build. Only using Cygwin because I am more familiar with the UNIX commands syntax than the Windows equivalents Invariably, I end up in a state characterized by:

1. build becomes much slower
2. My disk light is blinking at subsecond intervals, and never stops doing that
3. Rebooting Windows is the only fix for 1 and 2 that I have discovered. I have failed to find any Windows service that I could stop or restart to help fix the problem. I suppose something gets whacked in the kernel, but I am no expert.


I agree it looks like the generic Cygwin port is the issue. The thread model for Windows is different than the thread model for POSIX. Also file system accesses seems slower than it should be. 

I was trying to point out the large performance difference between the OS X make and the Cygwin make. 

Burt
On Sun, Mar 4, 2012 at 8:38 PM, Kinney, Michael D <michael.d.kinney@intel.com> wrote:

Andrew,

 

Top level makefiles have two major components today. 

 

build_libraries with a call to make for each lib

build_modules with a call to make for each module. 

 

This is not optimal for both full and incremental builds.  There are many scenarios where we could build a module earlier if its libs have been satisfied.  Always waiting for all the libs is not ideal.  Also, some libs are bigger than others.  In order to get optimal build performance on multiple threads, you want to schedule the building of libs and modules so all threads will complete around the same time.  There is more context in the Python code to understand relative complexity of each lib and module, and potentially profile build times and save the info to help scheduling for future builds.  These are all concepts that cannot be done in a makefile alone.

 

Thanks,

 

Mike

 

 

 

From: Andrew Fish [mailto:afish@apple.com]
Sent: Sunday, March 04, 2012 1:38 PM
To: edk2-devel@lists.sourceforge.net
Cc: edk2-buildtools-devel@lists.sourceforge.net
Subject: Re: [edk2-buildtools] [edk2] Proposal: Remove top level and architectural level Makefiles from the Build tree

 

Mike,

 

Given the makefiles are auto-generated they should have full knowledge of all the dependencies. If they don't have full knowledge then that seems to imply they are not optimal? 

 

It seems the DSC and FDF files have to parsed first and everything is gated on that. Then the INF files get parsed and the makefiles for the components get generated, and if needed the AutoGen.* files get generated. At that point I guess you can call the makefile. Is that the basic issue? 

 

Could you give a high level outline on what happens in parallel (today and with the new proposal). I wonder if it is possible to build a parallel top level makefile the runs in parallel with the Python? Given the AutoGen is not triggered by the makefiles this may not be possible. I can try and track down a make expert and ask some questions. 

 

So in terms of optimizing the makefiles does the following work from a clean build? 

build -n 5 genmake

build -n 5 genc

make -j 5 GNUmakeifle 

 

So to help me understand does the build today basically perform these steps, and what you want to change is adding that ability for the Python thread to call a module makefile  when the dependencies are met (makefile generated and AutoGen.* created)? 

 

I think it might be useful to optimize the makefile structure to see how much time is spent in make with the optimized makefiles vs. calling from Python. It would be a good way to set the performance goal for the Python code. 

 

Thanks,

 

Andrew

 

On Mar 3, 2012, at 10:59 AM, Kinney, Michael D wrote:



Andrew,

 

I think there are two types of optimization to consider here. 

 

1)      Proper use of makefiles for best performance.

2)      Performance when the build n THREADNUMBER switch is used for > 1 THREADNUMBER to do parallel builds.  We think Python is a better place to evaluate all the components that need to be build and how to choose what tasks get performed and when they get performed to optimize the build performance for multiple threads.

 

If you have suggestions on better ways to produce makefiles in EDK II for individual Library and Module builds, then please provide detailed feedback for the autogen changes.

 

Thanks,

 

Mike

 

From: Andrew Fish [mailto:afish@apple.com] 
Sent: Saturday, March 03, 2012 9:44 AM
To: edk2-devel@lists.sourceforge.net
Cc: edk2-buildtools-devel@lists.sourceforge.net
Subject: Re: [edk2] Proposal: Remove top level and architectural level Makefiles from the Build tree

 

Elmer,

 

This sounds more like an issue in the recipes in the makefile.  I don't see how the conclusion can be do more work in Python, if the makefiles have not even been optimized. Seems like the Python is the slowest part of the build. I would imagine an enormous amount of resources have already been expended optimizing make/nmake for build system much bigger and more complicated then ours. Thus we should expend some energy optimizing the makefiles. 

 

As a data point we found the makefiles generated in the EDK horribly inefficient. A co-worker of mine ported the EDK build system to make and was able to optimize the generated makefiles to get a massive increase in build performance. 

 

Andrew Fish

 

 




 

On Mar 2, 2012, at 9:57 PM, Amaya, Elmer A wrote:




Hi All,

 

As we develop new methods and algorithms that can speed up the time it takes to build modules, in this case, being able to start building drivers after a set of libraries have been built, we  propose having the python build tool call nmake/make on the individual module Makefiles directly.  Currently, all of the libraries have to be built prior to building the driver modules the python build tool calls nmake/make on the top level Makefile, this Makefile calls nmake/make on the architectural Makefile (the Makefile under IA32/X64, etc.).  The architectural Makefiles calls nmake/make on the individual module Makefiles.

 

Were proposing removing the top level and architectural level Makefiles from the Build tree and integrating the work that those makefiles perform into the BUILD.EXE tool itself.

 

I would like to get your feedback/comments no later than end of day March 15, 2012.  If proposal is accepted, I will follow up with detail proposal and patch implementation for your review.

 

Thanks,

Elmer Amaya

Sr. Technical Marketing Engineer

Intel Corporation

 

------------------------------------------------------------------------------
Virtualization & Cloud Management Using Capacity Planning
Cloud computing makes use of virtualization - but cloud computing 
also focuses on allowing computing to be delivered as a service.
http://www.accelacomm.com/jaw/sfnl/114/51521223/_______________________________________________
edk2-devel mailing list
edk2-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/edk2-devel

 

------------------------------------------------------------------------------
Virtualization & Cloud Management Using Capacity Planning
Cloud computing makes use of virtualization - but cloud computing 
also focuses on allowing computing to be delivered as a service.
http://www.accelacomm.com/jaw/sfnl/114/51521223/_______________________________________________
edk2-devel mailing list
edk2-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/edk2-devel

 


------------------------------------------------------------------------------
Try before you buy = See our experts in action!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-dev2
_______________________________________________
edk2-buildtools-devel mailing list
edk2-buildtools-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/edk2-buildtools-devel