Re: [Edk2 Dev] clean build problem?

Subject: Re: [Edk2 Dev] clean build problem?

From: "Andrew J. Fish" <afish@apple.com>

To: dev@edk2.tianocore.org

Date: 2009-06-18 17:49:53

Ken,

I like the idea of automating the checking. 

Andrew Fish





On Jun 18, 2009, at 12:02 AM, Jian J Wang wrote:

Ken,
 
Good idea. But since timestamp check is not reliable, there should be always a way to do force remove or re-generation.
 

Cheers, 
Javen

 


From: Lu ken [mailto:ken.lu@intel.com] 
Sent: 2009618 14:53
To: dev@edk2.tianocore.org
Subject: RE: [Edk2 Dev] clean build problem?

Hi, Andrew, Javen and Jordan:
         I just read too long email thread, the issue addressed here is caused by build tool has no dependency checking with .cache/build.db and mismatch information confuse build tools. This issue only maybe exist when build tool is in development phase.
 
         I do *not* think this dependency checking need to be explicitly addressed in edk2s make file or script file. Because it is not the problem raised by edk2 source or script.
         I also do *not* think this dependency checking need be addressed in GNUMakefile in build tool project, the reason is the building for build tools need *not* affect edk2 project. They are quit in different path, I give you example, In my linux machine, I have two trees one is for working and another is work validation. So if tool is updated, the .cache/build.db in two trees all need to be regenerated.
 
         I am thinking this dependency checking need to be done by build tools python code by itself. Currently, build tool provide an options -e to regenerate the build.db for user. But maybe we can make it more intelligent that build tool check the date of build.db when building is started,
a) If build tool is in source:
if build.dbs date is older than *.py in under the latest update time of build, autogen and Workspace folder.  Build tool will regenerate build.db
         b) If build tool is in frozen
                   if build.dbs date is older than build.exe, build tool will regenerate build.db
 
         I guess no more than 20 line python could solve this problem in BaseTools\Source\Python\Workspace\WorkspaceDatabase.py line 2050.
        
 
         Thanks
Best Regard
K
 

From: Andrew J. Fish [mailto:afish@apple.com] 
Sent: 2009
618 13:42
To: dev@edk2.tianocore.org
Subject: Re: [Edk2 Dev] clean build problem?
 
I don't like the developer having to update the number by hand. We should automate this feature in the makefile/GNUmakefile.
 
Andrew Fish
 
 
 
 
 
On Jun 17, 2009, at 10:33 PM, Jian J Wang wrote:


Andrew,
 
For this situation, I think Jordan's idea is good. We could add a version in build.db to see if the build tools is compatible with the build.db or not. If not, the build.db will be removed and re-generated. The only thing needing to care about is  the one who updated the tools code must remember to update the version number.
 

Cheers, 
Javen

 
 

From: Andrew J. Fish [mailto:afish@apple.com] 
Sent: 2009
618 13:25
To: dev@edk2.tianocore.org
Subject: Re: [Edk2 Dev] clean build problem?

Javen,
 
Carl's change was in the python parsing code for the inf, dsc and dec files. This is why the caching of the $(WORKSPACE)/Conf/.cache/build.db caused him so much trouble. 
 
Carl is experimenting with adding enum support to the PCD.  So a dec file can define enumeration values and use enumerated types for the defaults. C code would get access to the enum via AutoGen.h. The inf or dsc file can use the enum for the value set. So it makes the PCD setting more human readable, and removed the need for a duplicate copy in C code that could get out of sync. 
 
Andrew Fish
 
 
 
 
 
On Jun 17, 2009, at 10:16 PM, Jian J Wang wrote:


Andrew,
 
So this has nothing to do with the sqlite database file ($(WORKSPACE/Conf/.cache/build.db). This file doesn't cache .pyc file. It's cache meta files, inf, dsc and dec files. I think pre-generating the *.pyc file won't speed up much the build. But forcing pre-generating the *.pyc will solve the sync issue between *.py and *.pyc, if Python failed this during run time.
 

Cheers, 
Javen

 
 

From: Andrew J. Fish [mailto:afish@apple.com] 
Sent: 2009
618 13:06
To: dev@edk2.tianocore.org
Subject: Re: [Edk2 Dev] clean build problem?

Javen,
 
We solved Carl's build break quickly, but it took us a lot longer to figure out how Carl could have tested his fix and still checked in bad python code.  His system kept running, even though all the systems that pulled from svn failed. 
 
The python interpreter will generate *.pyc files when the python code runs automatically.  Thus I'm not sure given the way the build system works how much speed up you would really get from pre-generating the *.pyc files.
 
Andrew Fish
 
 
 
 
 
On Jun 17, 2009, at 9:49 PM, Jian J Wang wrote:


Andrew,
 
You mean the cache triggered a bad python code which will not run without a cache (or vice versa, the same anyway). So Carl has solved the problem, right? If so, then the problem is when to remove or sync cache or .pyc files?
 
One of possible way is to add target in python makefile to compile *.py files to *.pyc files, if it's running in non-Windows platform. In Windows platform, it will still do freezing.
 

Cheers, 
Javen

 
 

From: Andrew J. Fish [mailto:afish@apple.com] 
Sent: 2009
618 11:29
To: dev@edk2.tianocore.org
Subject: Re: [Edk2 Dev] clean build problem?

Javen,
 
I don't have the stack trace handy. It looks like caching is the problem. The problem is quite simple. If you have a cache your path through the python code is different than if you don't have a cache. Carl made a change that was basically a syntax error into the python. Since the the bad python code never runs if the cache exists his version never crashed. When he checked it in and folks did not have Carl's version of cache the bad python code was executed. 
 
So Carl did his python development. Then he did a clean of the build tools before he checked in  (we know now that it does not really do anything currently) and tested before checking in.  When other developers pulled his change their version crashed with the python syntax error Carl had checked in. 
 
Andrew Fish
 
 
 
 
 
On Jun 17, 2009, at 8:18 PM, Jian J Wang wrote:


Andrew,
 
I think Jordan's idea is to check the version of current build tools and remove any intermediate files if there's a big change in tools or the tools's version is changed.
 
Do you have any log file or stack trace? We may help to find out the root cause. It's hard to know exactly what's going on per the Carl's description. It might not be the out-of-sync caused this problem.
 

Cheers, 
Javen

 
 

From: Andrew J. Fish [mailto:afish@apple.com] 
Sent: 2009
618 11:04
To: dev@edk2.tianocore.org
Subject: Re: [Edk2 Dev] clean build problem?

Jordan,
 
The removal of the .pyc files is not really needed for correct operation, but is good hygiene for the build system to clean intermediate files generated by the build. They have the tendency to get checked in by accident and they make diff'ing and some source control operations more complex than it should be. 
 
I don't understand your logic on the Conf directory. It is OK to store intermediate data that can get out of sync in "Some One Else's" part of the tree, but it is not OK to remove that data to make sure it is in sync.
 
We have a real world bug here. Carl made a change to the python in our local repository. He did a make clean in the tools directory and tested his change. Every thing worked OK, so he checked it in.  When we pulled Carl's change python would crash on our machines.  Bugs like this waist a lot of developer time for no reason. 
 
We have not got around to getting cx_freeze working on OS X yet, so actually have the python checked into the edk2 and run it directly. 
 
 Andrew Fish
 
 
 
 
 
On Jun 17, 2009, at 7:25 PM, Jordan Justen wrote:


I don't think we need to be concerned about cleaning .pyc files:

http://www.python.org/doc/1.5.1p1/tut/node43.html
The modification time of the version of "spam.py" used to create "spam.pyc" is recorded in "spam.pyc", and the file is ignored if these don't match.


'make clean' for the BaseTools should not clean out the Conf directory either.  But, if upgrading the BaseTools results in the Conf/.cache not working properly, then I think this is a BaseTools bug.

Perhaps there should be a version stored in Conf/.cache, and the build process can clear Conf/.cache out if it doesn't match the current expected version for the BaseTools.

-Jordan

On Wed, Jun 17, 2009 at 7:11 PM, Andrew J. Fish <afish@apple.com> wrote:
Javen,

"make clean" for the tools removes all the *.pyc files. It looks like
it does not remove the $(WORKSPACE)/Conf/.cache/build.db. I think this
is the problem Carl was running into.

cat BaseTools/Source/Python/GNUmakefile
all:

clean:
       find . -name '*.pyc' -exec rm '{}' ';'

We don't have support for a "frozen binary" on our system, so we
always run the python via the interpreter.

It looks like the Makefile has the same issue.

Also given this issue cropped up, it looks like there is a dependancy
issue, but given the way *.pyc files work I guess you could argue that
if you change the python code you should do a clean to make sure your
fix is truly working. At least before  you check in a python change
into source control.

Andrew Fish





On Jun 17, 2009, at 6:29 PM, Jian J Wang wrote:

> Carl,
>
> "clean build" always means the EDK-II code not the build tools code.
> "build cleanll" will never clean the python byte-code generated from
> tools source code, if you run the build from python script not the
> frozen binary. I think python interepter will update the .pyc file
> if the corresponding .py file is updated.
>
> If you doublt the database out-of-sync, please try to add "-e"
> switch to build command or remove $(WORKSPACE)/Conf/.cache/build.db
> directly and build again.
>
>
> Cheers,
> Javen
>
> -----Original Message-----
> From: Carl Norum [mailto:carl.norum@apple.com]
> Sent: 2009
618 08:43
> To: dev@edk2.tianocore.org
> Subject: [Edk2 Dev] clean build problem?
>
> Hi folks,
>
> I've been working on this Pcd enumeration stuff, and it's been going
> smoothly - up until today when I checked it in and the build proceeded
> to fail for everyone who synced my changes.  A clean build still works
> on my development machine though, but not on another machine I synced
> the code up on.  That's a little scary...
>
> I do see that cleaning the build results doesn't clean my python
> object code, but even doing that didn't line up my results.  Is the
> sql database generated by the tools left around somewhere, that my
> development machine is succeeding on builds, but other machines arent?
>
> --
> Carl Norum
> Firmware Engineer
> Apple, Inc.
> Cupertino, CA
>
> ------------------------------------------------------
> https://edk2.tianocore.org/ds/viewMessage.do?dsForumId=135&dsMessageId=45007
>
> To unsubscribe from this discussion, please e-mail [unsubscribeURL]
>
> ------------------------------------------------------
> https://edk2.tianocore.org/ds/viewMessage.do?dsForumId=135&dsMessageId=45016
>
> To unsubscribe from this discussion, please e-mail [unsubscribeURL]

------------------------------------------------------

To unsubscribe from this discussion, please e-mail [unsubscribeURL]