Re: [Edk2 Dev] clean build problem?

Subject: Re: [Edk2 Dev] clean build problem?

From: Jordan Justen <jljusten@gmail.com>

To: dev@edk2.tianocore.org

Date: 2009-06-18 08:16:44

Andrew,

You mentioned that the code had a Python syntax error, but it was not caught because the code path was not run which would import the Python source file.

In r1627 I added a test which will run py_compile.compile on each .py file under Source/Python.

This test will be run when GNU make is used under BaseTools.  (Ie, the 'make' command.)  It can also be run on it's own by running 'make test' under BaseTools.

This is only a Python syntax check, so functionality checks would require more BaseTools unit tests to be developed.

-Jordan

2009/6/17 Andrew J. Fish <afish@apple.com>
Javen,

I don't have the stack trace handy. It looks like caching is the problem. The problem is quite simple. If you have a cache your path through the python code is different than if you don't have a cache. Carl made a change that was basically a syntax error into the python. Since the the bad python code never runs if the cache exists his version never crashed. When he checked it in and folks did not have Carl's version of cache the bad python code was executed. 

So Carl did his python development. Then he did a clean of the build tools before he checked in  (we know now that it does not really do anything currently) and tested before checking in.  When other developers pulled his change their version crashed with the python syntax error Carl had checked in. 

Andrew Fish





On Jun 17, 2009, at 8:18 PM, Jian J Wang wrote:

Andrew,
 
I think Jordan's idea is to check the version of current build tools and remove any intermediate files if there's a big change in tools or the tools's version is changed.
 
Do you have any log file or stack trace? We may help to find out the root cause. It's hard to know exactly what's going on per the Carl's description. It might not be the out-of-sync caused this problem.
 

Cheers,
Javen

 


From: Andrew J. Fish [mailto:afish@apple.com]
Sent: 2009618 11:04
To: dev@edk2.tianocore.org
Subject: Re: [Edk2 Dev] clean build problem?

Jordan,

The removal of the .pyc files is not really needed for correct operation, but is good hygiene for the build system to clean intermediate files generated by the build. They have the tendency to get checked in by accident and they make diff'ing and some source control operations more complex than it should be. 

I don't understand your logic on the Conf directory. It is OK to store intermediate data that can get out of sync in "Some One Else's" part of the tree, but it is not OK to remove that data to make sure it is in sync.

We have a real world bug here. Carl made a change to the python in our local repository. He did a make clean in the tools directory and tested his change. Every thing worked OK, so he checked it in.  When we pulled Carl's change python would crash on our machines.  Bugs like this waist a lot of developer time for no reason. 

We have not got around to getting cx_freeze working on OS X yet, so actually have the python checked into the edk2 and run it directly. 

 Andrew Fish





On Jun 17, 2009, at 7:25 PM, Jordan Justen wrote:

I don't think we need to be concerned about cleaning .pyc files:

http://www.python.org/doc/1.5.1p1/tut/node43.html
The modification time of the version of "spam.py" used to create "spam.pyc" is recorded in "spam.pyc", and the file is ignored if these don't match.

'make clean' for the BaseTools should not clean out the Conf directory either.  But, if upgrading the BaseTools results in the Conf/.cache not working properly, then I think this is a BaseTools bug.

Perhaps there should be a version stored in Conf/.cache, and the build process can clear Conf/.cache out if it doesn't match the current expected version for the BaseTools.

-Jordan

On Wed, Jun 17, 2009 at 7:11 PM, Andrew J. Fish <afish@apple.com> wrote:
Javen,

"make clean" for the tools removes all the *.pyc files. It looks like
it does not remove the $(WORKSPACE)/Conf/.cache/build.db. I think this
is the problem Carl was running into.

cat BaseTools/Source/Python/GNUmakefile
all:

clean:
       find . -name '*.pyc' -exec rm '{}' ';'

We don't have support for a "frozen binary" on our system, so we
always run the python via the interpreter.

It looks like the Makefile has the same issue.

Also given this issue cropped up, it looks like there is a dependancy
issue, but given the way *.pyc files work I guess you could argue that
if you change the python code you should do a clean to make sure your
fix is truly working. At least before  you check in a python change
into source control.

Andrew Fish





On Jun 17, 2009, at 6:29 PM, Jian J Wang wrote:

> Carl,
>
> "clean build" always means the EDK-II code not the build tools code.
> "build cleanll" will never clean the python byte-code generated from
> tools source code, if you run the build from python script not the
> frozen binary. I think python interepter will update the .pyc file
> if the corresponding .py file is updated.
>
> If you doublt the database out-of-sync, please try to add "-e"
> switch to build command or remove $(WORKSPACE)/Conf/.cache/build.db
> directly and build again.
>
>
> Cheers,
> Javen
>
> -----Original Message-----
> From: Carl Norum [mailto:carl.norum@apple.com]
> Sent: 2009618 08:43
> To: dev@edk2.tianocore.org
> Subject: [Edk2 Dev] clean build problem?
>
> Hi folks,
>
> I've been working on this Pcd enumeration stuff, and it's been going
> smoothly - up until today when I checked it in and the build proceeded
> to fail for everyone who synced my changes.  A clean build still works
> on my development machine though, but not on another machine I synced
> the code up on.  That's a little scary...
>
> I do see that cleaning the build results doesn't clean my python
> object code, but even doing that didn't line up my results.  Is the
> sql database generated by the tools left around somewhere, that my
> development machine is succeeding on builds, but other machines arent?
>
> --
> Carl Norum
> Firmware Engineer
> Apple, Inc.
> Cupertino, CA
>
> ------------------------------------------------------
> https://edk2.tianocore.org/ds/viewMessage.do?dsForumId=135&dsMessageId=45007
>
> To unsubscribe from this discussion, please e-mail [unsubscribeURL]
>
> ------------------------------------------------------
> https://edk2.tianocore.org/ds/viewMessage.do?dsForumId=135&dsMessageId=45016
>
> To unsubscribe from this discussion, please e-mail [unsubscribeURL]

------------------------------------------------------
https://edk2.tianocore.org/ds/viewMessage.do?dsForumId=135&dsMessageId=45020

To unsubscribe from this discussion, please e-mail [unsubscribeURL]