[if gte mso 9]>

Re: [edk2] Bug in EFI Shell V2.0

Subject: Re: [edk2] Bug in EFI Shell V2.0

From: "Mcdaniel, Daryl" <daryl.mcdaniel@intel.com>

To: "edk2-devel@lists.sourceforge.net" <edk2-devel@lists.sourceforge.net>

Date: 2014-07-31 02:12:03

Re: Python.  The standard functions in Python work on ASCII (actually MBCS on many platforms and UTF8 specifically on UEFI).  There are a set of codecs and other functions to support other character encodings.

 

There is a lot of good documentation on Python’s support of Unicode in the Python manual.  Check out “Unicode HOWTO” and “Unicode Objects and Codecs”.  Essentially, “Unicode strings are expressed as instances of the unicode type, one of Python’s repertoire of built-in types.”

 

I was unable to reproduce Dennis’ append problem in Python.

 

In C, the standard functions (fopen, fscanf, fprintf, fputs, …) work on ASCII/MBCS/UTF8.  The wide character functions, declared in <wchar.h> and <wctype.h>,  process Unicode; UTF16 (UCS2) on UEFI.

 

EDK II provides native libraries, <Library/BaseLib.h> is one, that have both ASCII and Unicode variants.  Functions like StrCmp() process Unicode and AsciiStrCmp() handles ASCII.

 

Within StdLib (a set of standard libraries, including the C Standard Library), which provides the system calls that Python uses, there is no distinction between binary and text files.  There are some adjustments one can do by changing the buffering modes or using termios.

 

Daryl McDaniel

 

"It is the mark of an educated mind to be able to entertain a thought without accepting it."

– Aristotle

 

From: Lowell_Dennis@Dell.com [mailto:Lowell_Dennis@Dell.com]
Sent: Tuesday, July 29, 2014 10:06 PM
To: edk2-devel@lists.sourceforge.net
Subject: Re: [edk2] Bug in EFI Shell V2.0

 

Dell - Internal Use - Confidential

I think that the bigger question has to do with the programming interface … recall that I said that opening the file for append in Python was not working … exactly how does one open a file in Python or C in ‘ascii’ mode as opposed to ‘unicode’ mode anyway?

 

</lowell>

 

From: Carsey, Jaben [mailto:jaben.carsey@intel.com]
Sent: Tuesday, July 29, 2014 6:15 PM
To: edk2-devel@lists.sourceforge.net
Subject: Re: [edk2] Bug in EFI Shell V2.0

 

I think that the question is:

 

Should appending to a non-existent UNICODE file automatically write the FFFE?

 

From: Mcdaniel, Daryl [mailto:daryl.mcdaniel@intel.com]
Sent: Tuesday, July 29, 2014 4:08 PM
To: edk2-devel@lists.sourceforge.net
Subject: Re: [edk2] Bug in EFI Shell V2.0

 

Regarding appending from Uefi Shell 2.0, it is behaving as written:

 

echo 1234 > test.dat       # Writes 14 bytes to file “test.dat”: L“1234\r\n” prefixed with the Unicode Byte Ordering Mark <BOM> (0xFFFE).

 

del test.dat

echo 1234 >> test.dat    # Writes 12 bytes to file “test.dat”: L“1234\r\n” with NO prefix.

 

echo 5678 >> test.dat    # Attempts to append 12 bytes to the Unicode file “test.dat”.

# Fails because “test.dat” is NOT Unicode, No UBOC prefix

 

echo 6789 >>a test.dat  # Appends 6 ASCII bytes to test.dat.  Success, but first line is UCS2 and second is ASCII.

 

Thus, the echo command is failing because you are attempting to append Unicode text to what appears to be an ASCII file.  The Shell is trying to protect you from error.

 

Some would think that redirection should identify a request to append to an empty or non-existent file and go ahead and prepend the BOM.  That isn’t the case here.

 

I am unable to reproduce the Python error.  Here is a transcript of what I did.

 

HD15a1:\Efi\StdLib\tmp\> ver

UEFI Interactive Shell v2.0

Copyright 2009-2013 Intel(r) Corporation. All rights reserved.

UEFI v2.31 (EDK II, 0x00010000)

HD15a1:\Efi\StdLib\tmp\> rm baz.txt

Deleting 'HD15a1:\Efi\StdLib\tmp\baz.txt'

Delete successful

HD15a1:\Efi\StdLib\tmp\> echo “Now is the time” >> baz.txt

HD15a1:\Efi\StdLib\tmp\> echo “Rainbow pony” >>a baz.txt

HD15a1:\Efi\StdLib\tmp\> echo “Born Free” >>a baz.txt

HD15a1:\Efi\StdLib\tmp\> type baz.txt

..o.w. .i.s. .t.h.e. .t.i.m.e.

.Rainbow Pony

Born Free

 

HD15a1:\Efi\StdLib\tmp\> Python

Python 2.7.2 (default, Jul 23 2014, 15:04:28) [C] on uefi

Type "help", "copyright", "credits" or "license" for more information.

>>> f = open('baz.txt', 'a')

>>> f.write('Zanzibar\n')

>>> f.close()

>>> r = open('baz.txt', 'r')

>>> r.readline()

'N\x00o\x00w\x00 \x00i\x00s\x00 \x00t\x00h\x00e\x00 \x00t\x00i\x00m\x00e\x00\r\x00\n'

>>> r.readline()

'\x00Rainbow Pony\r\n'

>>> r.readline()

'Born Free\r\n'

>>> r.readline()

'Zanzibar\n'

>>> r.close()

>>> exit()

HD15a1:\Efi\StdLib\tmp\>

 

I get identical results using a version of Python built in 2012.

 

Daryl McDaniel

 

"It is the mark of an educated mind to be able to entertain a thought without accepting it."

– Aristotle

 

From: Lowell_Dennis@Dell.com [mailto:Lowell_Dennis@Dell.com]
Sent: Tuesday, July 29, 2014 11:18 AM
To: edk2-devel@lists.sourceforge.net
Subject: [edk2] Bug in EFI Shell V2.0

 

Every copy of EFI Shell V2.0 that I have found seems to have a file processing bug.

 

It looks like appending to a file does not work as expected.  The following shell snippet does not work:

 

                del test.dat

                echo 1234 >> test.dat                    # This works

                echo 5678 >> test.dat                    # This give “Error. Unable to redirect file”

 

If I modify that shell snippet as follows it works:

 

                del test.dat

                echo 1234 > test.dat

                echo 5678 >> test.dat

 

Additionally appending as ASCII also works:

 

                del test.dat

                echo 1234 >>a  test.dat

                echo 5678 >>a  test.dat

 

This error carries over to programming interfaces.  If I try to open an existing file for append in Python I get an error”

 

f = open(‘ExistingFile’, ‘a’)

# ERROR

 

However, if I open the file in mode ‘r+’ there is no error:

 

f = open(‘ExistingFile’, ‘r+’)

f.seek(0,2)

 

What is going on here?

 

Thanks,

Lowell Dennis

Platform Software Engineer

Dell | Business Client

office +1 512 723 8880

    fax +1 512 283 0149