On Jan 21, 2008, at 3:33 PM, Linus Torvalds wrote:

quoted text
> On Mon, 21 Jan 2008, Kevin Ballard wrote:
>>
>>> It's what shows up when you sha1sum, but it's also as simple as =20
>>> what shows
>>> up when you do an "ls -l" and look at a file size.
>>
>> It does? Why on earth should it do that? Filename doesn't =20
>> contribute to the
>> listed filesize on OS X.
>
> Umm. What's this inability to see that data is data is data?

I'm not sure what you mean. I stated a fact - at least on OS X, the =20
filename does not contribute to the listed filesize, so changing the =20
encoding of the filename doesn't change the filesize. This isn't a =20
philosophical point, it's a factual statement.

quoted text
> Why do you think Unicode has anything in particular to do with =20
> filenames?

I don't, but I do think this discussion revolves around filenames, =20
therefore it should not surprise you when I talk about filenames.

quoted text
> Those same unicode strings are often part of the file data itself, and
> then that encoding damn well is visible in "ls -l".
>
> Doing
>
> 	echo =E4 > file
> 	ls -l file
>
> sure shows that "underlying octet" thing that you wanted to avoid so =20=

quoted text
> much.
> My point was that those underlying octets are always there, and they =20=

quoted text
> do
> matter. The fact that the differences may not be visible when you =20
> compare
> the normalized forms doesn't make it any less true.

Yes, I am well aware that the encoding of the *file contents* affects =20=

filesize. But when did I suggest changing the encoding of filenames =20
inside file contents? If you treat filenames as strings, there's no =20
requirement to change the encoding of filenames inside file contents. =20=

I'm talking specifically about the filenames, not about file contents, =20=

so stop trying to argue against that which is irrelevant.

quoted text
> You can choose to put blinders on and try to claim that =20
> normalization is
> invisible, but it's only invisible TO THOSE THINGS THAT DON'T WANT =20
> TO SEE
> IT.

Don't want to, or don't need to? It's not a matter of ignoring =20
encoding because I don't want to deal with it, it's ignoring encoding =20=

because it's simply not relevant if I treat filenames as strings.

quoted text
> But that doesn't change the fact that a lot of things *do* see it. =20
> There
> are very few things that are "Unicode specific", and a *lot* of =20
> tools that
> are just "general data tools".
>
> And git tries to be a general data tool, not a Unicode-specific one.

Yes, I realize that. See my previous message about discussing ideal vs =20=

practicality.

quoted text
>> I'm not sure what you mean. The byte sequence is different from =20
>> Latin1 to
>> UTF-8 even if you use NFC, so I don't think, in this case, it makes =20=

quoted text
>> any
>> difference whether you use NFC or NFD.
>>
>> Yes, the codepoints are the same in Latin1 and UTF-8 if you use =20
>> NFC, but
>> that's hardly relevant. Please correct me if I'm wrong, but I believe
>> Latin1->UTF-8->Latin1 conversion will always produce the same Latin1
>> text whether you use NFC or NFD.
>
> The problem is that the UTF-8 form is different, so if you save =20
> things in
> UTF-8 (which we hopefully agree is a sane thing to do), then you =20
> should
> try to use a representation that people agree on.
>
> And NFC is the more common normalization form by far, so by =20
> normalizing to
> something else, you actually de-normalize as far as those other =20
> people are
> concerned.
>
> So if you have to normalize, at least use the normal form!

Was NFC the common normalization form back in 1998? My understanding =20
is Unicode was still in the process of being adopted back then, so =20
there was no one common standard that was obvious for everyone to use.

quoted text
>> The only reason it's particularly inconvenient is because it's =20
>> different from
>> what most other systems picked. And if you want to blame someone =20
>> for that,
>> blame Unicode for having so many different normalization forms.
>
> I blame them for encouraging normalization at all.
>
> It's stupid.
>
> You don't need it.
>
> The people who care about "are these strings equivalent" shouldn't =20
> do a
> "memcmp()" on them in the first place. And if you don't do a =20
> memcmp() on
> things, then you don't need to normalize.
>
> So you have two cases:
> (a) the cases that care about *identity*. They don't want =20
> normalization
> (b) the cases that care about *equivalence*. And they shouldn't do
>      octet-by-octet comparison.
>
> See? Either you want to see equivalence, or you don't. And in =20
> neither case
> is normalization the right thing to do (except as *possibly* an =20
> internal
> part of the comparison, but there are actually better ways to check =20=

quoted text
> for
> equivalence than the brute-force "normalize both and compare results
> bitwise").

I could argue against this, but frankly, I'm really tired of arguing =20
this same point. I suggest we simply agree to disagree, and move on to =20=

actually fixing the problem.

-Kevin Ballard

--=20
Kevin Ballard
http://kevin.sb.org
kevin@sb.org
http://www.tildesoft.com