Formatting a mail

[Cameron Simpson]

On 16Oct2007 15:26, Steven W. Orr <steveo at syslang.net> wrote:
| On Sunday, Oct 14th 2007 at 22:05 -0000, quoth Tony Nelson:
| =>>Sure, but in the code we're discussing:
| =>>  for m in subdir/*report*
| =>>that limit is not hit. Why not? Because the limit only applies to exec()
| =>>calls. It is an OS interface limit. For loops take place entirely in
| =>>user space. There is no "command line" being constructed in the sense
| =>>you're thinking.
| => ...
| =>
| =>Whaddayaknow.
| =>
| =>    $ ls /*/*/*/*/* | wc
| =>    bash: /bin/ls: Argument list too long
| =>          0       0       0
| =>    $ for n in /*/*/*/*/* ; do echo $n ; done | wc
| =>     211860  218203 7290275
| =>    $
| =>-- 
| 
| Ok. I now believe it but I don't understand why. The ls command gets 
| globbed and grows to be too big.  But the for commandline has 
| to also grow to the same size. Is it because the forloop is special?

No. Sigh.

The limit is ONLY FOR exec*() CALLS!

If you don't make the shell exec*() something (i.e. run another program)
with your huge glob result, you'll be fine. So:

  for foo in /*/*/*/*/*         fine
  set -- /*/*/*/*/*             fine
  ls -d /*/*/*/*/*              E2BIG
  echo /*/*/*/*/*               probably fine because echo is usually built
                                into the shell, so no sepearate program
                                exec*()ed.  

Please got and read "man execve". It documents the E2BIG error result, which
is "Argument list too long". It is the ONLY system call that does so, for the
simple reason that it is the only system call that takes a command line.

The command line limit is _nothing_ to do with the shell, and everything
to do with this one system call.

| Try this 
| 
| $ for n in /*/*/*/*/* ; do echo $n ; done | wc

Happy. No program is exec*()ed with a big command line argument list.

| Vs.
| 
| $ bash -c "for n in /*/*/*/*/* ; do echo \$n ; done | wc"
| 
| Now the stars get expanded by the outer bash and when the inner bash 
| starts, he doesn't know that there ever were any stars.

No, they do not. They get run by the _inner_ bash.
So you're performing the same thing as the first one.

| The funny part is 
| that in the new case the output numbers are different.

Very different? I would not expect this.

| Note that we are 
| exec'ing a new bash and that that bash only starts after the terminal 
| session has expanded the stars.

No! The glob is _inside_ the -c string. The are expanded by the _ionner_
shell. You can easily see this by going:

  $ set -x
  $ bash -c "for n in /*/*/*/*/* ; do echo \$n ; done | wc"

You will see the bash invoked with an _unexpanded_ glob.

Compare:

  $ set -x
  $ set -- /*/*/*/*/*
  $ bash -c "for n in $* ; do echo \$n ; done | wc"

which should explode in your face.

| AFAICT, we seem to be exec'ing an 8M 
| commandline which is bigger than the 130K in limits.h
| So what's the deal? Am I missing something? 

Yes, you are missing the order of operations in the shell. Globs are
_not_ expanded inside quotes.

And don't try these tests on the new 2.6.23 kernels. They've implemented a
long overdue half measure for command lines which greatly expands the exec*()
argument limit (to something like 1/4 of the stack size). The limit is not
gone, but is harder to test against.

Cheers,
-- 
Cameron Simpson <cs at zip.com.au> DoD#743
http://www.cskk.ezoshosting.com/cs/

Machine, you are about to be lobotomised, please don't take it personally. It
happens to the best of us. Then they become managers.
        - peter at ryutai.co.jp (Peter Evans)