bash completion now quotes shell variable references rather than expanding them

Fixed by editing line 1587 of /etc/bash_completion (see https://bugs.launchpad.net/ubuntu/+source/bash/+bug/769866/comments/10)

My comment #2 was incorrect, sorry.

Trailing space and quoting backslash have not disappeared after doing #2.

Was reported on bug-bash mailinglist:
http://lists.gnu.org/archive/html/bug-bash/2011-02/msg00274.html

And Chet has posted a partial fix:
http://lists.gnu.org/archive/html/bug-bash/2011-03/msg00235.html

This bug is really ugly. I often use 'll $HEAD/' <tab><tab> which now doesn't work anymore. Unfortunately I don't hace a 10.10 system anymore to compare...

This bug is really horrible. Anyone know what line in bash_completion I need to fix? How did it make it into the release??? Also, I sometimes see completion adding an extra space when completing directory names, which makes it difficult to traverse through a directory hierarchy with tab completion.

As a workaround while this is fixed I've downgraded to maverick package:
http://packages.ubuntu.com/maverick/bash

same here .. had to downgrade to maverick package .. hope it will be fixed.

Since a couple of weeks on 10.11 and still the same behavior. Who maintains bash?

Maybe this forum post might help to fix the bug: http://www.linuxquestions.org/questions/linux-general-1/bash-auto-complete-of-environment-variables-613203/#post3055141

Please... Using the tab is much important for the developers/administrators. We can't remove the backslash every time we use the tab ! It is a bug that can't be in a supported version ! (12.0.4 LTS). It is the kind of things that makes you hate an environment ! Because we use the tab ALL THE TIME !

Who makes such specification?
And who come out ahead this specification (or bug) ?

If someone know reason that this specification is better than before one, please tell me.

This is a real pain for me as I set 101 variables to point to directories I want to get at and am in the habit of typing $foodir/abc<tab> to get what I want and now I can't do it so well.

SOLUTION: This was a bug in bash which is fixed in patch bash4.2-029. See
http://lists.gnu.org/archive/html/bug-bash/2012-07/msg00018.html

I build from source as follows:
  - downlaod bash-4.2.tar.gz from ftp://ftp.gnu.org/pub/gnu/bash/
  - extract
  - download patches bash42-001 through bash42-029 (actually I did all through 037) from ftp://ftp.gnu.org/pub/gnu/bash/bash-4.2-patches/
 - apply patches
 - ./configure --prefix=/usr/local; make; make install

Using this /usr/local/bash fixes the problem -- tab completion on environment variables works again.

Can these patches please be applied to the package in precise and oneiric?
Thanks!

Note: This bug affects 12.04 Precise (not just natty). The solution in comment 14 was tested on 12.04

I upgraded to bash 4.2.36(1), and the problem is only half fixed. On older versions of bash (certainly in the bash2 vintage), cd $VARIABLE/<tab> offered completions. At lest in the newer version it doesn't escape the dollar sign anymore, but this is still a removed, highly used feature.

This bug was first reported in May 2011 and still not fixed?
It's driving me crazy. A long-established very helpful bash feature
has been trashed.

What's going on?

This is bad.

But, alas, it seems to be upstream in bash itself.

Interestingly I can reproduce this issue only on some machines, but not on all of them despite they all have bash 4.2.

I could reproduce it on Natty and Precise, on Debian Wheezy, and on one, but not on all of my Debian Sid boxes (4.2-5).

Disabling the sourcing of /etc/bash_completion (either manually enabled in /etc/bash.bashrc or ~/.bashrc) made the problem vanish on Debian and on Natty, but not on Precise. Even purging bash-completion did not help on Precise.

And that's not all: While on precise using "bash --rcfile /dev/null" still did not fix the issue, it brought the issue back on Natty. And on Debian Sid it still worked as expected, i.e. did not add any backslash.

I am also suffering this problem in Ubuntu 12.10.
Commenting out the sourcing of bash-completion managed to fix it, but the problem is that it disables all of its advantages. The problem must be in one of the various files at /usr/share/bash-completion/completions ,I'd bet. But debugging that until finding the culprit would be a task of true patience.

I tried to build bash manually and it always does the same thing unless I disable sourcing bash-completion.

Ok:

This bug does not affect Q or R (I checked.)

I make an internal package (for myself), and it resolves the problem.
I've attached a debdiff which creates bash-4.2-2ubuntu2-gg1; it should be trivial to modify it to produce bash4.2-2ubuntu3.

Nominated for Precise. As Comment 21 says Quantal and Raring are unaffected, i'm not nominating them.

Given that this bug was initially filed against Natty, can someone test this on Oneiric to see if it is also affected?

------
Thomas Ward
Ubuntu Bug Squad Member

antarus@antarus-glaptop:~$ cd Downloads/
antarus@antarus-glaptop:~/Downloads$ ls
bash_4.2-0ubuntu4_amd64.deb
bash_4.2-2ubuntu2-gg1.debdiff.txt
bash_4.2-5ubuntu2_amd64.deb
antarus@antarus-glaptop:~/Downloads$ ar x bash_4.2-0ubuntu4_amd64.deb
antarus@antarus-glaptop:~/Downloads$ ls
bash_4.2-0ubuntu4_amd64.deb control.tar.gz
bash_4.2-2ubuntu2-gg1.debdiff.txt data.tar.gz
bash_4.2-5ubuntu2_amd64.deb debian-binary
antarus@antarus-glaptop:~/Downloads$ tar -zxf data.tar.gz
antarus@antarus-glaptop:~/Downloads$ ls
bash_4.2-0ubuntu4_amd64.deb control.tar.gz etc typescript.txt
bash_4.2-2ubuntu2-gg1.debdiff.txt data.tar.gz usr
bash_4.2-5ubuntu2_amd64.deb debian-binary1_i386.deb
bin
antarus@antarus-glaptop:~/Downloads$ bin/bash
eantarus@antarus-glaptop:~/Downloads$ echo \$HOME

Oneric also appears affected by this bug.

I'm on Quantal (Kubuntu) and I'm affected by the bug. See also comment 20.

I need more details from Quantal users then. Quantals' bash has bash42-029.diff, which fixes the bug in bash that caused a (and I presume this) problem. It could be Quantal has another bug?

I just downloaded bash_4.2-5ubuntu1_amd64.deb and could not reproduce. What version of bash are you using?

-A

$ bash --version
GNU bash, version 4.2.37(1)-release (x86_64-pc-linux-gnu)

$ wajig list bash
ii bash 4.2-5ubuntu1 amd64 GNU Bourne Again SHell
ii bash-completion 1:2.0-1ubuntu2 all programmable completion for the bash shell
ii command-not-found 0.3ubuntu5 all Suggest installation of packages in interactive bash sessions

$ wajig detail bash
Package: bash
Essential: yes
New: yes
State: installed
Automatically installed: no
Multi-Arch: foreign
Version: 4.2-5ubuntu1
Priority: required
Section: shells
Maintainer: Ubuntu Developers <email address hidden>
Architecture: amd64
Uncompressed Size: 1,450 k
Depends: base-files (>= 2.1.12), debianutils (>= 2.15)
PreDepends: dash (>= 0.5.5.1-2.2), libc6 (>= 2.15), libtinfo5
Recommends: bash-completion (>= 20060301-0)
Suggests: bash-doc
Conflicts: bash-completion (< 20060301-0), bash-completion (< 20060301-0), bash
Replaces: bash-completion (< 20060301-0), bash-completion (< 20060301-0), bash-doc (<= 2.05-1), bash-doc (<= 2.05-1)
Provides: bash
Provided by: bash
Description: GNU Bourne Again SHell

$ wajig detail bash-completion
Package: bash-completion
State: installed
Automatically installed: no
Multi-Arch: foreign
Version: 1:2.0-1ubuntu2
Priority: standard
Section: shells
Maintainer: Ubuntu Developers <email address hidden>
Architecture: all
Uncompressed Size: 892 k
Depends: bash (>= 3.2)
PreDepends: dpkg (>= 1.15.7.2~)
Breaks: xen-tools (<= 4.1-1), xen-tools (<= 4.1-1)
Replaces: bash, bash, cryptsetup (< 2:1.1.2-2), cryptsetup (< 2:1.1.2-2), xen-tools (<= 4.1-1), xen-tools (<= 4.1-1)
Provides: bash-completion
Description: programmable completion for the bash shell

This bug was fixed in 4.2-3 upload, and hence is fixed in quantal and raring.

However, there is a new option that one needs to set - direxpand.

direxpand
 If set, bash replaces directory names with the results of word expansion when performing filename completion. This
 changes the contents of the readline editing buffer. If not set, bash attempts to preserve what the user typed.

bash (4.2-3) unstable; urgency=low

  * Apply upstream patches 025 - 029, fixes for
    - When used in a shell function, `declare -g -a array=(compound assignment)'
      creates a local variable instead of a global one.
    - The `lastpipe' option does not behave correctly on machines where the
      open file limit is less than 256.
    - When the `extglob' shell option is enabled, pattern substitution does not
      work correctly in the presence of multibyte characters.
    - When using a word expansion for which the right hand side is evaluated,
      certain expansions of quoted null strings include spurious ^? characters.
    - Bash-4.2 tries to leave completed directory names as the user typed them,
      without expanding them to a full pathname. One effect of this is that
      shell variables used in pathnames being completed (e.g., $HOME) are left
      unchanged, but the `$' is quoted by readline because it is a special
      character to the shell. This patch introduces two things:
      - A new shell option, `direxpand', which, if set, attempts to emulate the
        bash-4.1 behavior of expanding words to full pathnames during
        completion;
      - A set of heuristics that reduce the number of times special characters
        such as `$' are quoted when the directory name is not expanded.
  * Mark bash and bash-static as Multi-Arch: foreign . Closes: #674120.
  * Avoid warning for unset debian_chroot with -u (Marko Mäkelä).
    Closes: #659940.
  * Fix string expansion: \u and \U for code points U+0080 to U+00FF (Darran
    Salt). Closes: #661386.
  * Search $- for `i' for the interactivity test. Closes: #482194.
  * Search for bash_completion in /usr/share/bash-completion too.
    Closes: #655071, #655208.

 -- Matthias Klose <email address hidden> Sun, 01 Jul 2012 13:45:35 +0200

I do not believe the fix is a valid SRU candidate, because it changes quoting behaviour, introduces a new option and requires a user to opt-in / change that option.

If you need a fix for the bug in precise, please update the bug description as per Stable Release Updates template.

[1] https://wiki.ubuntu.com/StableReleaseUpdates#SRU_Bug_Template

@Alec Warner (antarus):
The proposed patch is ok, sans missing this bug reference and not using correct SRU version number. But that's easy enough to fix when sponsoring. Please update bug description as per SRU guidelines if you want this uploaded in Precise. Thanks.

unsubscribing sponsors, please resubscribe when the bug description is updated as per SRU guidelines.

[Impact]
 * Explanation see above.
 * This is really a bug, ie the behaviour expected by the user changes unexpectedly.
 * In addition, only the interactive command line is affected, bash scripts should not see this change.
[Test Case]
 * See above.
[Regression Potential]
 * I see a low potential for regressions since this bug is related to the interactive command line and would not influence any scripts.
[Other Info]

So IMHO this is a very good candidate for Precise.

@ joerg:
but the default interactive behaviour has changed
$ shopt direxpand
direxpand off

And one needs to manually re-enable that option to get previous behaviour.

The template you posted lacks required information. Specifically:

The test case should check:
- that $ is escaped less
- that direxpand does not expand as expected
- that direxpand option is available and if enabled, expands directories with variable names.

The impact should describe what is being changed (underquoting + new option).

Regression potential w.r.t. to underquoting should be considered (e.g. security issues?!)

If anyone still has a reproducer environment setup with a patch bash version on precise and can easily reproduce please feel free to do so and alter the description above. In the meantime, Ill get a reproducer setup and update the description once my test is done.

Thanks
Adam

Updated debdiff to include bug lp in changelog

$ dput ubuntu bash_4.2-2ubuntu2.1_source.changes
Uploading to ubuntu (via ftp to upload.ubuntu.com):
  Uploading bash_4.2-2ubuntu2.1.dsc: done.
  Uploading bash_4.2-2ubuntu2.1.diff.gz: done.
  Uploading bash_4.2-2ubuntu2.1_source.changes: done.
Successfully uploaded packages.

The package was uploaded into precise-proposed unapproved queue, pending for a member of the SRU team to accept the package into the precise-proposed archive.

Hello Dan, or anyone else affected,

Accepted bash into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/bash/4.2-2ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hi,

I installed bash 4.2-2ubuntu2.1 from precise-proposed.
Enabling the direxpand bash shell option fixes this issue to my satisfaction, thanks!

- Dan

Yes, this fixes it.

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

This bug was fixed in the package bash - 4.2-2ubuntu2.1

---------------
bash (4.2-2ubuntu2.1) precise-proposed; urgency=low

  * Fix directory expansion. (LP: #778627)
 -- Alec Warner <email address hidden> Mon, 18 Mar 2013 22:12:54 +0000

Finally, this bug is fixed!

Any idea why `ls $d/f[TAB]` still becomes `ls \$d/f `? Other commands seem fine, but not `ls`.
Does anyone also have the same problem after update? I'm running Precise, btw

Ladios,

just updated my precise installation to the latest bash package and now it works as expected.
Including "ls".

Investigated. All the commands that failed to do completion properly are defined with `-F _longopt`.
dpkg-query shows me that there are a lot of conffiles... OMG I wonder how many they are throughout my installation.
Will try to get rid of those conffiles to see if that fixes my problem.

Oh... I thought conffiles is short for conflict files... lol

Actually, most defined completions would add '\' in front of '$'. It only works when the command has no completion defined, for instance, the `ll` alias does.
Don't know what went wrong. I think I have to go for the direxpand option.

On 10 April 2013 14:48, Ladios Jonquil <email address hidden> wrote:
> Actually, most defined completions would add '\' in front of '$'. It only works when the command has no completion defined, for instance, the `ll` alias does.
> Don't know what went wrong. I think I have to go for the direxpand option.
>

Did you enable direxpand option?

$ shopt -s direxpand

All this SRU does is backport "direxpand" option and leaves it
disabled by default. One must enable it, if variable expansion is
desired.

Regards,

Dmitrijs.

Yes, I enabled direxpand option eventually. It works.
I misunderstood this update. I thought this would actually fix the shell variables quoting issue (by not quoting them), but it didn't.

I'd just like to chime in and say, whoever had the bright idea of escaping $ during tab completion... go to hell.

If the proposed fix requires setting direxpand variable, then that is not a fix, it is a workaround fix. (something is better than nothing though). Historical behaviour should not be changed and users must not be expected to set this new variable. This is affecting so many people and this is real pity.

I strongly agree with thava (in comment #51).

It's now 2017 and this issue is still unresolved and continues to require a sub-optimal workaround and no proper fix returning the original behaviour and allowing the variable name to remain in the command-line, post tab-completion.

It's now 2021 - I'm on Ubuntu 20.04.2 LTS and I'm still being annoyed by this bug!

Agreed, this is an annoying bug and has been around far too long (a decade?).

ls $top/.ss<TAB> should give: ls $HOME/.ssh/<CURSOR>
"shopt -s direxpand" expands $HOME which is not wanted.
"shopt -u complete_fullquote" should give me what I want but has no effect.

(corrected)
Agreed, this is an annoying bug and has been around far too long (a decade?).

ls $HOME/.ss<TAB> should give: ls $HOME/.ssh/<CURSOR>
"shopt -s direxpand" expands $HOME which is not wanted.
"shopt -u complete_fullquote" should give me what I want but has no effect.

I still get \$HOME when pressing Tab in bash 5.0 & 5.1 (ubuntu 20.4)
Can the bug we reopened?

It is now 2022, over a decade later and this annoying bug still exists.
Ubuntu 21.10
bash 5.1.8