grep -P is not supported

PCRE support would require libpcre3, which is in /usr/lib, while grep is in
/bin. As I'm not so sure that moving libpcre to /lib (and bloating grep's
dependencies), the "fix" for this is probably just to remove references to "-P'
from the manpage.

Based on the previous comment, removing the option -P from the grep documentation seems like the best solution.

By the way, the -P option is also shown in `grep --help`.

-P is too valuable to miss (since the rest of the world uses that perl regexp syntax)

Proposed fix (without breaking boot):

By default, in ubuntu, /usr/bin preceded /bin in the path. So install grep without -P (no libpcre) in /bin during boot.

Then install grep with -P (with libpcre) in /usr/bin, once /usr is mounted.

Or, "apt-get install pcregrep" if this feature it so important to you. Or use perl itself, which is prett perl-compatible. :)

Installing binaries with conflicting names in /usr/bin and /bin is against Debian Policy, and for good reason, since it completely changes the behaviour of your system based on nothing more than what order your PATH is in.

I think that the easiest thing is to update the documentation and to remove the references to the switch -P. It's anoying seeing that the documentation states something that's not working. I don't mind not having the flag -P.

It's true that the call to grep can be substituted by a call to perl quite easily. Although, in this case using perl as one disavantage: the lack of the switch --color which grep has.

>> Or, "apt-get install pcregrep" if this feature it so important to you.

pcregrep is not an alternative for grep. It is considerably slower, and supports far fewer options (including, for example, GNU longopts, -R, --color)

>> Or use perl itself, which is prett perl-compatible. :)

lol. Not a serious option for anyone.

>> Installing binaries with conflicting names in /usr/bin and /bin is against Debian Policy, and for good reason, since it completely changes the behaviour of your system based on nothing more than what order your PATH is in.

I looked through Debian Policy, and cant find the reference. I was under the impression that thats what PATH is for. A comprismise could be to install it as /usr/bin/pgrep (similar for egrep and fgrep), and let the user symlink it themselves.

Ah, /usr/bin/pgrep already exists. Well, something else in the same vein. perlgrep, prehaps?

"pgrep" is from "procps" package.

I'd really like to have the -P switch, too.

The idea advanced in this thread that the documentation be changed is a terrible one. The current manpage is for the standard grep (aka egrep fgrep) available in every linux distro that I've worked with. And why would ubuntu ever want to distinguish itself by the things it omits that others include?

Because I have in production scripts using grep -P running on Mandriva, it complicates moving those servers to ubuntu even though I'm convinced that in the long-term ubuntu is the way to go.

In reply to Steve Hollingsworth's comment:
"The idea advanced in this thread that the documentation be changed is a terrible one."

I disagree with you, this bug was open over a year ago and the option -P hasn't appeared, yet the option is documented as being available. I would really love to have the option -P in grep, but I understand that adding the option -P seems very hard, we are still waiting after one year, the only reasonable solution is that the option is removed from the documentation.

I can live with a tool that's lacking some options, in fact if you use different kinds of Linux/UNIX systems you will get used to check the man pages of every command as soon as you want to run a command with some esoteric arguments.

What I expect is that the documentation of a tool will list the options available. If grep can't have the option -P then it should at least be removed from the documentation.

Emmanuel Rodriguez reporting the bug in the first place should not mean that his opinion that doing without the option may be OK should have more weight than those of us who need the option and whose acceptance of ubuntu is impeded by it.

If his solution is implemented and this bug closed then I will open a new bug report noting that Gnu grep is broken. Gnu grep is a package available on most linuxes and it has a standard manpage. I'm appalled at the thought that it might never work correctly on ubuntu.

Rather than discussing fixing the documentation, can we discuss fixing
the problem, and having grep -P supported?

As I stated before, pcregrep isnt a serious solution. Nor is using
perl. So the options are to include libpcre.so in /lib, or replacing
/bin/grep with /usr/bin/grep (I have yet to hear about why this is
against policy).

I'm sure there are other solutions. Anyone?

Paul

On 5/22/06, Paul <email address hidden> wrote:
> >> Or, "apt-get install pcregrep" if this feature it so important to
> you.
>
> pcregrep is not an alternative for grep. It is considerably slower, and
> supports far fewer options (including, for example, GNU longopts, -R,
> --color)
>
> >> Or use perl itself, which is prett perl-compatible. :)
>
> lol. Not a serious option for anyone.
>
> >> Installing binaries with conflicting names in /usr/bin and /bin is
> against Debian Policy, and for good reason, since it completely changes
> the behaviour of your system based on nothing more than what order your
> PATH is in.
>
> I looked through Debian Policy, and cant find the reference. I was under
> the impression that thats what PATH is for. A comprismise could be to
> install it as /usr/bin/pgrep (similar for egrep and fgrep), and let the
> user symlink it themselves.
>
> --
> grep -P is not supported
> https://launchpad.net/bugs/15051
>

--
Paul Biggar
<email address hidden>

pcregrep is not a valid option, but glark certainly is. Either, however, is an alternative (they don't have the same syntax, and they don't have the same features), _not_ a solution, and either way, the man page is still an issue, as well as compatibility problems which Steve addresses. Paul's proposition was:

>Proposed fix (without breaking boot):

>By default, in ubuntu, /usr/bin preceded /bin in the path. So install grep without -P (no libpcre) in /bin during boot.

>Then install grep with -P (with libpcre) in /usr/bin, once /usr is mounted.

Is this doable? While this may be difficult, the alternative is to have a modified man page for grep, /and/ a patch for grep which removes -P from the help listings, and still not have -P

I am adding my vote to those who would like to have the -P option
available. In the enterprise world, many corporate scripts are written
and since RHEL and SLES support the -P option, our internal scripts are
making use of this option. Obviously, we have a problem when porting
them to Ubuntu.

Yes, we could change our scripts, or compile ourselves grep with Perl
regular expression support. But this is beside the point. Ubuntu
should make it easy for migration. If other distributions get it right,
why not Ubuntu?

Please seriously consider providing Perl regular expression to grep.

Please allow me to jump on the bandwagon. Perl-style regular expressions are important and I would very much like to see this feature added.

Just to provide additional information:

SLED10 provides grep with -P installed in the following paths:

> ls -l /usr/bin/grep /bin/grep
-rwxr-xr-x 1 lmwadmin root 134252 2006-06-16 09:50 /bin/grep
lrwxrwxrwx 1 lmwadmin root 9 2007-05-16 07:36 /usr/bin/grep -> /bin/grep

Fedora Core 2 (old, but I have none more recent to test) provides grep with -P installed in the following paths:

> ls -l /bin/grep
-rwxr-xr-x 1 root root 85360 Jun 28 2005 /bin/grep

Is it that costly to add Perl regex to grep under /bin? The two other major distributions do not believe so.

Still doesn't work on Feisty...

kmos@bash:~$ grep -P s/foo/bar/; x
grep: The -P option is not supported
bash: x: command not found

x is a filename with "foolish bigfoot"

In reply to Marco Rodrigues:

The command grep can only be used for finding text strings, it can't be used for replacing text in a file. Based on the command that you have posted it looks like if you where trying to replace all the occurrences of 'foo' by 'bar', unless if you where really looking for the string 's/foo/bar/;', where you?

First of all you have to be careful with the text that's passed to grep, in your case you wanted to pass a semicolon ';' but you didn't escaped it. This had for effect of splitting your single command into two commands:
grep -P s/foo/bar/
x

Finally, if you want to replace text in a file you have to use another program such as sed or perl, here are some examples:

# Create a modified copy of x
sed 's/foo/bar/g' x > modified-x

# Modify the file in place, x will be replaced
sed -i 's/foo/bar/g' x

# Create a modified copy of x
perl -pe 's/foo/bar/g' x > modified-x

# Modify the file in place, x will be replaced
perl -i -pe 's/foo/bar/g' x

I hope that it helps.

Other popular distributions do have grep -P available - for example: RHEL 4 and SLES 9.
RHEL places PCRE in /lib while SLES appears to link PCRE statically.

I'd argue that Ubuntu, either on the server or desktop, targets systems with enough resources to have libpcre in /lib. If there's room for BDB, Perl and Python, surely there's room for libpcre.
This takes several hundred KiB which, on a modern system, is usually acceptable.

Still not supported on Gutsy Tribe-5

As of Oct. 16th, grep -P is still not available in Gusty.

Grep is a common package with a standard man page. Grep isn't broken, Ubuntu is. It is necessary for many scripts used in corporate environments. Additionally, the ease at which someone can return a match (such as "grep -P -o -e '/regex/'") makes it an important tool.

Also, recommendations to use sed and perl itself are horrible. Each of these tools have benefits over each other for different applications. Perhaps this can actually be fixed?

This question was discussed by the Ubuntu Technical Board during a regularly scheduled meeting an 2007-11-20.

It was proposed that libpcre3 could be moved from /usr/lib to /lib to accomodate its use by /bin/grep. A quorum present at the meeting voted and confirmed this. Hopefully this resolves any policy question pertaining to this bug.

Cheers,
 - mdz, on behalf of the Ubuntu Technical Board

The last two messages I've received about this bug (the message above and the message that the bug is once again "unassigned") puzzle me. With the technical board agreeing that libpcre3 can be moved to /lib, what actually must happen now that the library _will_ be moved and this bug fixed? I continue to think that it is an embarrassment to the whole Ubuntu project that it hasn't been fixed and is called a "wishlist" item.

Steve,

The Technical Board clarified that we're allowed to move the library, and that this is the preferred course of action, but did not assign anyone to move it. In order to get it moved, we would need to find a core dev who cares enough to move it. See the log of the meeting:
http://irclogs.ubuntu.com/2007/11/20/%23ubuntu-meeting.txt ( Search for "[TOPIC] Perl regular expressions in grep").

At this point, the best bet would be to try to find a core dev willing to move it. The library probably won't be moved until Hardy+1, which is unfortunate.

Thanks, Joe. That was an interesting and clarifying link. This seems to be the crux of it:

"I don't think worrying about who's going to do something this trivial is our problem."

Too trivial to assign! Ouch.

I've grudgingly worked around this by compiling a version with -P support because I need it and it was the last barrier to moving away from Mandriva, but it remains a thorn in my side.

On Thu, Jan 24, 2008 at 09:47:05PM -0000, Steve Hollingsworth wrote:
> The last two messages I've received about this bug (the message above
> and the message that the bug is once again "unassigned") puzzle me.
> With the technical board agreeing that libpcre3 can be moved to /lib,
> what actually must happen now that the library _will_ be moved and this
> bug fixed? I continue to think that it is an embarrassment to the whole
> Ubuntu project that it hasn't been fixed and is called a "wishlist"
> item.

The Technical Board makes policy decisions, and does not force Ubuntu
developers to implement particular changes. This bug report, being a
request for new functionality which is not currently available in Ubuntu (or
Debian), is correctly classified as a "wishlist" item.

Any Ubuntu core developer interested in volunteering to fulfil your request
has the authority and privileges to do so.

--
 - mdz

Mr. Zimmerman,

try typing the following:
grep --help
also try:
man grep

Note that both say that -P enables perl regular expressions. Here's a simple grep to print out all the UUIDS assigned in my fstab:
grep -o 'UUID=[-A-Za-z0-9]* ' /etc/fstab

It prints out this list:
UUID=ffdd991f-2fb5-4f7c-b45b-3d1923044e88
UUID=41fbf61a-6c4c-4a23-8864-d7a59d111a4e
etc

That's great. Now, I don't want to match the UUID= , as I'm using this in a script. So, I go ahead and enable perl regular expressions, and use lookbehind:
grep -oP '(?<=UUID=)[-A-Za-z0-9]* ' /etc/fstab

And, guess what? I get this:
grep: The -P option is not supported

That's a bit of a contrived example, but I found this bug for a very similar reason to the one I just described. Perl regular expressions aren't compiled in, even though both the man and --help say it is. That's a bug. Missing Perl Regex support is inconsistent with upstream's --help and man page at best, and breaks scripts (and admins) at worst. Sure, there are workarounds, but nothing quite matches the convenience of grep.

I'm glad that the Technical Board approved the change in regards to policy. That said, I respectfully disagree with this bug being demoted to wishlist. Hopefully I'll remember to ask around for a core dev to make this change when the Hardy+1 repos open.

You're correct that the inconsistency should be resolved. As the documentation is inconsistent with the program, this is a bug, and I've adjusted its importance accordingly. There's no reason why this couldn't be fixed for 8.04 if someone is willing to implement the changes in grep and libpcre.

The changes needed for pcre3 are attached to bug 186187.

Debdiff for grep, to not disable perl-regexp anymore.
Obviously, pcre3 should get upgraded before this.

several comments on the provided patch:

- as noted in bug #186187, pcre3 is not currently ready for merging from Debian with the present patch, due to binary incompatibilities in libpcrecpp0. Under the circumstances, the patch to move pcre3 to /lib should be broken out.

- the patch moves both libpcre and libpcreposix to /lib. Only libpcre is needed for grep; libpcreposix should be left in /usr/lib if there isn't a specific need to move it.

- the patch also moves the .a and .so files to /lib. These are development files that are not useful without /usr (in particular, /usr/include) and should not be moved. The custom is to create .so as a symlink in /usr/lib pointing to the library in /lib. There are some packages that do put their .so symlinks in /lib, but this is the exception rather than the rule; and the .a files don't belong there in any case.

- as binaries in /bin must have their libraries in /lib, so the grep package must end up with a versioned dependency on the version of libpcre3 that ships libpcre.so.3 in /lib. This should be done using the shlibs mechanism; either by incrementing the shlibs in pcre3, or using a debian/shlibs.local in the grep package. Normally I would hesitate to recommend bumping shlibs for the benefit of a single package, but in this case the shlibs already require a version of libpcre3 newer than what's shipped in gutsy, so it makes only a minimal difference for other packages.

Please provide a new patch for pcre3 that addresses these issues and, freeze allowing, I can sponsor this change.

Thanks, Steve, for your feedback.
There's an updated patch at bug 186187 now.
I'd like to get feedback on it before asking for a freeze exception.

I'm not sure about the following change:
 * bump version in dh_makeshlibs call to ">= 7.4-1ubuntu3"

but apart from that this should provide the functionality required for the grep patch:
  * Move libpcre3 to /lib (LP: #15051)
    - debian/dirs: add "lib"
    - debian/libpcre3.install, debian/libpcre3-udeb.install:
      - Move libpcre3.so.* to /lib
    - debian/rules:
      - bump version in dh_makeshlibs call to ">= 7.4-1ubuntu3"
      - include /lib in dh_shlibdeps call
    - debian/libpcre3-dev.links:
      - Create symlink in /usr/lib/libpcre.so to /lib/libpcre.so.3

Please subscribe ubuntu-release, if the patch is OK and a FeatureFreeze exception is required for this. Thank you.

Update?

I've subscribed ubuntu-release now.
From using it myself now for some time, I can say that "grep -P" is really useful.. ;)

Well.. it's a shame that it won't (probably) make it into Hardy.
It hasn't caused any problems on my desktop machine since about 3 months.

I would like to add my voice to those who find it perplexing that two-and-a-half years have passed since the first reporting of this bug, and still it remains outstanding. In particular, I am disappointed that the version of grep in the LTS server distribution does not support pcre.

Surely this is important enough to do something about?

On Tue, May 20, 2008 at 08:42:59AM -0000, Simon wrote:
> I would like to add my voice to those who find it perplexing that two-
> and-a-half years have passed since the first reporting of this bug, and
> still it remains outstanding. In particular, I am disappointed that the
> version of grep in the LTS server distribution does not support pcre.
>
> Surely this is important enough to do something about?

Literally tens of thousands of bugs were resolved during the 8.04 cycle, but
unfortunately it simply isn't possible to tend to every request, and Ubuntu
developers (rightly) prioritize regressions and serious bugs over feature
requests like this one.

We appreciate the effort which has gone into preparing a patch to add this
functionality, and it has not been forgotten.

--
 - mdz

I keep re-discovering this fundamental incompatibility between Ubunutu and other Linux distributions. I'd like to just clear up some items that are repeated several times by developers who have replied to this bug:

"Perl" is not involved here. PCRE is a broadly used RE library, and only happens to have started as an attempt to bring modern Perl RE features to other languages.

Also, while many tools exist which would allow users to search for PCRE-style patterns in text files, that does not address the problem, here: Ubuntu provides a command called "grep", but it fails to provide one of the most frequently used parameters to that command on other Linux platforms. This presents a fundamental incompatibility between Ubuntu and other Linux platforms.

The reason which was presented was one of decreasing the bloat in boot-time commands, but that doesn't hold any water; on the same machine, I have Fedora 9 and Ubuntu 8.04 installed. The Ubuntu /bin/grep is 99K. The fedora binary is 84K:

 $ ls -lh /bin/grep
 -rwxr-xr-x 1 root root 84K 2008-02-20 01:33 /bin/grep
 $ ldd /bin/grep
 linux-gate.so.1 => (0x00110000)
 libpcre.so.0 => /lib/libpcre.so.0 (0x009a8000)
 libc.so.6 => /lib/libc.so.6 (0x0083d000)
 /lib/ld-linux.so.2 (0x0081d000)

 $ ls -lh /bin/grep
 -rwxr-xr-x 1 root root 99K 2007-10-23 16:58 /bin/grep
 $ ldd /bin/grep
 linux-gate.so.1 => (0xb7f21000)
 libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7dc4000)
 /lib/ld-linux.so.2 (0xb7f22000)

Now, granted 1) this small difference is likely due to the version of grep under F9 being older than that under U8.04 and 2) libpcre is 165K, but that's less than 1/10th the size of the libraries we're already linking grep against, so again... what's the objection?

Another suggestion was having a /usr/bin/grep which supported the standard options, while /bin/grep could be a cut-down version for boot only. This was shot down because it would change behavior based on the ordering of PATH... which isn't quite accurate. What it would do is fail to meet the default expectation that the -P option would work when /bin was first in your path. If /usr/bin were first, then obviously all of the same commands that worked from /bin would continue to work. Picture, if you will, that this were the case. The confusion caused is exactly the same confusion that systems administrators and developers suffer when they attempt to use the same program from the same author in their shell scripts and Makefile on two distributions of the same operating system... If you feel that that situation is untenable, then you really have to feel that the current situation is as well, unless you're simply taking the provincial view that users should not expect core commands to behave the same between Linux distributions.

To circle back around to the question at hand: what breaks if libpcre is placed in /lib? Do we cross or even approach some critical threshold, or are we just resisting this change on the basis of a phantom concern?

On Fri, Jun 06, 2008 at 08:23:35PM -0000, AaronSherman wrote:
> To circle back around to the question at hand: what breaks if libpcre is
> placed in /lib? Do we cross or even approach some critical threshold, or
> are we just resisting this change on the basis of a phantom concern?

Nothing breaks. There is no objection; in fact, the Ubuntu Technical Board
explicitly said this was OK:

https://bugs.edge.launchpad.net/ubuntu/+source/grep/+bug/15051/comments/23

The only thing which is holding back this change is that it has not found
its way to an Ubuntu developer with the time, ability and willingness to
effect the change: this is a problem of resources, not of policy.

--
 - mdz

I'll have a look at this if nobody else beats me to it. But not very soon, I'm still on Hardy duty.

I simplified Daniel's pcre3 patch and updated it for intrepid.

This bug was fixed in the package pcre3 - 7.6-2ubuntu1

---------------
pcre3 (7.6-2ubuntu1) intrepid; urgency=low

  * Move libpcre3 to /lib (LP: #15051, Debian #350468)
    - debian/libpcre3.install, debian/libpcre3-udeb.install:
      Move libpcre3.so.* to /lib
    - debian/rules: include /lib in dh_shlibdeps call
    - debian/libpcre3-dev.links:
      Create symlink in /usr/lib/libpcre.so to /lib/libpcre.so.3

 -- Daniel Hahler <email address hidden> Wed, 18 Jun 2008 10:22:41 +0000

This bug was fixed in the package grep - 2.5.3~dfsg-5ubuntu1

---------------
grep (2.5.3~dfsg-5ubuntu1) intrepid; urgency=low

  * debian/rules, debian/control: Enable perl-regexp (-P) (LP: #15051,
    Debian #238237)

 -- Daniel Hahler <email address hidden> Wed, 18 Jun 2008 10:34:11 +0000