Please remove the non-free 3rdparty folder

Bug #1591763 reported by Thomas Goirand
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
congress
In Progress
High
Eric K

Bug Description

Since I've done the work in Debian, python-antlr3 contains the necessary Python bindings for Congress, which doesn't need to contain antlr3 itself. This is also available in Ubuntu Xenial. So there shouldn't be any problem anymore for the Ubuntu gate if that's what you're using.

The 3rdparty folder is considered non-free, as it contains already pre-built files (ie: antlr-3.5-complete.jar). This makes the whole package non-free from the point of view of all downstream distributions (Debian, Ubuntu, RDO, SuSE and Gentoo).

Revision history for this message
Tim Hinrichs (thinrichs) wrote :

Let's look into this soon, as it impacts downstream.

Changed in congress:
importance: Undecided → High
Revision history for this message
Eric K (ekcs) wrote :

Thank you Thomas! We've been wanting to do this for a while. Having python-antlr3 in xenial is a big help. Thanks for the alert!

The main issues we still have are:
1. Not available in 14.04 (including backports). I need to confirm if it's okay to ditch 14.04 now in a new release.
2. For python3, we rely on upstream antlr3/python3 runtime not available in packages.

Long term we're still looking at replacing antlr3 completely because upstream development has moved on, but that's a complex job and in the mean time we still want to see what we can do.

@Tim, the antlr-3.5-complete.jar file is only used because the Makefile calls on antlr3 to re-generate the parser and lexer from the grammar file on each setup. Do you know why this was needed instead of just using the pre-generated parser and lexer in the repo?
A developer who changes the grammar will still need to use antlr3, but I don't think the end-users need it.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on congress (master)

Change abandoned by Eric K (<email address hidden>) on branch: master
Review: https://review.openstack.org/420273
Reason: wait for full fix.

Eric K (ekcs)
Changed in congress:
assignee: nobody → Eric K (ekcs)
Changed in congress:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to congress (master)

Reviewed: https://review.openstack.org/420273
Committed: https://git.openstack.org/cgit/openstack/congress/commit/?id=cdfb121cf4eef96b917b0e3d7e86616b965e3506
Submitter: Jenkins
Branch: master

commit cdfb121cf4eef96b917b0e3d7e86616b965e3506
Author: Eric K <email address hidden>
Date: Fri Jan 13 19:10:28 2017 -0800

    Remove antlr3 files except Python runtime

    The removed files are not needed by users, and some files
    are incompatible with free software packaging (e.g. jar)

    Drawback: if a developer changes congress grammar, s/he
    needs to compile or download the antlr3 file to generate
    the Parser/Lexer.

    Instructions added to download antlr3 and generate files.

    Partial-Bug: 1591763

    Change-Id: I716ffe9f651dc48eb513efbb653aadf8f33f604a

Revision history for this message
Thomas Goirand (thomas-goirand) wrote :

Hi,

Thanks Eric for this patch. Now, it'd be nice to also remove the Python runtime files: since they are also available in Debian and Ubuntu, there's no reason to keep them embedded.

Also, CongressLexer.py CongressParser.py should be removed from the Git, as they are generated. In the Debian package, we had to remove them from the original sources.

Revision history for this message
Eric K (ekcs) wrote :

Thank you Thomas for the guidance.

Would it help packaging if we made the runtime and generated files unnecessary for python2 usage on newer distros (hence deletable for packaging), but still kept them in the repo for python3 tests (python3 runtime not available in distros) and to support older distros?

I plan to rewrite the parsing bits to remove antlr completely in the Pike cycle to properly support python3.

Also what are some critical packaging deadlines to be aware of?

Thanks again!

Eric K (ekcs)
Changed in congress:
milestone: none → pike-2
tags: added: pike-goal
Eric K (ekcs)
Changed in congress:
milestone: pike-2 → none
tags: removed: pike-goal
Revision history for this message
Thomas Goirand (thomas-goirand) wrote :

Eric, could you please reopen this bug? The 3rd party folder is still present in master, there's absolutely no reason to close this bug, especially that antlr3 is in both Debian and Ubuntu. Please do remove the 3rdparty folder.

Revision history for this message
Thomas Goirand (thomas-goirand) wrote :

Would it help if python3 runtime for antlr3 was packaged?

Revision history for this message
Eric K (ekcs) wrote :

Hi Thomas! Not closing the bug, but slipping to Queens.

If hitting Pike is really important, what I can do is this:
Keep the generated files and the remaining antlr runtime in the repo, but clearly document that they can be removed for python2 packaging. For python3, the antlr runtime is still required in repo because like you said it's not been packaged.

Thanks so much for helping us along!

Revision history for this message
Corey Bryant (corey.bryant) wrote :

Will this be fixed in Rocky? It's causing packaging issues in Ubuntu.

Revision history for this message
Thomas Goirand (thomas-goirand) wrote :

Same over here. There are packages for both py2 and py3 so there is no reason to keep this folder. Please make this a priority.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.