glance-simplestreams-sync does not use new proxy settings

Subscribed field-medium as glance-simplestreams-sync usage is mentioned in the official guide for using octavia-diskimage-retrofit:

https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/app-octavia.html#amphora-image

The charm should be extended to use env_proxy_settings from charm-helpers to get proxy settings in either format and make glance-simplestreams-sync.py script invocations to use them.

see https://github.com/juju/charm-helpers/pull/248 for more information on handling juju-{http,https,no}-proxy model configs.

This is currently blocking the deploy of the charm in 2 Bootstack customers.

Reviewed: https://review.opendev.org/694174
Committed: https://git.openstack.org/cgit/openstack/charm-glance-simplestreams-sync/commit/?id=33b9da2bc5106a33ab0c87fb19056532a94de0dd
Submitter: Zuul
Branch: master

commit 33b9da2bc5106a33ab0c87fb19056532a94de0dd
Author: Alex Balderson <email address hidden>
Date: Wed Nov 13 14:37:45 2019 -0800

    Use current juju proxy config when running sync.

    Adds lines to source current juju proxy environment variables.

    Closes-Bug: #1843486

    Change-Id: I2b461ca6112839effcd7efa3bd9d821f92857584

This fix does not seem to work with current charm releases (tested from 30 to 32).

Proxy is configured in juju-*-proxy vars in juju model, but are *not* configured in /etc/environment. If I do set normal http_proxy vars, it works.

All the files sourced in the glance-simplestreams-sync.sh script are empty in the environment I'm deploying. That script is copied to /usr/share and runs from cron. From there, it just cannot use the juju-*-proxy vars anymore, and as I said, all places where it should be are empty.

I created a workaround in which on "install" hook I check if the juju proxy vars are set and write them to /etc/juju-proxy.conf (this is one of the sourced files in the script). It works, fixed the problem. I just don't know if this is the best approach.

The changed code is here: https://code.launchpad.net/~andre-ruiz/+git/charm-glance-simplestreams-sync-proxy-fix

The charm with the applied fix is here: https://jaas.ai/u/andre-ruiz/glance-simplestreams-sync/0

Just a note to inform that I updated the charm, latest version is:

cs:~andre-ruiz/glance-simplestreams-sync-1

https://jaas.ai/u/andre-ruiz/glance-simplestreams-sync/1

Reopened based on dup @ https://bugs.launchpad.net/charm-glance-simplestreams-sync/+bug/1883656.

FYI, Additional context in the conversation @ PR https://github.com/juju/juju/pull/11713.

The triage (next steps disposition) from the dup bug:

https://bugs.launchpad.net/charm-glance-simplestreams-sync/+bug/1883656/comments/9

Merge request in review: https://review.opendev.org/743022/

For what it's worth, I've been using https://jaas.ai/u/vern/glance-simplestreams-sync/2 while waiting for this merge request to land.

And I also agree with Dmitrii that simplestreams should support --mirror-proxy style arguments so that we can be more targeted with what connections require/use proxies and not have to rely on no_proxy.

Hi, can we please have this MR merged? We'll need this for the upcoming cloud deployment. Thanks in advance!

There are still outstanding questions on the review - hence its not been landed yet. Those reflect the comment in #11

If we are waiting on changes to sstream-mirror-glance, that comes from python3-simplestreams-openstack, which is built from simplestreams, it should be added to this bug.

I believe the request is to modify sstream-mirror-glance and add the ability to set something like mirror-http-proxy, mirror-https-proxy, mirror-no-proxy for the purpose of connecting to the mirror -- but not for any other connections.

The idea is that communication to the mirror sometimes requires a proxy whereas communication with glance and swift rarely does. In those rare situations, a system-wide proxy could be used. But mirror*proxy settings would allow us to better target what needs a proxy without having to carefully craft a no-proxy variable (which can sometimes be difficult if the subnet where glance and swift are is particularly large and they can get any IP in that subnet).

This sounds like a reasonable feature request to me.

any news on this ?
It breaks automatic amphora image creation when behind a proxy.

One workaround is to change the juju proxy settings to use the legacy "http_proxy/https_proxy" env variables then redeploy the charm but this makes all other charm deployment failing unless you revert the proxy env variables to new ones.

Best regards

I seem to be hitting this with Focal / Ussuri and with Gss charm rev 36. It will just do nothing after installed and set to "run=true". I tested with my modified charm (probably too old now, it was based on gss charm rev 32), did not work. Tested with manually setting proxy vars and calling the script (as per Vern's suggestion) and it also did not work. Will have to investigate more.

Just a clarification, the official templates have:

config/openstack_versioned_overlay.yaml: charm: cs:glance-simplestreams-sync-36

While the "stable" is at rev 23.

Please ignore my last comment about "stable is at 23", I was mixing numbers with a different problem.

Bumping this to ~field-high as it affects all of proxy based customer environments. And we cannot keep using random workarounds in the field.

Change abandoned by "Vern Hart <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/charm-glance-simplestreams-sync/+/743022
Reason: Abandoning to allow someone else to provide a solution.

We had a discussion about it during OE team's standup.

While exposing options to control proxy behavior in simplestreams would be preferred, it may take some time to implement this.

A viable alternative seems to be generating a NO_PROXY list from the model context since Juju knows about the Keystone and Swift API units that need to be taken into account for which proxying needs to be avoided.

Can we at least get the sync-images action to pull the env_proxy_settings and append them to the env so that the action does the right thing, while trying to figure out proper settings for the cronjob? Maybe even allowing setting a proxy as an arg to the action. We need a stop-gap that doesn't require logging into the machine to run the sync manually.

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-glance-simplestreams-sync/+/801077

Reviewed: https://review.opendev.org/c/openstack/charm-glance-simplestreams-sync/+/801077
Committed: https://opendev.org/openstack/charm-glance-simplestreams-sync/commit/009c8a7b929c9b961c9cf388c122cffe6f6ae41c
Submitter: "Zuul (22348)"
Branch: master

commit 009c8a7b929c9b961c9cf388c122cffe6f6ae41c
Author: Dmitrii Shcherbakov <email address hidden>
Date: Thu Jul 15 20:49:16 2021 +0300

    Dynamically generate proxy settings for image syncs

    sstream-mirror-glance has several endpoints it needs to talk to:

    * Image mirrors - typically, public Internet endpoints;
    * Keystone - typically, a directly reachable endpoint;
    * Glance - typically, a directly reachable endpoint;
    * Object store (Swift) - typically, a directly reachable endpoint but
      sometimes it may be deployed externally and added to the region
      catalog in Keystone (in which case it might be accessible via a proxy
      only).

    While sstream-mirror-glance does not support specifying proxy settings
    for individual directions, since we know all of them based on the
    Keystone catalog, a list of endpoints to add to NO_PROXY environment
    variable can be generated dynamically.

    The complication is that image syncs are periodically done via a cron
    job so a juju-run invocation is needed to retrieve relevant proxy
    settings from model-config at each invocation of the synchronization
    script.

    Additionally, the charm is long-lived so there may be some environments
    that rely on legacy proxy settings. This change accounts for that and
    acts both on juju-prefixed (new) and unprefixed (legacy) proxy settings.

    Whether to use proxy settings for connections to the object store API
    is controlled by a charm option which the script is made to react to.
    Proxy settings are ignored for object store connections by default.

    Closes-Bug: #1843486
    Change-Id: Ib1fc5d2eebf43d5f98bb8ee405a3799802c8b8dc

Fix proposed to branch: stable/21.04
Review: https://review.opendev.org/c/openstack/charm-glance-simplestreams-sync/+/802052

Reviewed: https://review.opendev.org/c/openstack/charm-glance-simplestreams-sync/+/802052
Committed: https://opendev.org/openstack/charm-glance-simplestreams-sync/commit/a18cca2ec0d843072c4cc732d6393110ac0124af
Submitter: "Zuul (22348)"
Branch: stable/21.04

commit a18cca2ec0d843072c4cc732d6393110ac0124af
Author: Dmitrii Shcherbakov <email address hidden>
Date: Thu Jul 15 20:49:16 2021 +0300

    Dynamically generate proxy settings for image syncs

    sstream-mirror-glance has several endpoints it needs to talk to:

    * Image mirrors - typically, public Internet endpoints;
    * Keystone - typically, a directly reachable endpoint;
    * Glance - typically, a directly reachable endpoint;
    * Object store (Swift) - typically, a directly reachable endpoint but
      sometimes it may be deployed externally and added to the region
      catalog in Keystone (in which case it might be accessible via a proxy
      only).

    While sstream-mirror-glance does not support specifying proxy settings
    for individual directions, since we know all of them based on the
    Keystone catalog, a list of endpoints to add to NO_PROXY environment
    variable can be generated dynamically.

    The complication is that image syncs are periodically done via a cron
    job so a juju-run invocation is needed to retrieve relevant proxy
    settings from model-config at each invocation of the synchronization
    script.

    Additionally, the charm is long-lived so there may be some environments
    that rely on legacy proxy settings. This change accounts for that and
    acts both on juju-prefixed (new) and unprefixed (legacy) proxy settings.

    Whether to use proxy settings for connections to the object store API
    is controlled by a charm option which the script is made to react to.
    Proxy settings are ignored for object store connections by default.

    Closes-Bug: #1843486
    Change-Id: Ib1fc5d2eebf43d5f98bb8ee405a3799802c8b8dc
    (cherry picked from commit 009c8a7b929c9b961c9cf388c122cffe6f6ae41c)

Unsubscribing ~field-high since the stable backport of the charm has been completed and the other task attached to this bug as simplestreams itself is not urgent at this moment.