Add support for converting to Raw images

Hi Yoshi,

This sounds like a reasonable feature request, thanks for filing it!

Dan

Also made a merge request to glance-simplestreams-sync charm.
This is still WIP. Unit test is not implemented yet.

Sorry forgot to add the link of the merge request.
https://review.opendev.org/#/c/756047/

This would be really great to have. Ceph native storage format is raw. When booting from volume, the images must be uploaded as raw.

Any chance we could get this merged?

@Yoshi: thanks for your MP. Are you planning to move it ahead of its WIP state and submit it for review?

+1 this is really needed when using Ceph or S3 storage backend for Glance otherwise you'll have to convert images to raw and re-import them to benefit from advanced features with native backend Copy on Write.
Could simply be a charm configuration parameter to let us choose the image format (qcow2 or raw).

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-glance-simplestreams-sync/+/809392

Sorry for the late reply.
I have updated for both charm and the simplestream snap package.
The approach is a bit different now, and this utilizes the "image_conversion" function available in Glance.
https://bugs.launchpad.net/charm-glance/+bug/1899767

At the moment, the above Glance fix is only available in the -next charm[0].
So the MPs that I have raised will be only usable with the -next charm at the moment.

[0] https://jaas.ai/u/openstack-charmers-next/glance/469#charm-config-image-conversion

Change abandoned by "James Page <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/charm-glance-simplestreams-sync/+/809392
Reason: This review is > 12 weeks without comment, and failed testing the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Can someone mark this as deprecated or 'won't fix?' There's a better solution that is already in place in the charm-glance charm[1]. That not only solves the problem for images sync by simplestreams but also for other user-uploaded images.

[1] https://charmhub.io/glance/configure#image-conversion

You must have misunderstood something. These two bug tasks are exactly for leveraging glance conversion feature from g-s-s through simplestreams. Please take a look at
https://review.opendev.org/c/openstack/charm-glance/+/843546
For more context.

This bug is fixed with commit cbc0ac9d to simplestreams on branch master.
To view that commit see the following URL:
https://git.launchpad.net/simplestreams/commit/?id=cbc0ac9d

This bug for glance-simplestreams-sync is still valid. If the glance charm has been configured with image-conversion=true and we use image_import instead of upload, the image will get converted to raw. It looks like Yoshi's fix here is valid, though seems it needs to be rebased: https://review.opendev.org/c/openstack/charm-glance-simplestreams-sync/+/809392

Reviewed: https://review.opendev.org/c/openstack/charm-glance-simplestreams-sync/+/809392
Committed: https://opendev.org/openstack/charm-glance-simplestreams-sync/commit/536540781ae15c9793120f875a196f1954968d01
Submitter: "Zuul (22348)"
Branch: master

commit 536540781ae15c9793120f875a196f1954968d01
Author: Yoshi Kadokawa <email address hidden>
Date: Thu Sep 16 11:24:56 2021 +0900

    Add support of image format conversion on importing

    Optinally enable conversion of images to raw format when uploading
    images to Glance. Only supported when "image_conversion" is enabled in
    Glance.

    This is an equivalent change to the one in simplestreams:
    https://git.launchpad.net/simplestreams/commit/?id=cbc0ac9

    Closes-Bug: #1889729
    Change-Id: Id15fab364c27242347c520b959065c490e4da6b6

I tested this fix. I set the glance-simplestreams-sync charm to latest/edge and the simplestreams snap to latest/edge (setting the gss charm's snap-channel did not have any effect).

Then I deleted my auto-sync'd bionic image and ran the sync-images action. A new bionic image showed up and it was RAW format. Nice!

However, each subsequent time I run the sync-images action, I get a new bionic image imported with the exact name as the existing one. I tried the same thing with the jammy image: deleted, ran sync-images twice, I end up with two raw jammy images. (And some more bionic images.)

It seems the code that detects if an image already exists doesn't see the raw images as equals. Note that my qcow2 focal image is not getting re-imported with each run.

Ah, the new raw images don't have any simplestreams-metadata. It seems probably that this was stripped when the image was converted.

It seems like the charm is doing the right thing but now we're back to the simplestreams snap.

I believe the simplestreams snap should be responsible for capturing the simplestreams-metadata and re-applying it to the newly converted image. Otherwise, new images will be created every time the sync job is run (daily by default, I think).

@Vern, thanks for the detailed test report. That definitely sounds we need follow-up activities.