not okForStorage error when deploying local charm

Bug #1308146 reported by Matthew Williams
24
This bug affects 4 people
Affects Status Importance Assigned to Milestone
juju-core
Fix Released
High
Unassigned
1.18
Fix Released
Low
Unassigned
1.20
Fix Released
Critical
Unassigned

Bug Description

Only happens on some local charms, not all.

from debug log:
ERROR error uploading charm: cannot update uploaded charm in state: not okForStorage
machine-0: no kvm containers possible

and forever after:

machine-0: 2014-04-15 16:31:04 ERROR juju resumer.go:68 worker/resumer: cannot resume transactions: not okForStorage
machine-0: 2014-04-15 16:32:04 ERROR juju resumer.go:68 worker/resumer: cannot resume transactions: not okForStorage
machine-0: 2014-04-15 16:33:04 ERROR juju resumer.go:68 worker/resumer: cannot resume transactions: not okForStorage

juju version: 1.18.1-precise-amd64.

Happening on aws

Tags: charms deploy

Related branches

Curtis Hovey (sinzui)
Changed in juju-core:
status: New → Triaged
importance: Undecided → High
tags: added: charms deploy
Revision history for this message
John A Meinel (jameinel) wrote :

not okForStorage seems to be MongoDB's way of telling you it doesn't accept the syntax of your document.
The ones that come up while googling are having either "." or "$" in your field names (they are allowed in values).

Revision history for this message
Matthew Williams (mattyw) wrote :

The problem was down to having bad character in the config.yaml field names. In this case a full stop "."

e.g:

options:
  db.path:
    default: "/var/lib/db"
    type: string
    description: The path where the data files will be kept.

Revision history for this message
John A Meinel (jameinel) wrote : Re: [Bug 1308146] Re: not okForStorage error when deploying local charm

We should try to teach something to warn about this sort of configuration,
since Mongo won't allow keys to have "." or "$" in them.

On Wed, Apr 16, 2014 at 11:36 AM, Matthew Williams <
<email address hidden>> wrote:

> The problem was down to having bad character in the config.yaml field
> names. In this case a full stop "."
>
> e.g:
>
> options:
> db.path:
> default: "/var/lib/db"
> type: string
> description: The path where the data files will be kept.
>
> --
> You received this bug notification because you are subscribed to juju-
> core.
> https://bugs.launchpad.net/bugs/1308146
>
> Title:
> not okForStorage error when deploying local charm
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-core/+bug/1308146/+subscriptions
>

Revision history for this message
Jacek Nykis (jacekn) wrote :

I just hit what looks like this bug:
ERROR error uploading charm: cannot update uploaded charm in state: not okForStorage

However "." in the charm option names are not a problem in another environment (deployed from the same server). For example I have "url.base" option and it works fine.

The only difference between the two is that one which works was deployed with older juju version, I can't tell which one exactly but likely 1.16.X

Mark Ramm (mark-ramm)
Changed in juju-core:
importance: High → Critical
Revision history for this message
Nate Finch (natefinch) wrote :

This was brought up today by James Westby:

We have an existing environment that we are trying to redeploy to a new
openstack deployment. One of the charms makes extensive use of dots in
the config keys, and is getting a "not okForStorage" error when trying
to deploy.

It sounds very much like
https://bugs.launchpad.net/juju-core/+bug/1308146.

We could rewrite the charm to avoid the dots, however:

  - Doing a smooth transition would be very difficult.
  - The charm makes extensive use of these, in particular the config
    file that they are serialized to uses them as well, so there would
    have to be a lot of code added to transform the keys in the charm.

It's not exactly clear from the bug log when this regression occured,
and the bug suggests it may only affect local charms.

Could someone look at this and tell us:

  - Whether it is considered a regression that will be fixed, or whether
    this is now a restriction on charms?
  - If it is a regression when we can expect a fix, and whether it will
    be backported to 1.18?

Otherwise I will get on with modifying the charm to avoid the dots, with
all the hassles that brings.

Revision history for this message
Nate Finch (natefinch) wrote :

The charm that is having the problem is lp:~ubuntuone-pqm-team/canonical-is-charms/ubuntuone-servers

Revision history for this message
Nate Finch (natefinch) wrote :

We do use the configuration names as a key in mongo, but we escape any periods or dollar signs before they are saved to the database, specifically to avoid this problem, so it is odd that this is still happening.

Revision history for this message
James Westby (james-w) wrote :

So it looks like 1.16 and earlier didn't use the upload HTTP api provided by the state server? That's presumably
why we are seeing this now? And if there are some codepaths that do the escaping it is because the HTTP
API doesn't, and so that's the regression?

Revision history for this message
Nate Finch (natefinch) wrote :

The bug is trivial to reproduce, I made what is essentially an empty charm, with a config.yaml that looks like this:

options:
  foo.bar:
    type: int
    default: 3
    description: Number of foo.bars.

And it causes the not okForStorage error.

you can reproduce it by doing juju deploy cs:~natefinch/testcharm

Revision history for this message
Dimiter Naydenov (dimitern) wrote :

The problem is even though we *do* escape $ and . for service settings stored in the settings collection, we don't do the same for charm config stored in the charms collection (either uploaded locally or from the charm store). I'm afraid this has been an issue ever since I introduced API for uploading charms, so I'm assigning this to myself and will propose a fix which does proper escaping.

Changed in juju-core:
status: Triaged → In Progress
assignee: nobody → Dimiter Naydenov (dimitern)
milestone: none → next-stable
Revision history for this message
Dimiter Naydenov (dimitern) wrote :

Proposed fix for review at http://reviews.vapour.ws/r/55/

Changed in juju-core:
status: In Progress → Fix Committed
Revision history for this message
Dimiter Naydenov (dimitern) wrote :

Fix landed in trunk in https://github.com/juju/juju/pull/786.
Backported to 1.20, proposed as https://github.com/juju/juju/pull/790, approved and pending to land (bot issues).
Working on a backport for 1.18 as well, as it's the release we have in trusty.

Curtis Hovey (sinzui)
Changed in juju-core:
milestone: next-stable → 1.21-alpha2
Revision history for this message
Dimiter Naydenov (dimitern) wrote :

All fixes and backports landed where they belong.

Revision history for this message
Curtis Hovey (sinzui) wrote :

We don't have any facility to test a juju 1.18.5. We can checkout an old version of juju release scripts from when 1.18.4 and attempt to create a release tarball and source package. There is no guarantee this will work because developers can and have moved deps in Lp and github. When CI detects this, we ask the developers to put the deps back. Since the release of 1.18.4, juju moved most of its packages to github.

Curtis Hovey (sinzui)
Changed in juju-core:
importance: Critical → High
Curtis Hovey (sinzui)
Changed in juju-core:
status: Fix Committed → Fix Released
Changed in juju-core:
assignee: Dimiter Naydenov (dimitern) → nobody
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.