Suspended and resumed image import seemingly stuck

Bug #1390097 reported by Christian Reis
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Invalid
High
Unassigned

Bug Description

I suspended and resumed my laptop which was running an image import into MAAS. When I resumed it at 9am, I expected the image import to resume, but it seems stuck and although the page still shows spinners, they are stuck at 14% and 1% respectively.

Here's what I see in maas-django.log:

ERROR 2014-11-06 09:03:04,363 twisted Unhandled Error
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 783, in __bootstrap
    self.__bootstrap_inner()
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
--- <exception caught here> ---
  File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 191, in _worker
    result = context.call(ctx, function, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/usr/lib/python2.7/dist-packages/maasserver/utils/async.py", line 153, in call_within_transaction
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/maasserver/bootsources.py", line 117, in _cache_boot_sources
    image_descriptions = download_all_image_descriptions(sources)
  File "/usr/lib/python2.7/dist-packages/provisioningserver/import_images/download_descriptions.py", line 204, in download_all_image_descriptions
    source['url'], keyring=source.get('keyring', None))
  File "/usr/lib/python2.7/dist-packages/provisioningserver/import_images/download_descriptions.py", line 195, in download_image_descriptions
    dumper.sync(reader, rpath)
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 82, in sync
    content, payload = reader.read_json(path)
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 39, in read_json
    raw = self.source(path).read().decode('utf-8')
  File "/usr/lib/python2.7/dist-packages/simplestreams/contentsource.py", line 143, in read
    self.open()
  File "/usr/lib/python2.7/dist-packages/simplestreams/contentsource.py", line 139, in open
    self.fd = self._open()
  File "/usr/lib/python2.7/dist-packages/simplestreams/contentsource.py", line 127, in _open
    return opener(*oargs, offset=self.offset)
  File "/usr/lib/python2.7/dist-packages/simplestreams/contentsource.py", line 302, in __init__
    self.req = requests.get(url, stream=True, auth=auth, headers=headers)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 455, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 558, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 378, in send
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='maas.ubuntu.com', port=80): Max retries exceeded with url: /images/ephemeral-v2/releases/s
treams/v1/index.sjson (Caused by <class 'socket.gaierror'>: [Errno -2] Name or service not known)

followed by sstreams logging every hour with what appears to be the same set of images. I've attached the log.

Tags: boot-images
Revision history for this message
Christian Reis (kiko) wrote :
Revision history for this message
Christian Reis (kiko) wrote :

From the admin shell:

>>> print locks.import_images.is_locked()
True

Revision history for this message
Christian Reis (kiko) wrote :

async@malzinho:/var/log/maas$ ps auxww | grep wsgi
maas 3087 0.7 1.5 142704 48456 ? Sl Nov05 10:08 (wsgi:maas) -k start
maas 3088 0.8 1.6 192164 49352 ? Sl Nov05 11:10 (wsgi:maas) -k start

async@malzinho:/var/log/maas$ sudo strace -p 3087
Process 3087 attached
restart_syscall(<... resuming interrupted call ...>
async@malzinho:/var/log/maas$ sudo strace -p 3088
Process 3088 attached
restart_syscall(<... resuming interrupted call ...>

Revision history for this message
Christian Reis (kiko) wrote :

async@malzinho:/var/log/maas$ ps auxww | grep wsgi
maas 3087 0.7 1.5 142704 48456 ? Sl Nov05 10:08 (wsgi:maas) -k start
maas 3088 0.8 1.6 192164 49352 ? Sl Nov05 11:10 (wsgi:maas) -k start

async@malzinho:/var/log/maas$ sudo strace -p 3087
Process 3087 attached
restart_syscall(<... resuming interrupted call ...>
async@malzinho:/var/log/maas$ sudo strace -p 3088
Process 3088 attached
restart_syscall(<... resuming interrupted call ...>

Restarting apache2 now, which may have unjammed things, but the UI still shows 14% and 1% respectively.

Revision history for this message
Julian Edwards (julian-edwards) wrote : Re: [Bug 1390097] Re: Suspended and resumed image import seemingly stuck

Which version of MAAS? That error is getting caught in trunk now so I'd not
expect to see that there, but stranger things have happened...

It asks for version info in the bug reporting guidelines, do I need to make
that more obvious? (Genuine question)

Revision history for this message
Christian Reis (kiko) wrote :

I just forgot to mention; all my testing is generally on the release branch. This was on the revision which became RC2.

Christian Reis (kiko)
Changed in maas:
milestone: none → 1.7.1
Revision history for this message
Graham Binns (gmb) wrote :

Triaging as high, though because it's not all that common for regions or clusters to be suspended mid-import, it might drop down in priority.

Changed in maas:
status: New → Triaged
importance: Undecided → High
tags: added: boot-images
Revision history for this message
Christian Reis (kiko) wrote :

async@malzinho:/var/log/maas$ ps auxww | grep wsgi
maas 3087 0.7 1.5 142704 48456 ? Sl Nov05 10:08 (wsgi:maas) -k start
maas 3088 0.8 1.6 192164 49352 ? Sl Nov05 11:10 (wsgi:maas) -k start

async@malzinho:/var/log/maas$ sudo strace -p 3087
Process 3087 attached
restart_syscall(<... resuming interrupted call ...>
async@malzinho:/var/log/maas$ sudo strace -p 3088
Process 3088 attached
restart_syscall(<... resuming interrupted call ...>

Restarted apache2 since, which may have unjammed things, but the UI still showed 14% and 1% respectively after the restart.

Changed in maas:
milestone: 1.7.1 → 1.7.2
Changed in maas:
milestone: 1.7.2 → 1.7.3
Changed in maas:
status: Triaged → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.