Ubuntu
maas package

Upgrade from 1.95 to 2.35 failure because column maasserver_domain.ttl does not exist

Xenial (16.04)
Bug #1880495

Bug #1880495 reported by Xav Paice on 2020-05-25

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
MAAS	Triaged	Medium	Unassigned	MAAS 3.4.x
maas (Ubuntu)	New	Undecided	Unassigned
Xenial	New	Undecided	Unassigned

Bug Description

Upgrading an older Trusty based Maas (1.9.5) to Xenial (2.3.5) resulted in a traceback during the do-release-upgrade procedure:

  Applying maasserver.0011_domain_data...Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
psycopg2.ProgrammingError: column maasserver_domain.ttl does not exist
LINE 1: ...ain"."name", "maasserver_domain"."authoritative", "maasserve...
                                                             ^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3/dist-packages/maasserver/__main__.py", line 9, in <module>
    raise SystemExit(execute_from_command_line())
  File "/usr/lib/python3/dist-packages/maasserver/__init__.py", line 73, in execute_from_command_line
    management.execute_from_command_line()
  File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 354, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 346, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 394, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 445, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python3/dist-packages/maasserver/management/commands/dbupgrade.py", line 426, in handle
    fake_initial=self._south_was_performed(database))
  File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 120, in call_command
    return command.execute(*args, **defaults)
  File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 445, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python3/dist-packages/django/core/management/commands/migrate.py", line 222, in handle
    executor.migrate(targets, plan, fake=fake, fake_initial=fake_initial)
  File "/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 110, in migrate
    self.apply_migration(states[migration], migration, fake=fake, fake_initial=fake_initial)
  File "/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 148, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/lib/python3/dist-packages/django/db/migrations/migration.py", line 115, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/lib/python3/dist-packages/django/db/migrations/operations/special.py", line 183, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/usr/lib/python3/dist-packages/maasserver/migrations/builtin/maasserver/0011_domain_data.py", line 109, in migrate_staticipaddress_hostname
    domain_id = maasserver.models.dnsresource.get_default_domain()
  File "/usr/lib/python3/dist-packages/maasserver/models/dnsresource.py", line 57, in get_default_domain
    return Domain.objects.get_default_domain().id
  File "/usr/lib/python3/dist-packages/maasserver/models/domain.py", line 108, in get_default_domain
    'updated': now,
  File "/usr/lib/python3/dist-packages/django/db/models/manager.py", line 127, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 405, in get_or_create
    return self.get(**lookup), False
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 328, in get
    num = len(clone)
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 144, in __len__
    self._fetch_all()
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 965, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 238, in iterator
    results = compiler.execute_sql()
  File "/usr/lib/python3/dist-packages/django/db/models/sql/compiler.py", line 840, in execute_sql
    cursor.execute(sql, params)
  File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/lib/python3/dist-packages/django/db/utils.py", line 98, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/lib/python3/dist-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: column maasserver_domain.ttl does not exist
LINE 1: ...ain"."name", "maasserver_domain"."authoritative", "maasserve...
                                                             ^

dpkg: error processing package maas-region-controller (--configure):
subprocess installed post-installation script returned error exit status 1
No apport report written because the error message indicates its a followup error from a previous failure.
dpkg: dependency problems prevent configuration of maas:
maas depends on maas-region-controller (= 2.3.5-6511-gf466fdb-0ubuntu1); however:
Package maas-region-controller is not configured yet.

To reproduce, install Maas on Trusty, with a working database, and run do-release-upgrade.

I suspect there's some edge case in this particular database, working now on trying to identify it. The particular database has been around for at least 3 years now.

Related branches

~freyes/maas:lp1880495

Rejected for merging into maas:2.3

Adam Collard (community): Needs Fixing on 2020-06-24

MAAS Lander: Approve on 2020-05-26

Revision history for this message

Xav Paice (xavpaice) wrote on 2020-05-25:

Download full text (11.7 KiB)

Given this is a test, I tried adding the missing column:

maasdb=# alter table maasserver_domain add column ttl integer;
ALTER TABLE
maasdb=# \d maasserver_domain
                                     Table "public.maasserver_domain"
    Column | Type | Modifiers
---------------+--------------------------+----------------------------------------------------------------
id | integer | not null default nextval('maasserver_domain_id_seq'::regclass)
created | timestamp with time zone | not null
updated | timestamp with time zone | not null
name | character varying(256) | not null
authoritative | boolean |
ttl | integer |
Indexes:
    "maasserver_domain_pkey" PRIMARY KEY, btree (id)
    "maasserver_domain_name_key" UNIQUE CONSTRAINT...

Given this is a test, I tried adding the missing column:

postgres=# \connect maasdb                                                      
You are now connected to database "maasdb" as user "postgres".                  
maasdb=# \d maasserver_domain                                                   
                                     Table "public.maasserver_domain"           
    Column     |           Type           |                           Modifiers 
---------------+--------------------------+----------------------------------------------------------------
 id            | integer                  | not null default nextval('maasserver_domain_id_seq'::regclass)
 created       | timestamp with time zone | not null                            
 updated       | timestamp with time zone | not null                            
 name          | character varying(256)   | not null                            
 authoritative | boolean                  |                                     
Indexes:                                                                        
    "maasserver_domain_pkey" PRIMARY KEY, btree (id)                            
    "maasserver_domain_name_key" UNIQUE CONSTRAINT, btree (name)                
    "maasserver_domain_946f3fba" btree (authoritative)                          
    "maasserver_domain_name_4403c8e5d405ce97_like" btree (name varchar_pattern_ops)
Referenced by:                                                                  
    TABLE "maasserver_dnsresource" CONSTRAINT "maasserver_d_domain_id_48f70cfc2a2704fa_fk_maasserver_domain_id" FOREIGN KEY (domain_id) REFERENCES maasserver_domain(id) DEFERRABLE INITIALLY DEFERRED
    TABLE "maasserver_node" CONSTRAINT "maasserver_n_domain_id_67d5fcf688c8fcd5_fk_maasserver_domain_id" FOREIGN KEY (domain_id) REFERENCES maasserver_domain(id) DEFERRABLE INITIALLY DEFERRED
                                                                                
maasdb=# alter table maasserver_domain add column ttl integer;                  
ALTER TABLE                                                                     
maasdb=# \d maasserver_domain                                                   
                                     Table "public.maasserver_domain"           
    Column     |           Type           |                           Modifiers 
---------------+--------------------------+----------------------------------------------------------------
 id            | integer                  | not null default nextval('maasserver_domain_id_seq'::regclass)
 created       | timestamp with time zone | not null                            
 updated       | timestamp with time zone | not null                            
 name          | character varying(256)   | not null                            
 authoritative | boolean                  |                                     
 ttl           | integer                  |                                     
Indexes:                                                                        
    "maasserver_domain_pkey" PRIMARY KEY, btree (id)                            
    "maasserver_domain_name_key" UNIQUE CONSTRAINT, btree (name)                
    "maasserver_domain_946f3fba" btree (authoritative)                          
    "maasserver_domain_name_4403c8e5d405ce97_like" btree (name varchar_pattern_ops)
Referenced by:                                                                  
    TABLE "maasserver_dnsresource" CONSTRAINT "maasserver_d_domain_id_48f70cfc2a2704fa_fk_maasserver_domain_id" FOREIGN KEY (domain_id) REFERENCES maasserver_domain(id) DEFERRABLE INITIALLY DEFERRED
    TABLE "maasserver_node" CONSTRAINT "maasserver_n_domain_id_67d5fcf688c8fcd5_fk_maasserver_domain_id" FOREIGN KEY (domain_id) REFERENCES maasserver_domain(id) DEFERRABLE INITIALLY DEFERRED

That allowed the migration to progress a little further, but still failing with a different traceback:

Setting up maas-region-controller (2.3.5-6511-gf466fdb-0ubuntu1) ...            
dbconfig-common: writing config to /etc/dbconfig-common/maas-region-controller.conf
creating postgres user maas:  already exists.                                   
resetting password:  success.                                                   
creating database maasdb: already exists.                                       
dbconfig-common: flushing administrative password                               
Operations to perform:                                                          
  Synchronize unmigrated apps: staticfiles, messages                            
  Apply all migrations: sites, contenttypes, metadataserver, piston3, maasserver, sessions, auth
Synchronizing apps without migrations:                                          
  Creating tables...                                                            
    Running deferred SQL...                                                     
  Installing custom SQL...                                                      
Running migrations:                                                             
  Rendering model states... DONE                                                
  Applying maasserver.0011_domain_data...Traceback (most recent call last):     
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 405, in get_or_create
    return self.get(**lookup), False                                            
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 334, in get
    self.model._meta.object_name                                                
__fake__.DoesNotExist: DNSResource matching query does not exist.               
                                                                                
During handling of the above exception, another exception occurred:             
                                                                                
Traceback (most recent call last):                                              
  File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main          
    "__main__", mod_spec)                                                       
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code                     
    exec(code, run_globals)                                                     
  File "/usr/lib/python3/dist-packages/maasserver/__main__.py", line 9, in <module>
    raise SystemExit(execute_from_command_line())                               
  File "/usr/lib/python3/dist-packages/maasserver/__init__.py", line 73, in execute_from_command_line
    management.execute_from_command_line()                                      
  File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 354, in execute_from_command_line
    utility.execute()                                                           
  File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 346, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)                     
  File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 394, in run_from_argv
    self.execute(*args, **cmd_options)                                          
  File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 445, in execute
    output = self.handle(*args, **options)                                      
  File "/usr/lib/python3/dist-packages/maasserver/management/commands/dbupgrade.py", line 426, in handle
    fake_initial=self._south_was_performed(database))                           
  File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 120, in call_command
    return command.execute(*args, **defaults)                                   
  File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 445, in execute
    output = self.handle(*args, **options)                                      
  File "/usr/lib/python3/dist-packages/django/core/management/commands/migrate.py", line 222, in handle
    executor.migrate(targets, plan, fake=fake, fake_initial=fake_initial)       
  File "/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 110, in migrate
    self.apply_migration(states[migration], migration, fake=fake, fake_initial=fake_initial)
  File "/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 148, in apply_migration
    state = migration.apply(state, schema_editor)                               
  File "/usr/lib/python3/dist-packages/django/db/migrations/migration.py", line 115, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/lib/python3/dist-packages/django/db/migrations/operations/special.py", line 183, in database_forwards
    self.code(from_state.apps, schema_editor)                                   
  File "/usr/lib/python3/dist-packages/maasserver/migrations/builtin/maasserver/0011_domain_data.py", line 116, in migrate_staticipaddress_hostname
    'updated': now,                                                             
  File "/usr/lib/python3/dist-packages/django/db/models/manager.py", line 127, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)                  
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 407, in get_or_create
    return self._create_object_from_params(lookup, params)                      
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 439, in _create_object_from_params
    obj = self.create(**params)                                                 
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 348, in create
    obj.save(force_insert=True, using=self.db)                                  
  File "/usr/lib/python3/dist-packages/maasserver/models/cleansave.py", line 28, in save
    self.full_clean()                                                           
  File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 1149, in full_clean
    self.clean_fields(exclude=exclude)                                          
  File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 1191, in clean_fields
    setattr(self, f.attname, f.clean(raw_value, self))                          
  File "/usr/lib/python3/dist-packages/django/db/models/fields/__init__.py", line 589, in clean
    self.run_validators(value)                                                  
  File "/usr/lib/python3/dist-packages/django/db/models/fields/__init__.py", line 541, in run_validators
    v(value)                                                                    
TypeError: validate_dnsresource_name() missing 1 required positional argument: 'rrtype'
dpkg: error processing package maas-region-controller (--configure):            
 subprocess installed post-installation script returned error exit status 1     
No apport report written because the error message indicates its a followup error from a previous failure.
                                                                                                          dpkg: dependency problems prevent configuration of maas:
 maas depends on maas-region-controller (= 2.3.5-6511-gf466fdb-0ubuntu1); however:
  Package maas-region-controller is not configured yet.                         
                                                                                
dpkg: error processing package maas (--configure):                              
 dependency problems - leaving unconfigured                                     
Errors were encountered while processing:                                       
 maas-region-controller                                                         
 maas                                                                           
E: Sub-process /usr/bin/dpkg returned an error code (1)

Revision history for this message

Felipe Reyes (freyes) wrote on 2020-05-26:

I could reproduce this issue locally, and this is what I believe is happening with this environment (and not for other upgrades) is that the database has no domains defined for the interfaces related, so this piece of code [0] is falling into the "else" section which calls get_default_domain() method and internally ends up query the 'ttl' column. So the migration step 0011 is relying on it and expecting that it was created in step 0010 and it wasn't, it was added way later in migration step 0023.

[0] https://github.com/maas/maas/blob/2.3/src/maasserver/migrations/builtin/maasserver/0011_domain_data.py#L106-L109

Revision history for this message

Dougal Matthews (d0ugal) wrote on 2020-05-26:

Yup, as Felipe links I think the problem here is that we are using the models within the database migration. However, the models assume all migrations have been applied. The call to `maasserver.models.dnsresource.get_default_domain()` should be replaced with something that doesn't use the models.

Changed in maas:
status:	New → Confirmed

Felipe Reyes (freyes) on 2020-05-26

Changed in maas:
assignee:	nobody → Felipe Reyes (freyes)

Dougal Matthews (d0ugal) on 2020-05-27

Changed in maas:
status:	Confirmed → Triaged

Felipe Reyes (freyes) on 2021-04-20

Changed in maas:
assignee:	Felipe Reyes (freyes) → nobody

Revision history for this message

Jerzy Husakowski (jhusakowski) wrote on 2022-06-09:

MAAS upgrade logic should not rely on MAAS models sources. This issue appeared on an old version of MAAS, but may appear in other upgrade scenarios. We will fix this on the current MAAS versions and do not plan to backport the fix to older versions.

Changed in maas:
importance:	Undecided → Medium
milestone:	none → 3.3.0
importance:	Medium → High

Revision history for this message

Jerzy Husakowski (jhusakowski) wrote on 2022-10-06:

This issue requires further analysis of MAAS upgrade mechanism, to determine if it's possible to prevent such problems, e.g. by eliminating references to MAAS code in migration logic

Changed in maas:
importance:	High → Medium
milestone:	3.3.0 → 3.4.0

Alberto Donato (ack) on 2023-06-29

Changed in maas:
milestone:	3.4.0 → 3.4.x

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

Ubuntumaas package

Upgrade from 1.95 to 2.35 failure because column maasserver_domain.ttl does not exist

Bug Description

Related branches

Other bug subscribers

Remote bug watches

Ubuntu
maas package