[Gerrit] Wierd behavior of review.fuel-infra.org time to time

Hello!

Detailed bug description:

Time to time I see, the following error in Fuel Gerrit's log:

[2016-06-08 08:12:40,948] ERROR com.google.gerrit.pgm.http.jetty.HiddenErrorHandler : Error in GET /
com.google.inject.ProvisionException: Guice provision errors:

1) Cannot open ReviewDb
  at com.google.gerrit.server.util.ThreadLocalRequestContext$1.provideReviewDb(ThreadLocalRequestContext.java:70)
  while locating com.google.gerrit.reviewdb.server.ReviewDb

1 error
 at com.google.inject.internal.InjectorImpl$3.get(InjectorImpl.java:1014)
 at com.google.gerrit.server.IdentifiedUser.getAccountDiffPreference(IdentifiedUser.java:251)
 at com.google.gerrit.httpd.raw.HostPageServlet.doGet(HostPageServlet.java:188)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
 at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:278)
 at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:268)
 at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:180)
 at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:93)
 at com.google.gerrit.httpd.auth.openid.XrdsFilter.doFilter(XrdsFilter.java:32)
 at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:70)
 at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:113)
 at com.google.gerrit.httpd.RequireSslFilter.doFilter(RequireSslFilter.java:68)
 at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:64)
 at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:57)
 at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:75)
 at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120)
 at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:132)
 at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:129)
 at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:206)
 at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:129)
 at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1539)
 at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:524)
 at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:219)
 at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1110)
 at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:453)
 at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
 at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1044)
 at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
 at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
 at org.eclipse.jetty.server.Server.handle(Server.java:459)
 at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:280)
 at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:229)
 at org.eclipse.jetty.io.AbstractConnection$1.run(AbstractConnection.java:505)
 at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
 at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
 at java.lang.Thread.run(Thread.java:745)
Caused by: com.google.gwtorm.server.OrmException: Cannot open database connection
 at com.google.gwtorm.jdbc.Database.newConnection(Database.java:130)
 at com.google.gwtorm.jdbc.JdbcSchema.<init>(JdbcSchema.java:40)
 at com.google.gerrit.reviewdb.server.ReviewDb_Schema_GwtOrm$$22.<init>(Unknown Source)
 at com.google.gerrit.reviewdb.server.ReviewDb_Schema_GwtOrm$$22_Factory_GwtOrm$$23.open(Unknown Source)
 at com.google.gwtorm.jdbc.Database.open(Database.java:122)
 at com.google.gerrit.server.config.RequestScopedReviewDbProvider.get(RequestScopedReviewDbProvider.java:45)
 at com.google.gerrit.server.config.RequestScopedReviewDbProvider.get(RequestScopedReviewDbProvider.java:27)
 at com.google.gerrit.server.util.ThreadLocalRequestContext$1.provideReviewDb(ThreadLocalRequestContext.java:70)
 at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
 at com.google.inject.internal.ProviderMethod.get(ProviderMethod.java:105)
 at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:86)
 at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:55)
 at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:66)
 at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:47)
 at com.google.inject.internal.InjectorImpl$3$1.call(InjectorImpl.java:1005)
 at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1051)
 at com.google.inject.internal.InjectorImpl$3.get(InjectorImpl.java:1001)
 ... 36 more

It happens the second time for 08.06.2016 day.

As far I can see, the problem detects when https://packaging-ci.infra.mirantis.net/job/sync-upstream-openstack/ is working on syncing of Openstack upstream's code.

Steps to reproduce
1. Run the job
2. Open Web UI of Fuel Gerrit
3. Open Fuel Gerrit's logs

Expected results
Available Fuel Gerrit system.

Actual result
The system doesn't work properly.

Reproducibility
The third time after the job enabling.

Workaround:
Fuel Gerrit's application restart