git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: MySQL HA


Yes, I'll keep it and do some stress tests on it to be sure about its
functionality.

On Dec 18, 2017 14:53, "Rafael Weingärtner" <rafaelweingartner@xxxxxxxxx>
wrote:

> So, this fixed the problem?
> Can you keep this running for a while longer? Just to make sure. Then, I
> can open a PR to fix it in master.
>
> On Mon, Dec 18, 2017 at 9:02 AM, Alireza Eskandari <
> astro.alireza@xxxxxxxxx>
> wrote:
>
> > Thank you Rafael,
> > I test your fix and it seems that I have got the expected result. You
> > can see the exception raised for database failover.
> > I should notice I replace the file for cloudstack-mnagement and
> > cloudstack-usage:
> > /usr/share/cloudstack-usage/lib/cloud-framework-cluster-4.9.3.0.jar
> > /usr/share/cloudstack-management/webapps/client/WEB-
> > INF/lib/cloud-framework-cluster-4.9.3.0.jar
> >
> >
> > Logs:
> >
> > WARN  [c.c.c.d.ManagementServerHostDaoImpl]
> > (Cluster-Heartbeat-1:ctx-073cca55) (logid:e652d00b) Unexpected
> > exception,
> > com.cloud.utils.exception.CloudRuntimeException: Unable to commit or
> > close the connection.
> >         at com.cloud.utils.db.TransactionLegacy.commit(
> > TransactionLegacy.java:740)
> >         at com.cloud.cluster.dao.ManagementServerHostDaoImpl.update(
> > ManagementServerHostDaoImpl.java:140)
> >         at sun.reflect.GeneratedMethodAccessor103.invoke(Unknown Source)
> >         at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > DelegatingMethodAccessorImpl.java:43)
> >         at java.lang.reflect.Method.invoke(Method.java:606)
> >         at org.springframework.aop.support.AopUtils.
> > invokeJoinpointUsingReflection(AopUtils.java:317)
> >         at org.springframework.aop.framework.ReflectiveMethodInvocation.
> > invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> >         at org.springframework.aop.framework.ReflectiveMethodInvocation.
> > proceed(ReflectiveMethodInvocation.java:150)
> >         at com.cloud.utils.db.TransactionContextInterceptor.invoke(
> > TransactionContextInterceptor.java:34)
> >         at org.springframework.aop.framework.ReflectiveMethodInvocation.
> > proceed(ReflectiveMethodInvocation.java:161)
> >         at org.springframework.aop.interceptor.
> > ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> >         at org.springframework.aop.framework.ReflectiveMethodInvocation.
> > proceed(ReflectiveMethodInvocation.java:172)
> >         at org.springframework.aop.framework.JdkDynamicAopProxy.
> > invoke(JdkDynamicAopProxy.java:204)
> >         at com.sun.proxy.$Proxy203.update(Unknown Source)
> >         at com.cloud.cluster.ClusterManagerImpl$4.runInContext(
> > ClusterManagerImpl.java:555)
> >         at org.apache.cloudstack.managed.context.
> > ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >         at org.apache.cloudstack.managed.context.impl.
> > DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >         at org.apache.cloudstack.managed.context.impl.
> > DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >         at org.apache.cloudstack.managed.context.impl.
> > DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >         at org.apache.cloudstack.managed.context.
> > ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >         at java.util.concurrent.Executors$RunnableAdapter.
> > call(Executors.java:473)
> >         at java.util.concurrent.FutureTask.runAndReset(
> > FutureTask.java:304)
> >         at java.util.concurrent.ScheduledThreadPoolExecutor$
> > ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> >         at java.util.concurrent.ScheduledThreadPoolExecutor$
> > ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> >         at java.util.concurrent.ThreadPoolExecutor.runWorker(
> > ThreadPoolExecutor.java:1152)
> >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > ThreadPoolExecutor.java:622)
> >         at java.lang.Thread.run(Thread.java:748)
> > Caused by: com.mysql.jdbc.exceptions.jdbc4.
> MySQLTransactionRollbackExcept
> > ion:
> > Deadlock found when trying to get lock; try restarting transaction
> >         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > Method)
> >         at sun.reflect.NativeConstructorAccessorImpl.newInstance(
> > NativeConstructorAccessorImpl.java:57)
> >         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
> > DelegatingConstructorAccessorImpl.java:45)
> >         ... 46 more
> > INFO  [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (AsyncJobMgr-Heartbeat-1:ctx-5ef0f4d1) (logid:4bfa48b2) Begin cleanup
> > expired async-jobs
> > INFO  [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (AsyncJobMgr-Heartbeat-1:ctx-5ef0f4d1) (logid:4bfa48b2) End cleanup
> > expired async-jobs
> > ERROR [c.c.u.d.ConnectionConcierge]
> > (ConnectionConcierge-1:ctx-d3460aeb) (logid:b8c62262) Unable to keep
> > the db connection for LockMaster1
> > java.sql.SQLException: Connection was killed
> >         at com.mysql.jdbc.SQLError.createSQLException(SQLError.
> java:1073)
> >         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
> >         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
> >         at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990)
> >         at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
> >         at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:
> 2625)
> >         at com.mysql.jdbc.LoadBalancedMySQLConnection.execSQL(
> > LoadBalancedMySQLConnection.java:155)
> >         at com.mysql.jdbc.PreparedStatement.executeInternal(
> > PreparedStatement.java:2119)
> >         at com.mysql.jdbc.PreparedStatement.executeQuery(
> > PreparedStatement.java:2283)
> >         at sun.reflect.GeneratedMethodAccessor75.invoke(Unknown Source)
> >         at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > DelegatingMethodAccessorImpl.java:43)
> >         at java.lang.reflect.Method.invoke(Method.java:606)
> >         at com.mysql.jdbc.LoadBalancingConnectionProxy$
> > ConnectionErrorFiringInvocationHandler.invoke(
> > LoadBalancingConnectionProxy.java:103)
> >         at com.mysql.jdbc.FailoverConnectionProxy$
> > FailoverInvocationHandler.invoke(FailoverConnectionProxy.java:51)
> >         at com.sun.proxy.$Proxy257.executeQuery(Unknown Source)
> >         at org.apache.commons.dbcp.DelegatingPreparedStatement.
> > executeQuery(DelegatingPreparedStatement.java:96)
> >         at org.apache.commons.dbcp.DelegatingPreparedStatement.
> > executeQuery(DelegatingPreparedStatement.java:96)
> >         at com.cloud.utils.db.ConnectionConcierge$
> > ConnectionConciergeManager.testValidity(ConnectionConcierge.java:148)
> >         at com.cloud.utils.db.ConnectionConcierge$
> > ConnectionConciergeManager$1.runInContext(ConnectionConcierge.java:203)
> >         at org.apache.cloudstack.managed.context.
> > ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >         at org.apache.cloudstack.managed.context.impl.
> > DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >         at org.apache.cloudstack.managed.context.impl.
> > DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >         at org.apache.cloudstack.managed.context.impl.
> > DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >         at org.apache.cloudstack.managed.context.
> > ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >         at java.util.concurrent.Executors$RunnableAdapter.
> > call(Executors.java:473)
> >         at java.util.concurrent.FutureTask.runAndReset(
> > FutureTask.java:304)
> >         at java.util.concurrent.ScheduledThreadPoolExecutor$
> > ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> >         at java.util.concurrent.ScheduledThreadPoolExecutor$
> > ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> >         at java.util.concurrent.ThreadPoolExecutor.runWorker(
> > ThreadPoolExecutor.java:1152)
> >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > ThreadPoolExecutor.java:622)
> >         at java.lang.Thread.run(Thread.java:748)
> > INFO  [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (AsyncJobMgr-Heartbeat-1:ctx-22399f19) (logid:0deff5fe) Begin cleanup
> > expired async-jobs
> > INFO  [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (AsyncJobMgr-Heartbeat-1:ctx-22399f19) (logid:0deff5fe) End cleanup
> > expired async-jobs
> > INFO  [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (AsyncJobMgr-Heartbeat-1:ctx-6004a86b) (logid:64e4a3b3) Begin cleanup
> > expired async-jobs
> > INFO  [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (AsyncJobMgr-Heartbeat-1:ctx-6004a86b) (logid:64e4a3b3) End cleanup
> > expired async-jobs
> > INFO  [c.c.h.v.m.HostMO] (DirectAgent-19:ctx-351cbbac
> > host01.cloud.local, cmd: GetVmStatsCommand) (logid:56c8e250) VM
> > i-2-124-VM not found in host cache
> > INFO  [c.c.h.v.m.HostMO] (DirectAgent-1:ctx-9c0f5042
> > host02.cloud.local, cmd: GetVmStatsCommand) (logid:56c8e250) VM
> > i-2-125-VM not found in host cache
> > ERROR [c.c.u.d.ConnectionConcierge]
> > (ConnectionConcierge-1:ctx-544154d7) (logid:bd0f585a) Unable to keep
> > the db connection for LockMaster1
> > com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
> > Communications link failure
> >
> > The last packet successfully received from the server was 19,479
> > milliseconds ago.  The last packet sent successfully to the server was
> > 19,479 milliseconds ago.
> >         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > Method)
> >         at sun.reflect.NativeConstructorAccessorImpl.newInstance(
> > NativeConstructorAccessorImpl.java:57)
> >         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
> > DelegatingConstructorAccessorImpl.java:45)
> >         at java.lang.reflect.Constructor.newInstance(Constructor.java:
> 526)
> >         at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> >         at com.mysql.jdbc.SQLError.createCommunicationsException(
> > SQLError.java:1116)
> >         at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3352)
> >         at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1971)
> >         at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
> >         at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:
> 2625)
> >         at com.mysql.jdbc.LoadBalancedMySQLConnection.execSQL(
> > LoadBalancedMySQLConnection.java:155)
> >         at com.mysql.jdbc.PreparedStatement.executeInternal(
> > PreparedStatement.java:2119)
> >         at com.mysql.jdbc.PreparedStatement.executeQuery(
> > PreparedStatement.java:2283)
> >         at sun.reflect.GeneratedMethodAccessor75.invoke(Unknown Source)
> >         at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > DelegatingMethodAccessorImpl.java:43)
> >         at java.lang.reflect.Method.invoke(Method.java:606)
> >         at com.mysql.jdbc.LoadBalancingConnectionProxy$
> > ConnectionErrorFiringInvocationHandler.invoke(
> > LoadBalancingConnectionProxy.java:103)
> >         at com.mysql.jdbc.FailoverConnectionProxy$
> > FailoverInvocationHandler.invoke(FailoverConnectionProxy.java:51)
> >         at com.sun.proxy.$Proxy257.executeQuery(Unknown Source)
> >         at org.apache.commons.dbcp.DelegatingPreparedStatement.
> > executeQuery(DelegatingPreparedStatement.java:96)
> >         at org.apache.commons.dbcp.DelegatingPreparedStatement.
> > executeQuery(DelegatingPreparedStatement.java:96)
> >         at com.cloud.utils.db.ConnectionConcierge$
> > ConnectionConciergeManager.testValidity(ConnectionConcierge.java:148)
> >         at com.cloud.utils.db.ConnectionConcierge$
> > ConnectionConciergeManager$1.runInContext(ConnectionConcierge.java:203)
> >         at org.apache.cloudstack.managed.context.
> > ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >         at org.apache.cloudstack.managed.context.impl.
> > DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >         at org.apache.cloudstack.managed.context.impl.
> > DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >         at org.apache.cloudstack.managed.context.impl.
> > DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >         at org.apache.cloudstack.managed.context.
> > ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >         at java.util.concurrent.Executors$RunnableAdapter.
> > call(Executors.java:473)
> >         at java.util.concurrent.FutureTask.runAndReset(
> > FutureTask.java:304)
> >         at java.util.concurrent.ScheduledThreadPoolExecutor$
> > ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> >         at java.util.concurrent.ScheduledThreadPoolExecutor$
> > ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> >         at java.util.concurrent.ThreadPoolExecutor.runWorker(
> > ThreadPoolExecutor.java:1152)
> >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > ThreadPoolExecutor.java:622)
> >         at java.lang.Thread.run(Thread.java:748)
> > Caused by: java.net.SocketException: Broken pipe (Write failed)
> >         at java.net.SocketOutputStream.socketWrite0(Native Method)
> >         at java.net.SocketOutputStream.socketWrite(
> > SocketOutputStream.java:115)
> >         at java.net.SocketOutputStream.write(SocketOutputStream.java:
> 161)
> >         ... 36 more
> >
> > On Mon, Dec 18, 2017 at 2:03 PM, Rafael Weingärtner
> > <rafaelweingartner@xxxxxxxxx> wrote:
> > > Here is a fix:
> > > https://www.dropbox.com/s/kgakhs3v05uz88x/cloud-
> > framework-cluster-4.9.3.0.jar?dl=1
> > > You need to replace this jar file in CloudStack installation. You
> should
> > > also backup the original jar and restore it as soon as you finish
> > testing.
> > > To replace the JARs, you need to stop ACS, and just then start it.
> > >
> > > If everything works fine, I will open a PR against master, and with a
> bit
> > > of luck we can push it into 4.11
> > >
> > > On Sat, Dec 16, 2017 at 8:03 AM, Alireza Eskandari <
> > astro.alireza@xxxxxxxxx>
> > > wrote:
> > >
> > >> I'm using CS 4.9.3.0-shapeblue0
> > >>
> > >> On Sat, Dec 16, 2017 at 12:49 PM, Rafael Weingärtner
> > >> <rafaelweingartner@xxxxxxxxx> wrote:
> > >> > Awesome!
> > >> > I found one method that might seem the cause of the problem.
> > >> > What is the version of ACS that you are using?
> > >> >
> > >> > On Sat, Dec 16, 2017 at 4:10 AM, Alireza Eskandari <
> > >> astro.alireza@xxxxxxxxx>
> > >> > wrote:
> > >> >
> > >> >> Hi
> > >> >>
> > >> >> Gabriel,
> > >> >> My configuration is same as your suggestion, but I get the errors.
> > >> >>
> > >> >> Rafael,
> > >> >> You are right. I confirm that CS works normally but I get those
> > >> warnings.
> > >> >> I would make me happy to help you for this fix :)
> > >> >>
> > >> >>
> > >> >> On Tue, Dec 12, 2017 at 3:30 PM, Rafael Weingärtner
> > >> >> <rafaelweingartner@xxxxxxxxx> wrote:
> > >> >> > Alireza,
> > >> >> > This is a warning and should not cause you much trouble. I have
> > been
> > >> >> trying
> > >> >> > to pin point this problem for quite some time now.
> > >> >> > If I generate a fix, would you be willing to test it?
> > >> >> >
> > >> >> > On Tue, Dec 12, 2017 at 8:56 AM, Gabriel Beims Bräscher <
> > >> >> > gabrascher@xxxxxxxxx> wrote:
> > >> >> >
> > >> >> >> Hi Alireza,
> > >> >> >>
> > >> >> >> I have production environments with Master to Master replication
> > and
> > >> >> >> we have no problems. We may need more details of your
> > configuration.
> > >> >> >> Have you configured the slave database? Are you sure that you
> > >> configured
> > >> >> >> correctly the ha heuristic?
> > >> >> >>
> > >> >> >> Considering that you already configured replication and
> "my.cnf",
> > I
> > >> will
> > >> >> >> focus on the CloudSack db.properties file.
> > >> >> >>
> > >> >> >> When configuring Master-Master replication, you should have at
> > >> >> >> /etc/cloudstack/management/db.properties something like:
> > >> >> >> -----------------------------
> > >> >> >> db.cloud.autoReconnectForPools=true
> > >> >> >>
> > >> >> >> #High Availability And Cluster Properties
> > >> >> >> db.ha.enabled=true
> > >> >> >>
> > >> >> >> db.cloud.queriesBeforeRetryMaster=5000
> > >> >> >> db.usage.failOverReadOnly=false
> > >> >> >> db.cloud.slaves=acs-db-02
> > >> >> >>
> > >> >> >> cluster.node.IP=<cluster node IP>
> > >> >> >>
> > >> >> >> db.usage.autoReconnect=true
> > >> >> >>
> > >> >> >> db.cloud.host=acs-db-01
> > >> >> >> db.usage.host=acs-db-01
> > >> >> >>
> > >> >> >> #db.ha.loadBalanceStrategy=com.mysql.jdbc.
> > SequentialBalanceStrategy
> > >> >> >> db.ha.loadBalanceStrategy=com.cloud.utils.db.StaticStrategy
> > >> >> >>
> > >> >> >> db.cloud.failOverReadOnly=false
> > >> >> >> db.usage.slaves=acs-db-02
> > >> >> >> -----------------------------
> > >> >> >>
> > >> >> >> "db.ha.loadBalanceStrategy" is confiugured with the heuristic
> > >> >> >> "com.cloud.utils.db.StaticStrategy"
> > >> >> >>
> > >> >> >> "db.ha.enabled" need to be “true”
> > >> >> >>
> > >> >> >> The primary database is configured with the variable
> > “db.cloud.host”.
> > >> >> The
> > >> >> >> secondary database(s) is(are) configured with the variable
> > >> >> >> “db.usage.slaves”. One variable that is different from both
> Apache
> > >> >> >> CloudStack servers is “cluster.node.IP”, being the ACS mgt IP.
> > >> >> >> Additionally, you will need to create a folder
> > >> >> >> “/usr/share/cloudstack-mysql-ha/lib/” and move the jar file
> > >> >> >> “cloud-plugin-database-mysqlha-4.9.3.0.jar” into the new
> folder.
> > >> >> >>
> > >> >> >> -----------------------------
> > >> >> >> mkdir -p /usr/share/cloudstack-mysql-ha/lib/
> > >> >> >> cp
> > >> >> >> /usr/share/cloudstack-management/webapps/client/WEB-
> > >> >> >> INF/lib/cloud-plugin-database-mysqlha-4.9.3.0.jar
> > >> >> >> /usr/share/cloudstack-mysql-ha/lib/
> > >> >> >> -----------------------------
> > >> >> >>
> > >> >> >> Cheers,
> > >> >> >> Gabriel.
> > >> >> >>
> > >> >> >> 2017-12-12 6:30 GMT-02:00 Alireza Eskandari <
> > astro.alireza@xxxxxxxxx
> > >> >:
> > >> >> >>
> > >> >> >> > I have opened a new jira ticket about this problem:
> > >> >> >> > https://issues.apache.org/jira/browse/CLOUDSTACK-10186
> > >> >> >> >
> > >> >> >>
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> > --
> > >> >> > Rafael Weingärtner
> > >> >>
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > Rafael Weingärtner
> > >>
> > >
> > >
> > >
> > > --
> > > Rafael Weingärtner
> >
>
>
>
> --
> Rafael Weingärtner
>