git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: upgrade Apache Cassandra 2.1.9 to 3.0.9


Schema won’t be transferred cross-majors


-- 
Jeff Jirsa


On Dec 4, 2018, at 10:51 PM, Shravan R <skr989@xxxxxxxxx> wrote:

Thanks Jeff. I tried to bootstrap a 3.x node to a partially upgraded cluster (2.1.9 + 3.x) and I was not able to do so. The schema never settled. 

How does the below approach sound like?
  1. Update the software binary on all nodes to use cassandra-3.x upon a restart.
  2. Restart all nodes in a rolling fashion
  3. Run nodetool upgradesstables in a rolling fashion

Is there a risk on pending nodetool upgradesstables? 

On Sun, Dec 2, 2018 at 2:12 AM Jeff Jirsa <jjirsa@xxxxxxxxx> wrote:


On Dec 2, 2018, at 12:40 PM, Shravan R <skr989@xxxxxxxxx> wrote:

Marc/Dimitry/Jon - greatly appreciate your feedback. I will look into the version part that you suggested. The reason to go direct to 3.x is to take a bi leap and reduce overall effort to upgrade a large cluster (development included). 

I have these questions from my original post. Appreciate if you could shed some light and point me in the right direction.

1) How do deal with decommissioning a 2.1.9 node in a partially upgraded cluster?

If any of the replicas have already upgraded, which is almost guaranteed if you’re using vnodes, It’s hard / you don’t. You’d basically upgrade everything else and then deal with it. If a host fails mid upgrade you’ll likely have some period of unavailables while you bounce the replicas to finish, then you can decom



2) How to bootstrap a 3.x node to a partially upgraded cluster? 

This may work fine, but test it because I’m not certain. It should be able to read the 2.1 and 3.0 sstables that’ll stream so it’ll just work

3) Is there an alternative approach to the upgrade large clusters. i.e instead of going through nodetool upgradesstables on each node in rolling fashion

Bounce them all as quickly as is practical, do the upgradesstables after the bounces complete





On Sat, Dec 1, 2018 at 1:03 PM Jonathan Haddad <jon@xxxxxxxxxxxxx> wrote:
Dmitry is right. Generally speaking always go with the latest bug fix release. 

On Sat, Dec 1, 2018 at 10:14 AM Dmitry Saprykin <saprykin.dmitry@xxxxxxxxx> wrote:
See more here 

On Sat, Dec 1, 2018 at 1:02 PM Dmitry Saprykin <saprykin.dmitry@xxxxxxxxx> wrote:
Even more, 3.0.9 is a terrible target choice by itself. It has a nasty bug corrupting sstables on alter.

On Sat, Dec 1, 2018 at 11:55 AM Marc Selwan <marc.selwan@xxxxxxxxxxxx> wrote:
Hi Shravan, 

Did you upgrade Apache Cassandra 2.1.9 to the latest patch release before doing the major upgrade? It's generally favorable to go to the latest patch release as often times they include fixes that smooth over the upgrade process. There are hundreds of bug fixes between 2.1.9 and 2.1.20 (current version) 

Best,
Marc

On Fri, Nov 30, 2018 at 3:13 PM Shravan R <skr989@xxxxxxxxx> wrote:
Hello,

I am planning to upgrade Apache Cassandra 2.1.9 to Apache Cassandra-3.0.9. I came up with the version based on [1]. I followed upgrade steps as in [2]. I was testing the same in the lab and encountered issues (streaming just fails and hangs for ever) with bootstrapping a 3.0.9 node on a partially upgraded cluster. [50% of nodes on 2.1.9 and 50% on 3.0.9]. The production cluster that I am supporting is pretty large and I am anticipating to end up in a situation like this (Hope not) and would like to be prepared.

1) How do deal with decommissioning a 2.1.9 node in a partially upgraded cluster?
2) How to bootstrap a 3.x node to a partially upgraded cluster? 
3) Is there an alternative approach to the upgrade large clusters. i.e instead of going through nodetool upgradesstables on each node in rolling fashion


As per [1] the general restriction is to avoid decommissioning or adding nodes but in reality there can be failures or maintenance that warrants us to do so. 


Please point me in the right direction.


Thanks,
Shravan




--
Marc Selwan | DataStax | Product Management | (925) 413-7079 


--
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade