Re: Housing longer-term Arrow development, design, and roadmap documents
I have a bias against wikis of all kinds. If left to their own devices, they tend to become an unstructured mess. Of course, the lack of structure is what makes them useful for what Wes is proposing: gathering knowledge and organizing it as it evolves. But someone will need to play the “librarian” role to keep it in shape.
I would advocate keeping bug/feature reports in JIRA, and user doc where it is now, and only use wiki for the small amount that doesn’t fit into that.
> On Jun 26, 2018, at 8:27 AM, Wes McKinney <wesmckinn@xxxxxxxxx> wrote:
> GitHub wiki pages lack collaboration features like commenting. It will
> be interesting to see what we can work up with JIRA integration, e.g.
> burndown charts for release management.
> I asked INFRA to create a Confluence space for us so we can give it a
> try to see if it works for us. Confluence seems to have gotten a lot
> nicer since I last used it:
> If any PMC members would like to be administrators of the space,
> please let me know your Confluence username. You have to create a
> separate account (it does not appear to be linked to JIRA accounts)
> On Sun, Jun 24, 2018 at 1:14 PM, Uwe L. Korn <uwelk@xxxxxxxxxx> wrote:
>> I would prefer Confluence over GitHub pages because I would hope that one can integrate the ASF JIRA via widgets into the wiki pages. The vast amount of issues should all be categorizable into some topic. Once these are triaged, they should pop up in the respective wiki pages that could form a roadmap. That way, newcomers should get a better start to find the things to work on for a certain topic.
>> On Sun, Jun 24, 2018, at 7:02 PM, Antoine Pitrou wrote:
>>> Hi Wes,
>>> I wonder if GitHub wiki pages would be an easier-to-approach alternative?
>>> Le 24/06/2018 à 08:42, Wes McKinney a écrit :
>>>> hi folks,
>>>> Since the scope of Apache Arrow has grown significantly in the last
>>>> 2.5 years to encompass many programming languages and new areas of
>>>> functionality, I'd like to discuss how we could better accommodate
>>>> longer-term asynchronous discussions and stay organized about the
>>>> development roadmap.
>>>> At any given time, there could be 10 or more initiatives ongoing, and
>>>> the number of concurrent initiatives is likely to continue increasing
>>>> over time as the community grows larger. Just off the top of my head
>>>> here's some stuff that's ongoing / up in the air:
>>>> * Remaining columnar format design questions (interval types, unions, etc.)
>>>> * Arrow RPC client/server design (aka "Arrow Flight")
>>>> * Packaging / deployment / release management
>>>> * Rust language build out
>>>> * Go language build out
>>>> * Code generation / LLVM (Gandiva)
>>>> * ML/AI framework integration (e.g. with TensorFlow, PyTorch)
>>>> * Plasma roadmap
>>>> * Record data types (thread I just opened)
>>>> With ~500 open issues on JIRA, I have found that newcomers feel a bit
>>>> overwhelmed when they're trying to find a part of the project to get
>>>> involved with. Eventually one must sink one's teeth into the JIRA
>>>> backlog, but I think it would be helpful to have some centralized
>>>> project organization and roadmap documents to help navigate all of the
>>>> efforts going on in the project.
>>>> I don't think documents in the repository are a great solution for
>>>> this, as they don't facilitate discussions very easily --
>>>> documentation or Markdown documents (like the columnar format
>>>> specification) are good to write there when some decisions have been
>>>> made. Google Documents are great, but they are somewhat ephemeral.
>>>> I would suggest using the ASF's Confluence wiki for these purposes.
>>>> The Confluence UI is a bit clunky like other Atlassian products, but
>>>> the wiki-style model (central landing page + links to subprojects) and
>>>> collaboration features (comments and discussions on pages) would give
>>>> us what we need. I suspect that it integrates with JIRA also, which
>>>> would help with cross-references to particular concrete JIRA items
>>>> related to subprojects. Here's an example of a Confluence landing page
>>>> for another ASF project:
>>>> What do others think?