git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Apache Beam Newsletter - December 2018



Beam.png

December 2018 | Newsletter


     What’s been done


KafkaIO (by: Alexey Romanenko, Raghu Angadi)

  • Added writing support with ProducerRecord

  • See BEAM-6063 for more details


HadoopFormatIO (by: Alexey Romanenko, David Moravek, David Hrbacek)

  • Added writing support (batching/streaming) for HadoopFormatIO.

  • See BEAM-5310 for more details


Flink Runner (by: Ankur Goenka, Maximilian Michels, Thomas Weise, Ryan Williams, Robert Bradshaw)

  • Portability: Integration of timers and state in user functions for streaming and batch execution

  • Portability: Combiner lifting (performance optimization)

  • Portability: More easily readable operator names

  • Support for Beam in Flink 1.5 and 1.6

  • Support for scaling up Beam applications on Flink

  • Bug fixes and improved testing


TableMacroUDF (by: Gleb Kanterov)

  • Support for user defined table functions (UDTF).

  • See BEAM-6133 for more details.



     What we’re working on...


Flink Runner (by: Ankur Goenka, Maximilian Michels, Thomas Weise, Ryan Williams, Robert Bradshaw)

  • Portability: Integration of metrics

  • Better support for Flink savepoints

  • Performance tuning of portable pipelines

  • Portability: Integration with Kubernetes

  • Improving the testing infrastructure on Jenkins


Load tests of Core Apache Beam Operations (by: Łukasz Gajowy, Katarzyna Kucharczyk)



     New committers



  • Matthias Baetens, London, England



     Talks & meetups


DevFest 2018 @ Warsaw

  • “Apache Beam - what do I Gain?” with speaker Łukasz Gajowy

  • Link to description here and slides here


Bay Area Apache Beam Meetup @ San Francisco

  • Kicking off the first Bay Area Apache Beam Meetup in SF, December 12th.

  • Speakers are Andrew Pilloud and Kenn Knowles

  • To be recorded and posted here.  


Open Source Summit @ Paris

  • “Apache Beam: portable and evolutive data-intensive applications”, December 5th.

  • Speaker: Alexey Romanenko

  • Slides: link


DevFest 2018 @ London / Bucharest

  • “Large scale stream analytics with Apache Beam” with speaker Matthias Baetens

  • Slides: here


Until Next Time!

This edition was curated by our community of contributors, committers and PMCs. It contains work done in December 2018 and ongoing efforts. We hope to provide visibility to what's going on in the community, so if you have questions, feel free to ask in this thread.
--
Rose Thị Nguyễn