[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[osprofiler] Distributed tracing in OpenStack


> WiadomoÅ?Ä? napisana przez Monty Taylor <mordred at> w dniu 12.04.2019, o godz. 16:34:
> On 4/11/19 9:42 PM, Ilya Shakhat wrote:
>> Hi,
>> Distributed tracing is one of must-have features when one wants to track the full path of request going through different services and APIs. This makes it similar to shared request-id, but with nice visualization at the end [1]. In OpenStack the tracing can be achieved via osprofiler library. The library was introduced 5 years ago, and back then there was no standard approach on how to do tracing and that's why it stays aside from what has become a mainstream. Yet there is no single standard, but the major players are OpenTracing and OpenCensus communities. OpenTracing is represented by Uber's Jaeger which is the default tracer from k8s world.
>> Issues and limitations to be fixed:
>> 1. Compatibility. While osprofiler library supports many different storage drivers, it has only one way of transferring trace context over the wire. Ideally the library should be compatible with other third-party tracers and allow traces to start in front of OpenStack APIs (e.g. in user apps) and continue after (e.g. in storage systems, or network management tools). [2]
>> 2. Operation mode. With osprofiler tracing is initiated by user request, while in industrial solutions the tracing can be managed centrally via dynamic sampling policies.
>> 3. In-process trace propagation. Depending on execution model (threaded, async) the ways of storing current trace context differ. OSProfiler supports thread-local model, which recently got broken with new async implementation in openstacksdk [3].
> FWIW - we should have re-fixed that issue in SDK for all instances other than parallel uploading of Large Objects segments to swift. The parallism support now relies on the calling context's parallism. The large-object segment uploader is a thing we should make sure we do things with to make sure we're not losing those interactions.
> That said - if we move forward with this plan - let's be sure to make sure it works in openstacksdk - and that we're testing it so that we don't break it.

There is even storyboard for that issue!/story/2004618

>> With OpenTracing it is possible to select the appropriate model alongside with tracer configuration.
>> What's the plan:
>> Switching to OpenTracing could be a good option to gain compatibility with 3rd-party solutions. The actual change should go to osprofiler library, but indirectly affects all OpenStack projects (should it be a global team goal then?). I'm going to make a PoC of proposed change, so reviews would be highly appreciated.
>> Comments, suggestions?
> Generally supportive. I have specific impl feedbacks - but I'll leave those on the patches.
>> Thanks,
>> Ilya
>> [1] e.g.
>> [2]
>> [3]

Slawek Kaplonski
Senior software engineer
Red Hat