git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Daffodil tuning


Weber,


Is this for PCAP data? or some other schema? We would need to look in detail at the schema unless it's one we already know.


I doubt there are any magic tuning knobs that will massively speed things up.


Daffodil is largely compute bound, and while we've improved the performance a bit, there is much more work to do here, especially on unparsing, which for formats like PCAP where values must be calculated to be stored into headers, those mechanisms which suspend unparsing, double buffer data, until the needed length value can be computed, and then resume unparsing once it is known,... those mechansims  have some real overhead.  Expressions are being evaluated, suspended, retried, etc.


This page on the Apache Daffodil Wiki


https://cwiki.apache.org/confluence/display/DAFFODIL/Unparsing%3A+Data+Output+Stream+Buffering+and+dfdl%3AoutputValueCalc


may give you some idea of the complexity involved in unparsing.


Are your tests both parsing and unparsing the data? I suspect unparsing is the bottleneck, but it would be good to tease those apart if possible.


--Mike Beckerle

Tresys

________________________________
From: Perpignand, Weber <wperpignand@xxxxxxxxxxxxxxxxxxx>
Sent: Tuesday, May 1, 2018 4:43:44 PM
To: dev@xxxxxxxxxxxxxxxxxxx
Cc: Clarke, Frederick
Subject: Daffodil tuning


Hello,

I have been running some performance tests on incubator-daffodil-master. The best throughput that I have been able to get is 100 Mb/sec on a full frame (1518 bytes) packet. I changed the java -Xmx1024m to -Xmx2048m and the cache size from 128 to 256, but this did not give better result.



Are there other tuning parameters you can recommend?

I do see the tunable options in daffodil (DaffodilTunables.scala), but these also had no noticeable effect.



Thanks.



Weber Perpignand

Senior Software Engineer



+1 (203) 894-9342 x189

wperpignand@xxxxxxxxxxxxxxxxxxx<mailto:wperpignand@xxxxxxxxxxxxxxxxxxx>

www.OwlCyberDefense.com<http://www.owlcyberdefense.com/>

[Email Signature Logo]



See us at the following 2018 events:

Rome Labs/Griffiss Park, Rome, NY (May 10th)<https://www.owlcyberdefense.com/events>

DOT Cybersecurity Summit, Washington, D.C. (May 23rd – May 24th)<https://www.owlcyberdefense.com/events>

Defense Industry Forum, Warsaw, Poland - Speaking! (May 23rd)<https://www.owlcyberdefense.com/events>