<div dir="ltr">Thanks a lot! </div> <div class="gmail_quote"><div dir="ltr">On Thu, Nov 10, 2016 at 11:41 AM Sylvain Afchain <<a href="mailto:safchain@redhat.com">safchain@redhat.com</a>> wrote: </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi, Yes Github issue is the right place to discuss implementation. I confirmed that we agreed adding counter on interfaces metadata but need to elaborate a bit :) For the Gremlin sutff : The "Skydive" Gremlin language is implemented in two packages : topology/graph/traversal and flow/traversal. flow/traversal register extra steps. For the step sum it should be implemented like the other flow extensions. Please take a look at the NewFlowTraversalExtension, ScanIdent, ParseStep in the flow/traversal/traversal.go. You can have a look at the step `Hops`, `Nodes` as examples. Each step has to implement the interface GremlinTraversalStep defined in topology/graph/traversal/traversal_parser.go About the aggregation on the WebUI side, with not but I would prefer to have a gremlin implementation too in order to have the feature exposed by the API. About the offloading, we could take the same approach that we took for the sort step. For the sort step there are currently two possible call path : - Time context (step context), request to the datastore, so the datastore does the job not Skydive - No Time context(live flows), sort done by agent then merge sort by the analyzer, so a part of the process is distributed. Regards, Sylvain ----- Original Message ----- > Hi kostya. > > I'm still interested in this topic. Let's discuss implementations details > on the github issue, But AFAIR, we agreed with safchain that we could rely > on third party go lib that polls counter interfaces : > > <a href="https://github.com/shirou/gopsutil/tree/master/net" rel="noreferrer" class="gmail_msg" target="_blank">https://github.com/shirou/gopsutil/tree/master/net</a> > > And add interesting data to node's metadata. > > Mat > > On Wed, Nov 9, 2016 at 5:18 PM, Kostya <<a href="mailto:konstantin.dorfman@gmail.com" class="gmail_msg" target="_blank">konstantin.dorfman@gmail.com</a>> wrote: > > > Hello Mathieu, > > Do you want to discuss design/implementation ideas on > > <a href="https://github.com/skydive-project/skydive/issues/109" rel="noreferrer" class="gmail_msg" target="_blank">https://github.com/skydive-project/skydive/issues/109</a> ? This will be very > > helpful for me. > > > > We need some general API to get/calculate bandwidth of the edge, it could > > be implemented as sum of every flow's ABbytes for specific node (if > > capture is active on it), otherwise it could be interface stats as we can > > get from OS (ifconfig stats). > > > > Syvain, can you guide me a little about Gremlin module? > > > > >We are going to add a `sum` step to the gremlin language for that and we > > could display the result in the >WebUI when pointing edges > > > > so, Gremlin request going through "api/topology" to Analyzer; it does > > general logic of Metric aggregation/sum: returns back synthetic data > > `flow` containing the result? > > > > Why it is better than to get all flows stored currently in the Vertex and > > compute aggregation on webUI side? It might be better to offload > > computation from Analyzer (better for scaling). > > > > Thanks, Kostya > > > > > > > > > > > > On Wed, Nov 2, 2016 at 3:35 PM Sylvain Afchain <<a href="mailto:safchain@redhat.com" class="gmail_msg" target="_blank">safchain@redhat.com</a>> > > wrote: > > > >> Hello, > >> > >> So there is already a way to retrieve metrics thanks to the WebUI, not on > >> edge but on node where the captures occur. There is an on going work to be > >> able to aggregate metrics from multiple capture so that we could get the > >> bandwidth on a link. We are going to add a `sum` step to the gremlin > >> language for that and we could display the result in the WebUI when > >> pointing edges ? > >> > >> But here I'm talking about flow capture not interface metrics. There is a > >> opened issue[1] to bring interface metrics to the graph and I know that > >> Mathieu wanted to work on this. What we could do is to store interface > >> metrics as node metadata ? > >> > >> About the WebUI and updates : > >> > >> The webui is listening graph events thanks to a websocket connection. > >> Each time there is a modification of the graph on the analyzer side, an > >> event for the change is sent to all the websocket clients so to the webui. > >> And as you said the event message are consumed by ProcessGraphMessage. > >> > >> Sylvain > >> > >> [1] <a href="https://github.com/skydive-project/skydive/issues/109" rel="noreferrer" class="gmail_msg" target="_blank">https://github.com/skydive-project/skydive/issues/109</a> > >> > >> ----- Original Message ----- > >> > Hello, > >> > I'm thinking about adding some metadata on webUI edges along active > >> capture > >> > - bandwidth, so one can understand traffic on the given topology. > >> > As I see this task can be divided into the following: > >> > 1. Add bandwidth metadata to the Edge > >> > 2. When webUI gets "NewEdge" оr "NewUpdate" message in > >> > ProcessGraphMessage() function, need to draw this bandwidth info on the > >> > edge of the graph representation. > >> > 3. Adding new or modifying existing probe in order it will be able to > >> > generate/compute bandwidth data and supply it to the Agent graph. > >> > > >> > So, what do you think about the idea? Where should I start with adding > >> such > >> > info? > >> > > >> > I suppose we can store the bandwidth info as metadata in these: > >> > 1) type graphElement struct { > >> > ID Identifier > >> > metadata Metadata > >> > host string > >> > } > >> > 3) type Edge struct { > >> > graphElement > >> > parent Identifier > >> > child Identifier > >> > } > >> > 3) type MemoryBackendEdge struct { > >> > *Edge > >> > } > >> > > >> > Another, more general question: can you please describe in two words, > >> what > >> > is data flow from in memory graph of Analyzer to webUI: > >> > - how update of webUI started > >> > - when it is updated (on changes, periodically) > >> > > >> > And what flows should I look into to understand and to start playing > >> with > >> > adding additional metadata described above? > >> > > >> > Thanks, > >> > Kostya > >> > > >> > > > </blockquote></div>