Kalanand's June 2016 Log

   May 2016   
June 2016
   July 2016   

June 1st

Compute mean and standard deviation after addining a new item to the dataset

Consider the following use case scenario. I am analyzing streaming data and I want to store just the current values of the mean and standard deviation of an observable. So every time I see a new event I would like to update the previously stored values of mean and standard deviation without much overhead. I definitely don't want to store the original set of values.

Here is the formula to do this

   new_mean = mean + δ
   new_stdev2 = (N/(N+1))*[stdev2 + δ2] + (d-δ)2/(N+1) 
where d = new_item - mean,
          N = number of items in the original dataset,
          δ  = d/(N+1)

So, I essentially need to carry along the number of items in the streaming data. Here is a simple implementation.

void updateMeanDtdev(size_t new_count, float& mean, float& stdev, const float new_elem)
   if (new_count==0) 
      mean  = 0.0;
      stdev = 0.0;

   auto c1 = (new_count - 1)/new_count;
   auto delta = (new_elem - mean)/new_count;
   auto first_term = c1*(pow(stdev,2) + pow(delta,2));
   auto second_term = pow(new_elem - mean - delta, 2)/new_count;

   mean += delta;    
   stdev = sqrt(first_term + second_term);

June 14th

VoIP protocols

There is a two-step process for transmitting audio and video packets between communicating devices over the internet. Here is a code to detect each step:

    bool isSIP(const uint8_t protocol, const uint16_t port, 
               const uint8_t* session_first_pkt, 
               const uint8_t* session_second_pkt) const 

      //UDP ports used by SIP or H.323 protocol 
      std::set<uint16_t> m_udp_voip_ports = {5060, 5061, 1719}; 
      //TCP ports used by SIP or H.323 protocol 
      std::set<uint16_t> m_tcp_voip_ports = {5060, 5061, 1720}; 

      //if the first packet has 'SIP' as the first 3 bytes, it's SIP 
      if (session_first_pkt != nullptr && memcmp(session_first_pkt, "SIP", 3) == 0) 
         return true;

      //sometimes it could be the second packet instead 
      if (session_second_pkt != nullptr && memcmp(session_second_pkt, "SIP", 3) == 0) 
         return true;

      //Else check for protocol/port information 
      bool is_tcp_sip = (protocol==IPPROTO_TCP && m_tcp_voip_ports.find(port) != m_tcp_voip_ports.end());
      bool is_udp_sip = (protocol==IPPROTO_UDP && m_udp_voip_ports.find(port) != m_udp_voip_ports.end());

      return  is_tcp_sip || is_udp_sip;

    //Is this RTP traffic? Must have done SIP auth and be using correct ports
    bool isRTP(const bool seen_sip_before, const uint8_t protocol, const uint16_t port) 
      //Check if we have seen SIP/H.323 between this src, dest pair
      if (!seen_sip_before) 
         return false;

      //RTP protocol uses UDP ports in the range 16384 - 32767
      return (protocol==IPPROTO_UDP && port >= m_rtp_port_min && port <= m_rtp_port_max); 

Go to May's log

Last modified: Wed Jun 29 16:02:44 PDT 2016