Kalanand's June 2016 Log

May 2016

July 2016

June 1st

Compute mean and standard deviation after addining a new item to the dataset

Consider the following use case scenario. I am analyzing streaming data and I want to store just the current values of the mean and standard deviation of an observable. So every time I see a new event I would like to update the previously stored values of mean and standard deviation without much overhead. I definitely don't want to store the original set of values.

Here is the formula to do this


   new_mean = mean + δ
   new_stdev² = (N/(N+1))*[stdev² + δ²] + (d-δ)²/(N+1)

where d = new_item - mean,
N = number of items in the original dataset,
δ = d/(N+1)

So, I essentially need to carry along the number of items in the streaming data. Here is a simple implementation.


void updateMeanDtdev(size_t new_count, float& mean, float& stdev, const float new_elem)
{
   if (new_count==0) 
   {
      mean  = 0.0;
      stdev = 0.0;
      return;
   }

   auto c1 = (new_count - 1)/new_count;
   auto delta = (new_elem - mean)/new_count;
   auto first_term = c1*(pow(stdev,2) + pow(delta,2));
   auto second_term = pow(new_elem - mean - delta, 2)/new_count;

   mean += delta;    
   stdev = sqrt(first_term + second_term);
}

June 14th

VoIP protocols

There is a two-step process for transmitting audio and video packets between communicating devices over the internet.

Call set-up/initiation:Before audio or video media can flow between two devices, a protocol has to be employed to find the remote device and to negotiate the means of media transfer. The most popular of such protocols are H.323 and Session Initiation Protocol (SIP). SIP clients typically use TCP or UDP on port numbers 5060 or 5061 to connect to SIP servers and other SIP endpoints. H.323 clients typically use UDP port 1718 or 1719 (Gatekeeper discovery/RAS, must be bidirectional) or TCP port 1720. A detailed comparison between the two protocols can be found here.
Media transfer:Virtually every device in the world uses Real-Time Protocol (RTP) for transmitting audio and video packets. The payload format, packet order, and mechanism to deal with delay and jitter are defined in IETF RFC 3551. RTP clients typically use UDP ports in the range 16384−32767.

Here is a code to detect each step:


    bool isSIP(const uint8_t protocol, const uint16_t port, 
               const uint8_t* session_first_pkt, 
               const uint8_t* session_second_pkt) const 
    {

      //UDP ports used by SIP or H.323 protocol 
      std::set<uint16_t> m_udp_voip_ports = {5060, 5061, 1719}; 
      
      //TCP ports used by SIP or H.323 protocol 
      std::set<uint16_t> m_tcp_voip_ports = {5060, 5061, 1720}; 

      //if the first packet has 'SIP' as the first 3 bytes, it's SIP 
      if (session_first_pkt != nullptr && memcmp(session_first_pkt, "SIP", 3) == 0) 
      {
         return true;
      }

      //sometimes it could be the second packet instead 
      if (session_second_pkt != nullptr && memcmp(session_second_pkt, "SIP", 3) == 0) 
      {
         return true;
      }

      //Else check for protocol/port information 
      bool is_tcp_sip = (protocol==IPPROTO_TCP && m_tcp_voip_ports.find(port) != m_tcp_voip_ports.end());
      bool is_udp_sip = (protocol==IPPROTO_UDP && m_udp_voip_ports.find(port) != m_udp_voip_ports.end());

      return  is_tcp_sip || is_udp_sip;
    }


    //Is this RTP traffic? Must have done SIP auth and be using correct ports
    bool isRTP(const bool seen_sip_before, const uint8_t protocol, const uint16_t port) 
    {
      //Check if we have seen SIP/H.323 between this src, dest pair
      if (!seen_sip_before) 
      {
         return false;
      }

      //RTP protocol uses UDP ports in the range 16384 - 32767
      return (protocol==IPPROTO_UDP && port >= m_rtp_port_min && port <= m_rtp_port_max); 
    }

Go to May's log

Last modified: Wed Jun 29 16:02:44 PDT 2016