Sunday, January 29, 2017
I've been working on a project recently to synchronize multiple raspberry pi's playing videos. The goal was to be within 50ms (about 1 frame) of each other. I'd started testing with a Pi-3 and a Pi-1B but quickly realized the Pi-1B was going to be painful to diagnose issues with the synchronization vs just being slow. The Pi-1B took a LOT longer to start playing a video and had more latency in seeking/changing speed. So I switched to testing with two Pi-3's and to my surprise, it was worse. I banged my head for weeks trying to figure out why different configurations behaved radically different. Data I collected with a packet sniffer was contradictory. In some configurations, specifically any configuration where a Pi-3 was the slave in the synchronization scheme, the latency from the master Pi had a jitter of 300ms. I would receive 10-20 synchronization packets at once then there would be silence for 300ms. With the Pi-3's I had been using wireless because for convenience that would be the easiest way to use them in production. I was guessing going to a wired connection would correct the issue but it bugged me that a wireless link with <2ms of latency with the ping command regardless of packet size would have such a wild jitter with UDP packets. It took a long time for the significance of the Pi-1B being wired and the jitter only appearing when the Pi-3 was a slave. It didn't help that when I used Wireshark on my desktop the packets came spaced give or take 2ms, in contrast my laptop would show the jitter. I was doing each test on a different day and then last night it finally hit me, Multicast packets from a device to my rt-ac66u over wireless or wired have little latency when viewed on a wired connection like my desktop. In contrast, those packets were being buffered and delivered to all wireless devices only 3-4 times per second. ICMP packets were being shipped same-day, but UDP multicast/broadcast were being snail-mailed in batches. GRR. I never did find a setting on the router to alter this behavior and have simply conceded to using wired connections for the Pi's. If anyone has thoughts on this phenomenon, I would be glad to hear them in the comments.