Tag: stackoverflow

SO – “What is all the fuss about TSN (Time Sensitive Networking)?

And now for a “way too broad” question.

This was originally entitled “Time Sensitive Networking” by Praveen here, and phrased as this:

I am aware of current Ethernet Technology and it’s non-deterministic behavior, due to CSMA-/CA/CD. I also see a lot of news around Time Sensitive Networking.

Could anyone briefly explain how TSN could change or enhance timing, synchronization, how is it related to IEEE 1588(PTP) etc ?

Since indeed this is “way too broad”, I’d rephrase it as “what is all the fuss about TSN (Time Sensitive Networking)?”

First, to put that aside:

All Ethernet link these days are full duplex, so collision avoidance such as CSMA-/CA/CD are a thing of the past since about 15 years I would say: Ethernet is not a shared media (like is air for Wi-Fi), and there simply isn’t any collision, so no need to avoid it. Did you get that “non-deterministic because of CSMA-/CA/CD” from a book or teacher? Then those would need a serious update.

(And so no, the collision risk and its avoidance mechanism is not the cause of “current Ethernet Technology and it’s non-deterministic behavior”, especially not with the word “current”.)

And this being done, here is my – way too broad and light – answer introduction pointers to this AVB/TSN topic:


About TSN (Time Sensitive Networking):

TSN is just the new name for an IEEE 802 task sub-group that was first called AVB for “Audio Video Bridging”, as you can see from here (2005 or 2008 to 2012):

http://www.ieee802.org/1/pages/avbridges.html

Note: The Audio/Video Bridging Task Group was renamed the
“Time-Sensitive Networking Task Group” in November, 2012. Please refer
to that page for further information. The rest of this page has been
preserved it existed at that time, without some obsolete meeting
information.

The change was made to reflect its now broader perspective: not only for Pro Audio/Video distribution use, but also to automotive, and latter industrial applications as well. So the work continues here:

http://www.ieee802.org/1/pages/tsn.html

(yes, I know, the look an feel of IEEE web pages really  really sucks, but that is good content though)

As a result, you’ll actually find most information about the fundamentals of TSN by googling “Ethernet AVB” rather than “Ethernet TSN”. The wikipedia page, carefully maintained by people directly involved with the technology, is a good start:

https://en.wikipedia.org/wiki/Audio_Video_Bridging

Also, as with every technology, there is a technical side, that’s the IEEE AVB->TSN group, and there is a marketing side, taking care of branding, use-cases, and (very important) certification programs to label and guarantee the interoperability of products, to have a healthy ecosystem. For AVB/TSN, this marketing side is handled by the AVnu Alliance SIG (Special Interest Group), founded in 2009:

http://avnu.org/

There too, you can find a lot of information in the knowledge base section of the web site (technologies, whitepapers, specifications, FAQs): why was it made (what is the problem to solve), how does it work, what are the use cases in the various fields it targets.

Some final words:

AVB/TSN it is not a single protocol, but rather a set of protocols, put together with possible variants according to the use case. Example: originally designed with auto-configuration/plug and play’ built in (geared towards sound engineers, without the need for network engineers), its automotive profile rather use static configuration (because of reduced boot and configuration time, lesser code/hardware for reduced cost embedded devices, and you’re not going to change a car networking topology nor roles of the nodes everyday anyway).

And that makes a BIG pile of standards: the base IEEE AVB standards put together, the last one published in 2013 IIRC, where already about 1,500 pages, and TSN is now expanding that. Distribution of a common wall-clock to participants, which is a prior need to synchronization, with sub-microsecond range of precision, is a big, complex problem in itself to start with . Be it with a static clock device reference (“PTP master”) in IEEE 1588, and even more with a elected, and then constantly monitored and possibly re-elected GM (“Grand Master) via a BMCA (Best master clock algorithm) as in IEEE 802.1AS.

Also, all this require implementation not only in the end nodes, but also in the switches (bridges), which takes an active part in it at pretty much every level (clocking, but also bandwidth reservation, then admission and traffic shaping). So part of the standards are relevant to nodes, others to the bridges.

So all this is really quite big and complex, and going from 1,500 pages (which I once read from cover to cover) – and it’s now more than that – to “briefly explain how TSN could change or enhance timing, synchronization, how is it related to IEEE 1588 (PTP) etc ?” is a bit challenging… especially the “etc” part… :-)

Hope these few pointers help though.

SO – how about multicast/broadcast Audio/Video streaming over 802.11 aka Wi-Fi?

Ahhhhh… now that’s a really, really good question, originally titled as “Wi-Fi Monitor mode listening to traffic”, from pratiklodha, here.

Actually, it may have been titled as this: “how about multicast/broadcast Audio/Video streaming over 802.11 aka Wi-Fi?”, and so will I.

And now that’s an interesting topic by itself. A real, industry-wide question about the state of technology.

And here was my answer, as 802.11 state of the Art in March of 2016: – warning: as presumption as it may sound, this is a bit more than educated guess:


Are you talking about this Wifibroadcast , here?

If so: well yes, monitor mode is the underlying technology, as can be seen here.

Now, if this is about doing a commercial product, sadly, you cannot expect any kind of interoperability from this.

Streaming audio/video over Wi-Fi is a business, and the the power in charge (Wi-Fi Alliance aka WFA) as some view on it, including certification programs. Have a look at Miracast, using Wi-Fi Direct.

As for multicast / broadcast, it is even more of a business and the realm of proprietary technologies for now (example here – and no, this is not limited to automobile). This is quite complicated, to start with because of the synchronization problem across receivers: you don’t want 2 radio receivers in the same room to play with a 1 seconds delay, this would be cacophony.

EDIT:

Meaning, be it with the Wifibroadcast OSS project or with the proprietary industry about it, since there is not yet an open protocol for this (as “publicly available standard specification”, I don’t even go about implementation, FLOSS or not), you will have to provide a specific application for every receiver to match your broadcaster protocol, and vice versa. And that is the state of the industry today. That is what the company I mentioned above, or this other one more well know, or these are doing. And so, they do not interoperate. This will be your problem: provide a receiver app for Windows, Mac OS, Android and iOS (where you may not even have access to sub-layer 3 API) that will match your radio broadcaster protocol. And Linux too, please.

Though, this is the direction of history because this is what the user wants: stream A/V to/from device/application X from brand A to device/application Y from brand B.

And so people have been working on this, on layer 2, because layer 3 and above have unsolvable challenges with it, at IEEE since 2004 with Ethernet AVB, which is a set of protocols. You can download some of its standards for free, others for a moderate fee depending on how old they are. There is a SIG taking care of certification(http://avnu.org/certified-products/) to guarantee interoperability.

It is for 802.3 (aka wired Ethernet), but there is some work done to bring this to 802.11 Wi-Fi. Because again, that’s what the user wants, the market is here, no question about that. It will take a long time. Even more to get consumer electronic grade devices or applications of the shelves. But they will interoperate out of the box, that’s the goal.

There’s even been work done on moving this to layer 3/IP as well BTW, with some performance sacrifice.

So come back in a few years, and all should be setup. Or, if you have lots of time and money and no urge to deliver, implement a solution based on these standards?

PS:

Link to AVnu (Ethernet AVB SIG) page about use cases for consumer electronics audio streaming, wired or wireless:

http://avnu.org/consumer/

…and its 10 pages white paper at the bottom of the page.

SO – how safe is it to parse the output of a CLI command?

The original question was actually “Detect wireless devices connected to Raspberry Pi in python on Linux”, from fragon, here.

To its fullest extent, the question was this:

“In my python code I need to get the list of “physical” WiFi network devices connected to Raspberry Pi

I’ve been doing this by calling:

raw_output = check_output(‘iw dev’, shell=True)

and then extracting all the data I need from raw_output

It works ok, but in iw help it says that Do NOT screenscrape this tool, we don’t consider its output stable. Is it really unsafe to get this data the way I did it? If yes, what is the correct way to do this?”

That is a classic broader question that one probably asked to itself at one time or another as this: “how safe/OK is this to parse the output of a CLI command?”

And my answer have been this (see here for full answer with pro/cons of alternate ways):

What is meant by “Do NOT screenscrape this tool, we don’t consider its output stable” is that as new releases of iw will be made, the output formating may change. So the developers of iw warn you that if you write software depending on the parsing of its output, it may break on future releases of iw.

Take the example of the venerable ifconfig command. For many many years, its output used to be formated like so:

 eth0 Link encap:Ethernet HWaddr 00:80:C8:F8:4A:51
 inet addr:192.168.99.35 Bcast:192.168.99.255 Mask:255.255.255.0
 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
 RX packets:190312 errors:0 dropped:0 overruns:0 frame:0
 TX packets:86955 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:100
 RX bytes:30701229 (29.2 Mb) TX bytes:7878951 (7.5 Mb)
 Interrupt:9 Base address:0x5000

And though it was considered stable (even deprecated and unmaintained by some), it changed a couple of years ago and now looks like this:

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.1.67 netmask 255.255.255.0 broadcast 192.168.1.255
inet6 fe80::8e89:a5ff:fe57:103c prefixlen 64 scopeid 0x20<link>
ether 8c:89:a5:57:10:3c txqueuelen 1000 (Ethernet)
RX packets 2219946 bytes 3178868967 (2.9 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1241676 bytes 102998523 (98.2 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

…so let’s say I did some soft which look at the MAC address by searching the string following “HWaddr”. Nowadays it would be broken, because it should look for the string following “ether” instead.

But as long as you don’t update iw, or perform regular testing of you work, you should not encounter any problem.

It is anyway always inherently a bit fragile to parse the output of a third part tool, you just have to be aware of it. For instance, the output may depend on the LOCALE setup by the user. Real life example, some scripting I did with the output of ifconfig failed on some users environment. Root cause: here is what the output look like in French locale:

eth0 Lien encap:Ethernet HWaddr 00:FF:F2:58:32:A1
UP BROADCAST MULTICAST MTU:1500 Metric:1
Packets reçus:0 erreurs:0 :0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 lg file transmission:1000
Octets reçus:0 (0.0 b) Octets transmis:0 (0.0 b)
Interruption:23 Adresse de base:0x2000

Notice the French “Packets reçus”, “erreurs”, and “Octets reçus” instead of “RX packets”, “errors”, and “RX bytes”.

EDIT:

So:

> Is it really unsafe to get this data the way I did it?

Not really. You just have to keep in mind that your software depends on the output strings of some third part software that is somewhat out of your control and may change in the future. That will be regular testing and maintenance job for you, nothing tragic, that’s software life.

> If yes, what is the correct way to do this?

Again, “no”, but if you want to be bulletproof to that: do not depend on the textual output of third part software. This usually involve writing your own code to replace these tools, which can be quite a task. And if to do so, you use some third part libraries, well, library API change over time too… :-)

Now, I added some addendum to this, related to the OP context, and the cost vs benefits of alternate ways – again, see my original post about this. But you get the substance of my answer here.

SO – Why do 802.11 Acknowledgement Frames have no source MAC?

Ah… now that’s a legitimate good question that deserves a bit of background.

To its to its fullest extent, the question was this:

“Anyone know why 802.11 Acknowledgement Frames have no source MAC address? I don’t see it when I capture the packets from TCPDUMP or Wireshark from Linux with a monitor-mode and promiscuous-mode driver. How does the Access Point distinguish ACK frames from different 802.11 clients if there is no source MAC addresses in the frame?

I can see from all the captures that the ACK comes immediately after the frame is sent (around 10 to 30 microseconds) but that alone can’t be enough to distinguish the source can it? Maybe each frame has some kind of unique identifier and the ACK frame has this ID inside it? Maybe there is identifying information in the encrypted payload since the WLAN uses WPA-PSK mode?”

From George Ou, here.

And my answer is this:

ACK frames, like all 802.11 management frames, have a DA (Destination Address) and SA (Source Address) in their MAC header (both not to be exactly confused with just “MAC address, see below), and that’s all what’s needed in this context.

TLD;DR: In 802.11 context, “MAC address”, SA (Source addr), TA (Transmitter addr), DA (Destination addr), BSSID or whatnot all look alike a 6 byte “MAC address” that we’re familiar with from other technologies, yet they should not be confused.

And now for the demolition of “MAC address” concept in 802.11 context.

802.11 Acknowledgement Frames are parts of 802.11 managements frames, and 802.11 is “a set of media access control (MAC) and physical layer (PHY) specifications” (source).

What this mean – and this is a very important concept to grasp when working with Wi-Fi – is that 802.11 in itself, including its management frames, have nothing to do with “traditional” (say 802.3, aka Ethernet) PHY (layer 1) nor MAC (layer 2) layers – they are a kind of their own.

802.3/Ethernet, to continue with this analogy – or rather counter example – have no such thing as ACK frames, beacons, probe request, RTS/CTS, auth/deauth, association etc, which are all types of 802.11 management frames. These are simply not needed with 802.3, for the most part because wired Ethernet is not a shared media (that’s IEEE terminology), that may lead to unreliability/collision as is air with 802.11/Wi-Fi.

The important consequence of this is that you should not expect a priori to meet other, more familiar concept nor data from other layer 1/2 technologies. Forget this, once for all.

Sure, Wi-Fi looks like it carry on some MAC and IP and TCP and or UDP or whatnot, and they do most of the time, but regarding management frames such as ACK, that’s a different world – its own world. As it is, 802.11 could perfectly be used – and maybe is used in some niche use cases – to carry other higher level protocols than TCP/IP. And its MAC concept, though looking familiar with its 6 bytes, should not be confused in its form nor use to the MAC of 802.3/Ethernet. To take another example, 802.15 aka Bluetooth also have a 6 bytes MAC, yet that is again another thing.

And to take the 802.11 a contrario example, 802.11 layer 1/2 beacon frames for instance carry some info about SSID, supported rates, Frequency-hopping (FH), parameter set etc etc that have no counterparts in other L1/2 technologies.

Now, embrace the complexity of what is/are “MAC addresses” in 802.11…

And this is why, to take an example in day to day use, pcap/tcpdump have such weird filters such as wlan ra, wlan ta, wlan addr1, wlan addr2, wlan addr3, wlan addr4 – and the like for wireshark capture and display filter.

SO articles like

A few month ago, in a click out of mixed “I’m bored today” and genuine curiosity, I registered to Stack Overflow.

Of course, being a developer, I stumble on problems / questions / puzzles multiples times a day.

That’s what make makes the job interesting after all: solve puzzles, build Lego. And that’s probably one of the reasons (though not the only one in retrospect) I/you ended up in this career, isn’t it? And of course, when googling about such or such problem, I couldn’t help but notice how a ridiculous amount of time I’m driven to the same web site – with, OMG, what are actually relevant, good answers.

Or sometimes I’m asked questions from my colleagues/users, just like any of ours deploying code to production use. And then, be it verbally or written, you have to articulate a clear answer. In such cases, I like to provide an answer that is not just an immediate “dot this”, but rather provide some background info about what the problem fundamentally is, and the “how” and “why” – and yes, that can be quite verbose, and end up with a “…so, as a conclusion, do this:…”

And so, some day during September of 2015, I registered to SO.

There would be some things to write about this registration alone (to this date 8 months long), and the active participation experience following it. A lot have been written already about this, probably by some smarter people than me. That would be the topic of another post. A post about the effectiveness of SO, the related intense gamification of the system, and a question that arise to me more and more as time passes: “Why do have not post a single question – only answers?”

Anyway…

As time passes, I noticed I have a clear tendency to write lengthy answers to questions that may have been simply answered by “this is not do-able”/”this is not the way to do it”. Because, precisely, I like to put some “why” and background info about the context. Some of my answers take me anytime between 20 minutes up to 2 hours to write because of this, sometimes half of this looking for precise pointers to provide, and in any case worth the time to write this to an Emacs buffer and saving it. And I’m happy with that, naively hopping that may be some help to the OP or someone else with the same question.

And so, now that I started this blog, I came to realize that some of my answers may very well be turned into “a blog post”, exploring such or such details of whatever technology. Being or not the accepted or most up-voted answer is not related to that – I stand my point for posting some content with not only an answer, but also some wider perspective on this.

Conclusion:

And this is why, starting of today, I’ll post these answers here as well. Because these “Ah! Now that’s an interesting question that deserve not just an answer but also a bit wider perspective!” led to what actually an article, and the best I consider I could write (without going to a doctoral thesis). As shameful self advertising as this may sound. As a matter of fact, I’ve just written one of these…