Wireless firing systems & signal interference problems

Discussion in 'Firing Systems And Fusing Fireworks' started by RocketRev, Sep 28, 2021.

  1. I guess, as always, you get what you pay for.

    Buy a cheap unknown brand system and you shouldn't expect much. Buy a more expensive known name mid-priced system and you'll be OK most of the while. Buy the best money can buy and you'll be investing in certainty.
  2. Dodgey

    Dodgey Pro Firer/Crew

    With Galaxis you are investing in the certainty you won't have much money for quite some time.
  3. I'm not a legal expert, but I'm not sure it would actually require additional legislation?

    Under OFCOM's FAQ entry for "What are the legitimate uses of PMSE frequencies?", one of the types of permitted activity is a "Public or private event, at any location, whether or not the event is to be broadcast." (so that could encompass a fireworks display), and for the intended purposes "As required for the holding, managing, running and termination of the event at the event location" (so that could apply to the "running" of the actual display via a firing system). In essence, PMSE frequencies aren't just for microphones, they can be used for "communication" channels (and it doesn't stipulate that these have to be "audio" channels, they could potentially also be "data" channels)

    ...so may be potentially legitimate to use wireless firing systems within the "scope" of PMSE frequencies.
  4. True but haven’t had any issues with what I’ve been using it for.
    PYRO Control
    Sensor DATA

    it managed the Mach 1.2 going up as well
    pyrobrit likes this.
  5. pyrobrit

    pyrobrit Pro Firer/Crew

    I’ve been thinking about channels this past week and I’m glad that someone has mentioned them. Unfortunately the use of channels can cause more confusion in the 2.4ghz frequency band allocated to wifi as much as the name Wifi does. Before we go further I just want to convert the gigahertz to megahertz to make the frequency in use clearer to read.

    The allocated Wifi band is 2400mhz to 2499mhz. So 99mhz that is split into 14 channels for what we know as wifi for phones and other IT equipment. It’s further complicated that only in Japan can you use all 14 channels. USA get to use 1 to 11 and the rest of the world is 1 to 13.

    Now the rest of the equipment in the field is going to use the same frequency range 2400mhz to 2499mhz but not the wifi channels IT people know about.
    Take for example Firetek which uses Zigbee radio modules. Zigbee splits the allocated frequency range into 16 channels with channel 1 starting at 2405mhz.
    Cobra (last time I looked) splits the band into 17 channels with channel 1 at 2403mhz.
    Bluetooth splits the same frequency range into 79 channels albeit tiny ones and very low power.
    Spectrum make popular radio control handsets for model aircraft, drones and helicopters. They split it into 80 channels and the handset will grab two channels. One for transmit and one for receive.

    The point is none of the channels line up on frequencies and this means more noise in the band when all this equipment goes searching for a clear channel within their constraints. A computer wifi channel is so wide that it blanks out fully one third of the allocated frequency range so theoretically you can fit only 3 wifi channels in with no interference into their neighbours. Of course that’s impossible in the above scenario when each radio module manufacturer has different channel configurations.

    You will hear the word “spread spectrum” and “frequency hopping” banded about and in a firing system this is useful to enable the hardware to maintain a link when a channel has too much noise. The modules will frequency hop to a different channel that is quieter. This is great when there are lots of channels in a lovely field in the middle of nowhere.
    As everyone’s equipment uses different channel configurations you will have radio data bleeding into other manufacturers channels causing them to frequency hop which may then cause someone else to hop. What you could end up with is a domino effect when everything goes down as they all go looking for a quiet space.

    Wifi (the computer and phone hardware) is quite aggressive when it claims it’s huge channels that the traffic will overwhelm a zigbee transmission due to the higher transmit power given to wifi and the higher amount of data in that wifi channel. Zigbee is limited on its transmit power and Bluetooth is even lower at 1 quarter of the power allocated to zigbee.

    Having read an interesting paper about how wifi messes with zig bee and Bluetooth, I would be inclined to agree that the Ubiquiti Access Points in your location were flooding the entire band. Certainly if I had set them up then I would have spread them across all channels so as not to interfere with each other. That would have been awful for your environment when trying to use more specialist hardware that has lower power, smaller channels and is trying to coexist in a tiny sliver of the 2ghz band.

    link to paper here: https://core.ac.uk/download/pdf/82049199.pdf

    So how could you coexist at future events? The same as what we do at Badminton. Allocate wifi channels to exist at one end of the spectrum and reduce the transmit power on a ubiquity. You only need enough power to make a good connection in point to point mode. Anything higher can cause problems.
    This would leave the other half of the frequency band for the firing system with it’s smaller channels and enough space to hop around. Better still, force all the point to point onto the 5.8ghz networks and don’t allow public wifi near to the Pyro control location.

    Here is a funny story that happened last Friday at a large rugby club. Said rugby club had just opened up from covid having upgraded all their bars to cashless payments only. The new wireless point of sale systems worked perfectly in testing until the bars filled up with punters bringing along all that wifi noise in their phones. A bar full of rugby fans standing shoulder to shoulder brought the card scanners down leaving the owners no option but to give the beer away for free. It was either that or have a riot on their hands.
  6. pyrobrit

    pyrobrit Pro Firer/Crew

    I’ve been messing around with LORA these past few years and I’m amazed at how far you can transmit on 433mhz. Considering it’s low power, I can leave a transmitter in the office and walk into the town Center and still receive a signal over a mile away. Why is this amazing? Well it’s not line of site and for 433, that’s incredible. I only lose signal when walk into an underpass.

    mind you, walking around holding a little antenna does get the strangest of looks.
  7. Jon

    Jon Pro Firer/Crew

    It's a bit of a misnoma that 433MHz requires line of sight. Basically if an obstacle is thinner than the wavelength then it will generally pass through. I quite often use the Fahnam 70cm repeater and I live 6 miles away with an antenna in the loft working a 6 watt handheld and lots in the way.

    When you move to GHz then you begin to see why line of sight becomes more critical as the wavelength is so small. Even in line of sight the fresnel effect can cause issues with nearby objects not in the direct path.
    pyrobrit likes this.
  8. It is nuts. I’ve even linked to some LEO satellites a via it.
    pyrobrit likes this.
  9. Nice post and analysis, thanks.
  10. We are coming in a little late to the party but, there may be some additional points to consider. There are three contributing problems in what's been described in this thread, and all need to be addressed to solve the problem. One is that it has long been known that wireless is fundamentally unreliable when applied to pyro. Most of the time it is OK, right up until the time it’s not. True, that can be said of wired as well, but it drives the probability of failure down a significant number of additional decimal places. The problem is that most RF systems either work on a single frequency and any wideband source can disrupt the signal at a critical time, or they are packetized (spread spectrum and frequency hopping) and the second issue arises -- these systems work like E-Mail -- it doesn't matter when the transmission happens, as long as it happens. This is not often taken into account. For a backyard show, the timing latency isn’t often noticeable, but if I'm trying to shoot at the level of precision for a competition show, it matters. This problem applies whether or not the show is downloaded to the modules, as the transmission delay and skews of the firing command still factor into the precision. Finally, mesh networking is a double-edged sword. It allows the message to be routed around RF shadows, but it eats into your available bandwidth. Every wireless system is constantly fighting for spectrum bandwidth. Each channel (which has a different meaning in Direct Sequence Spread Spectrum (DSSS) and Frequency Hopping (FHSS)) only have so much bandwidth. The spectrum at showtime is always very different from when you do testing. Every source from cell phones, DMX, wireless microphones, PA systems, etc., etc., all eat away at your available spectrum. The more that gets eaten away, the less you have to shoot with until you start seeing random dropouts and finally total collapse. We see this all the time in the theme parks. The mesh network can also eat away at your spectrum because the more modules you have, the more they have to communicate to maintain the network - again eating away at the available spectrum. Your biggest shows are the most likely to experience problems. If considering a system to use, one should always ask if the supplier is actively managing the mesh network, as Cobra noted above.

    In 2005 at the International Symposium on Fireworks Dan Barker and Ken Schroyer (Fire One) published a paper entitled, “Wireless Pyrotechnic Firing Systems Operational Reliability in World-Wide Environments” on these exact topics. Look it up if you can or email us for a copy. Their conclusion was, “A firing system that is frequency agile, utilizes spread spectrum technology, has the capability to operate fully wireless, wired and wired simultaneously, or completely wired, can offer the user a number of methods to set up a display. This type of system provides the flexibility to operate in one, or more, modes of operation, greatly enhancing the probability that the user can surmount any wireless interference that may be encountered.” In the same symposium we published a paper on “Wireless Fundamentals” where we noted that “We have shown that traditional Master/Slave architectures are fundamentally flawed when implemented with radio links between the Master and Slave, due to unpredictable delays in the radio communication link. One solution to this problem is to build a firing system based on Distributed Processing, thereby removing the radio link from the critical path.” You can never make a radio system 100% reliable. The harder you try, the more expensive it gets, but you can never reach 100%. The distributed processing approach turns the problem around so that it’s not a matter of making the radio reliable, but in making the system reliable even though the radio is not.

    The difficulty is in solving all of the above problems simultaneously. The new Firelinx Gen3 system does this by making all of the field modules Master firing controllers. We download the script to each, but the key difference is that we then synchronize the modules so they all fire exactly on time, and the timers are adjusted throughout the show to keep them synchronized to sub-millisecond accuracy. This also minimizes the RF traffic at showtime to only the heartbeat signals; the firing commands need not be transmitted during the show so we need very little spectrum bandwidth to keep the show running. This turns out to actually be a very difficult problem to solve, and our breakthrough was awarded US Patent 7,493,859. Secondly, we control the mesh network so that they’re not constantly overloading the spectrum with mesh traffic. Finally, we can use 2-Wire wired technology (using simple 22-24ga scab wire) and spread-spectrum wireless at the same time. Rather a “belt and suspenders” approach where if there is any momentary interruption in the wireless, the wire will pick up the message, and if someone steps on a wire or it’s broken or burned during the show, the wireless still continues to run. As suggested by Dan Barker, the system can be wired, wireless, or both greatly enhancing the probability that the show will perform optimally.

    On another topic that was mentioned in the thread, we agree that a compromise has to be reached between the safety of the system and the continuation of the show in the event of momentary RF signal loss. I’ve been building wireless systems for nearly 40 years, and even in ideal conditions momentary signal loss happens all the time. It should be taken into consideration that loss of signal does not, in itself, mean the system is unsafe. Overarching protocols from NFPA are also in effect: The field is clear and the spectators are as safe as we can make them. We call this situation Loss of Positive Control (LOPC). The operator can always turn the ARM switch and broadcast emergency shutdown codes. The problem we contend with is what happens if the operator determines an unsafe condition exists, coincident with a momentary (or permanent) drop in the radio link. This is the same problem we have to deal with when a firing command happens to coincide with a momentary loss of RF signal. These can only be truly solved by running wired and wireless together. We allow 10 seconds of continuous loss of comms (wired and wireless) before the modules will fully safe themselves – stop firing, shut down power to the firing system, AND discharge the capacitor banks. The communications and timing systems, however, keep running so that as soon as communications are restored the module can check the next heartbeat message it receives then independently re-arm, skip to the appropriate time and firing cue, and resume the show without any extraneous procedures required of the operator.

    Finally, different people seem to have different definitions of what “Latency” is, so I thought I should provide our definition for clarity. In the firing controller, put simply, the computer is reading the script and watching a timer. When the timer matches the time of the next cue in the script, the computer knows it’s time to send out firing commands. Everything that happens after this moment until the ematch ignites is latency. The problem is that all of these factors are highly unpredictable. First, in some systems a message must be sent to each firing module that is to fire. If there are 10 modules firing on this cue, 10 separate commands must be transmitted, one after another. In other systems the cues are downloaded so that only one message is required, basically “Fire Cue 10” and each module knows if it’s supposed to fire Cue 10. In either case, this firing message is sent to another processor in the system, the RF controller. It’s doing a lot of work on its own maintaining the mesh network and processing the transmission and reception of each message. The firing message from the show controller goes into a queue of messages, so at this time it’s unknown how many previous messages still need to be processed before the fire message gets sent. Once the fire message reaches the top of the queue, the RF controller usually first does a “read sense” first, to see if anyone is currently transmitting. There’s obviously no use in pushing out a message that will just collide with another message on the same channel. These messages can be from anyone, however – WiFi, audio systems, cell phones, etc. which is why the audience and the venue traffic are still affecting the latency of the messages, even though they may not be stopping the messages from getting through. At some point the RF controller sees a clear channel and begins transmission. Here we have to add the complexity of two forms of transmission – broadcast and unicast. Unicast messages are addressed to a specific module and that module provides an acknowledge that it received the message. Broadcast messages go to all modules, but the RF system never knows if it actually got there, so in many systems it will send the broadcast message over and over again, called a “Tell Me Three Times” strategy, to improve the probability that all the modules got the message. It could actually resend ten to even 20 times depending on the criticality of the message. If one module didn’t get it for some reason, it won’t fire but the others will. One can see how this would affect the latency of when each module receives the fire command (and therefore when it actually fires) and how long the RF system takes to complete the message before it can start working on the next one. In the Unicast message, it’s quite possible that some interference happens during transmission and it is not received. The RF system sits there waiting for the acknowledge from the module (because it could be busy doing something else as well), and then times out and starts the whole sequence over again. It will do this many times to keep attempting to get the message through without consideration for how long it’s taking. Eventually it will give up and go on to the next message, and only the audience sees you missed the cue. That, in a nutshell, is the Latency problem. With Distributed Processing it’s the field module, not the controller, that is matching the time against the script and when it’s time to fire it sets the outputs. The radio is no longer in the critical path.
    Saltpetre, RocketRev and Pyro Pete like this.
  11. For clarification, we did not note this nor does your post represent how our MESH works as it may be a bit ambiguous to readers what is suggested in some of the details. If you did not suggest this, then all good :)
    pyrobrit and pyroplayer like this.
  12. pyroplayer

    pyroplayer Pro Firer/Crew Supports UKFR

    I would disagree with this point, no system by their very nature can be 100% reliable in 100% of real-life situations. Such is the long tail of situations that can occur. Wired systems are notorious for broken cables, burnt out cables mid-show, being pulled out with crew tripping over them in the dark. I certainly wouldn't say they're superior to wireless by any stretch, you can't make a wired system 100% reliable either.

    I also think that 99% of what makes Firelinx "unique" is already being done by other systems. From the Firelinx website "Only Firelinx Firing Systems downloads the scripts to the firing modules so they can fire with no latency or RF drop-outs." <- completely untrue claim. Pre-transmitting scripts and syncing with heartbeats isn't new. I've seen great and shockingly terrible implementations of this, the approach alone isn't enough, it has to be implemented well.

    In fact both systems in this incident work this way, one clearly better than the other. It appears the ultimate reason the show didn't continue was a fundamental issue in that Pyrosure can't be stopped from continuing to fire if the signal is lost to the field modules as they have the script running locally, risk was too high and quite rightly so. Wired + wireless fallback is somewhat unique but I'd say for the vast majority of shows is overkill and also doesn't guarantee 100% reliability.
  13. Just some clarification of the above post, forgive me if I'm missing a point somewhere. True, nothing is 100%. The trick is whether you're looking at 99.9, 99.999, or 99.99999 etc. You are correct that simple implementations of sync on heartbeat have been around, and are still around. What we were able to achieve and receive the patent on is dealing with both precision and skew in the synchronization -- the "implemented well" part of your post. We have achieved sync between modules on the order of 0.0000004 (400 nano) seconds, with skews across the system (all module synchronization) of 0.000002 seconds. Most importantly, this is achieved without latency -- a great deal of the problems I described are due to latency. In fact, the patent is entitled "Zero Latency". All other systems that I know of do not have this ability, which was the basis of the claim you cited. If I'm wrong, I'd be happy to be educated. I would also hazard the opinion that having two systems in the show, one wired and one wireless is less effective than one system that does both. But that's just my opinion.
  14. pyrobrit

    pyrobrit Pro Firer/Crew

    [QUOTE="We have achieved sync between modules on the order of 0.0000004 (400 nano) seconds, with skews across the system (all module synchronization) of 0.000002 seconds. Most importantly, this is achieved without latency -- a great deal of the problems I described are due to latency. In fact, the patent is entitled "Zero Latency". All other systems that I know of do not have this ability, which was the basis of the claim you cited. If I'm wrong, I'd be happy to be educated.[/QUOTE]

    But there is latency in your system as mentioned in your patent by the use of offsets. All you have designed is a method to account for the latency, acknowledge it is there and work around it. It's not really Zero latency as the latency is still there in the network but the modules are aware of it and will start their scripts with the latency calculated as an offset.

    Other systems do the same and one system can use GPS to increase the accuracy even lower than your 400 nano seconds. Not that it's really required when you time how long it takes to burst an igniter.

    I agree with your distributed processing. It's the better method in my opinion and happy to see it widely used in the industry.
  15. pyrobrit

    pyrobrit Pro Firer/Crew

    Interesting! Are you bypassing SNAP OS then and talking direct to the hardware?
  16. Actually, the patent was written for our Gen1 system, but in Gen 3 we refined it and filed an additional patent (pending). This allowed for the tighter precision and removing or at least mitigating the last of the latency. You are correct that where there is consistent, repeatable latency it can be taken into account in a number of ways. Most latency sources are not predictable as I outlined above. There is still a problem with this, however, that existed in Gen1 and is mentioned in the patent -- if you are firing a script that starts at 0:00:00.000 -- at the moment you push the Fire button. No SMPTE preroll to let the timing settle in. In Gen1, this would cause latency in the script as it was based on receiving the Fire message. In Gen3, the first shot at Tzero will have latency, but because of the close synchronization of the system, all subsequent firings will be timed well within the 0.001 second precision of the scripts, independent of the latency induced in the first button press. Essentially the system "knows" it's firing the first cue late because of latency, so the induced latency doesn't work its way into the rest of the script. The scripts are relative to the synchronization, the time you pushed the button, not the time the fire message was received. It's kind of weird, but works. We do handle all of the timing, safety, and hardware diddling in the dedicated hardware, not the SNAP OS. The radio system as I said is out of the critical path during the show. There are messages being received and transmitted, but none that are time or show sensitive except for shutdown or other emergency commands which are handled on a priority basis. Still some latency there, of course, inherent in the RF system, but we've removed all we could at each end. Another example of this is what we call a Hardware-Software Interlock (HSI). Software in any processor including the RF Processor or the internal control processors can jump to any random point in the code if there's a "glitch" or a bug hiding in the program code. That includes the possibility of jumping right to the firing code that diddles the firing outputs. The HSI is specialized hardware that ensures that eight safety cross-check steps in the firing process have to all be performed before the firing outputs are enabled. Skip over any of the safety checks and it won't fire even if the software wants to. Switching both the high-side and low-side of every channel to prevent cross-currents, etc. these are all in the hardware, not the software.