Issue staying connected to Pi's

Solved
Enerstar's picture
Enerstar
Junior(0)

I have pis out in the field and been having disconnects that only are resolved power powering your board off/on. They route default eth1 at boot and also run that command at midnight just to try and fix this issue. When first installed they were on old firmware so this happened often. I went out there and update the firmware and it has been better for about two weeks but now i lost them both last night. According to my dashboard the sim cards lost and re-established a session overnight. sudo route default eth1 ran but did not repair the issue. I think I can duplicate this by pulling the sim and putting it back in. When i do this eth1 is no longer viewed when I ifconfig. So whenever the sim loses connection the pi can never even send the command to default all traffic through it. Ive emailed support again but really need a resolution because going out and unplugging the card is not viable as some are hours away from my office. Any help would be appreciated. Thank you!

LightCollector's picture
LightCollector
Moderator(20)

Could be a lot of things, can't tell from your post for sure.  You need to remove any entry in the route table of the Pi before adding another of the same.  You didn't say if you did that or not but you need to if not.

 

If the 14A2A is not reconnecting to the celluar system you could try to use AT commands through the USB serial to reconnect it.  The AT command manual is on ATT's site:  https://s3-us-west-2.amazonaws.com/starterkit-assets/AT%26T_Commands_Gui...

 

Also you could damage the board by removing the SIM while it's on.  I would not do that.

 

Best regards

Enerstar's picture
Enerstar
Junior(0)

I do not delete the old route it just says the route already exists but I know that is not the issue because this happens with or without me running route multiple times. That was just an attempt to fix the issue reconnecting. The sim cards were not pulled from any of the devices in the field just my test one to try and duplicate this issue. If the sims ever lose reception and come back the pi will not recognize the 14A2A and eth1 will not show up until it is powered down. Rebooting the pi doesnt work nor does disconnecting the board from the pi and reconnecting just the powering down the 14A2A. Do I have to schedule AT commands? I just need it to reconnect if the sim loses service. I do believe it is reconnecting to the cellular service as the sim goes back to being in session. 

LightCollector's picture
LightCollector
Moderator(20)

Enerstar,

 

It is hard to say yet whether the 14A2A is at fault or the PI or the combination.  What I have seen happen is that when the 14A2A loses signal the IP given back becomes a local one and is invalid for routing.  The PI sees an IP and doesn't know that it is invalid.  You can determine if it is doing that with ifconfig and noting the IP returned is a 10.0.0.x one.  If that is the problem, other than a power cycle of the 14A2A the only other thing I can think of to try is ifdown and ifup to try to renew the effective DHCP given IP.

 

Through the USB serial port there are AT commands to disconnect or connect to the celluar system.  I do not see one to do a soft reboot.

 

If rebooting the PI doesn't fix it though, ifup/down probably won't either.  It may turn out a manual power cycle is the only way to get it back.  It's not a great answer but they do make usb controlled relays, you could use one of those to reboot the 14A2A.  It would be nice if you didn't have to do that but that is likely the quickest solution, sorry.

 

Best regards

Enerstar's picture
Enerstar
Junior(0)

I have thought about the USB relay as a last resort but it just seems like such a work around. I have found the reset button on the unit works to bring eth1 back up. When the sim goes offline the IP goes to 10.0.0.0.X. When the sim is back online there is actually 30 to 60 seconds that it will have a valid IP address. After about a minute eth1 dissappears. I checked and ttyACM0 is gone at this point and the only way to bring it back is the reset button or power cycle. I do not know if this is an issue I can resolve on my end with the pi if I can not communicate. Being IOT devices physically hitting the reset button is not an option. Would this be an issue your company would be looking into in the future? It seems to be a pretty big deal for anyone running pi's longterm in the field. Thank you.

LightCollector's picture
LightCollector
Moderator(20)

Hi Enerstar,

 

I understand and I am truly sorry the device is not doing what you need it to do. The cellular modem 14A2A is made by a company called WNC.  Since it is a cellular device every time a firmware change is made it can take a long time to go through testing and then publically released.

 

It is why I think for you in the short term you should use something like a USB modem if you can.  Even if it is not ideal, it is something that you should be able to quickly attach to the system and use it.

 

Another idea would be to attach a Pi GPIO to have it effectively push the reset switch, meaning electrically, not mechanically.  I am not sure this would work however it was not designed to do that.  You would have to study the schematic to try and see if the reset switch of the 14A2A was doing a pull-up or pull-down.  You would also need to make sure the voltage of the Pi and 14A2A are compatible.  The 14A2A is 1.8V,  a level translator might also have to be placed in between the Pi and 14A2A.  If you don't do it correctly you could damage either the 14A2A or the Pi.

 

If you have a chance when you get it in this state, could you see what happens if you unplug the 14A2A USB cable from the Pi and without any power cycle of the 14A2A, try to plug the USB cable into a PC that is running the Connection Manager?  Can Connection Manager wake it back up?  I am not sure whether the problem is all in the 14A2A or whether it is some interaction with the Pi, doing this might give us another clue.

 

Best regards

Enerstar's picture
Enerstar
Junior(0)

I have one unit that discconnects and reconnects almost daily with no issue. Then Thursday over night I had two permanently disconnect. I am going to try the really low tech route of just plugging the 14A2A into a timer that goes off for a half an hour daily, then power back up the board and re add the route. I will let you know how it works out. This almost looks like the issues i was having running the old firmware which would go offline like this every other day instead of weeks. I wonder if its something with going to sleep.