Machsupport Forum

Third party software and hardware support forums. => Mach3 and G-Rex => Topic started by: moreil on July 25, 2007, 09:30:15 PM

Title: G100 crashing?
Post by: moreil on July 25, 2007, 09:30:15 PM
I have a G100 that I'm trying to use the 2.01 g100 plugin with. I've uploaded the firmware and the FPGA files that come with the plugin, but it's not working.

If I start mach3, it repeatedly says 'Found G100', initializing, Found G100, initialing, etc etc etc.

If I just leave it, the G100 starts up with the run LED red. Then after 10 seconds (autoboot timeout), the run LED goes out for about 1 second, following with it comes back on again as red. Rinse, repeat.
When it comes on again red, it's the boot loader running (I can connect to it with a browser).

It looks like the G100 is starting the firmware, running for about 1 second, crashing and restarting the boot loader?

The bootloader says:

Firmware size (bytes):   190806
Core:   RCM3720/RCM3750

So it sure looks like it has the right firmware? I've tried re-uploading the firmware without any change in behaviour.

If I click 'test' in the FPGA section I get:

May 26 21:35:38 [1]: Download manager started
May 26 21:35:41 [2]: ifconfig: ipaddr=C0A80160/FFFFFF00 dhcp=no
May 26 21:35:41 [3]: Starting network...
May 26 21:35:42 [4]: IF_ETH0 is down
May 26 21:35:45 [5]: CRC_OK -- yes
May 26 21:35:45 [6]: Auto boot DLP in 10 sec
May 26 21:35:51 [7]: Ethernet interface is up at 192.168.1.96
May 26 21:35:52 [8]: Auto boot postponed, HTTP used
May 26 21:38:06 [9]: Loading FPGA config 1...
May 26 21:38:07 [10]: XSVF loader completed
May 26 21:38:07 [11]: Programming time = 1046ms
May 26 21:38:07 [12]: Config completed
May 26 21:38:07 [13]: FPGA version register (addr 0x01) reads 0x08
May 26 21:38:07 [14]: FPGA vers.release = 0.8
May 26 21:38:07 [15]: FPGA status register (addr 0x00) reads 0x94

Which looks pretty right to me?

Any ideas??
Title: Re: G100 crashing?
Post by: Chaoticone on July 25, 2007, 09:49:40 PM
I have sent you an e-mail that I hope will help you. Sounds to me like you don't have the frimware loaded right or the right firmware. I hope the e-mail helps you. It is the begining of the G100 set-up guide.

Brett
Title: Re: G100 crashing?
Post by: Chaoticone on July 25, 2007, 10:04:27 PM
May be a network issue. Can you ping the G100?

Brett
Title: Re: G100 crashing?
Post by: moreil on July 26, 2007, 02:55:37 AM
I'm reasonably sure it's not a network issue. I can bring up the download manager web page without any problems, and I can ping the G100.

At least, I can ping it when it's up. It follows the cycle of ping'ing fine for about 15 seconds, then it's off-line for about 5 - 10 seconds and the it's pingable again (As it's constantly cycling..)

What's the byte size of the right firmware? And which firmware is the 'right' one?


Note that I managed to get it working once. After posting initially, I went over and started mach3 and it come right up. I was able to move all the axis on the machine. Then I closed mach3 and restarted it again, and it went back to cycling and I haven't managed to get it to work again.

(Clearly shutting it down was the wrong thing to do! :)

I suspect that there's some sort of race-condition or timing issue with the way that mach3 initialises the G100. It's not waiting long enough for the G100 to come back onto the network or something.

Ps: Thanks for the updated setup guide, but I'd already done those items.  NI've double checked it, and re-re-re-uploaded the firmware just to be sure. :( Given that it worked once, I'm strongly doubting that the firmware is the wrong file. Note that the firmware size listed in the setup guide is the same size that the download manager is reporting.
Title: Re: G100 crashing?
Post by: moreil on July 26, 2007, 03:49:36 AM
I just put a sniffer on the link between the PC and the G100. The results seem to indicate that the plugin is broken. The plugin issues a /run to the G100, waits 10 seconds and tries to talk. A couple of second after that, the plugin seems to give up and goes back to trying to send /run again. The G100 seems to take a little bit longer than this to come up as a few seconds after the TCP SYN is sent, the G100 sends out a series of broadcast packets (presumably announcing it's alive?).  The plugin keeps on trying to run through it fixed routine with fixed timing where it's not listening when the G100 is talking, and it's not talking when the g100 is listening.

Could someone possibly post the snipper of code that handles the G100 discovery and sync so I can have a go at fixing it for you?

To date, this G100 has just been endless grief. :( I really regret buying it.
Title: Re: G100 crashing?
Post by: moreil on July 26, 2007, 04:34:31 AM
More detail from the sniffer (sorry, I had to type this back in by hand as the mach PC is off the network at the moment).

time  action

4.5   PC sends UDP discover to 13887
4.9   PC sends UDP discover to 13887
5.4   PC sends UDP discover to 13887
6.1   PC sends SYN
9.1   PC re-sends SYN
9.4   PC sends UDP discover to 13887
9.9   PC sends UDP discover to 13887
10.4   PC sends UDP discover to 13887
11.1   PC re-sends SYN
11.469   G100 sends SYN-ACK (segment lost)
11.472   G100 sends reply to discover to 255.255.255.255
11.473   G100 sends reply to discover to PC
11.477   G100 sends reply to discover to 255.255.255.255
11.478   G100 sends reply to discover to PC
11.480   G100 sends reply to discover to 255.255.255.255
11.481   G100 sends reply to discover to PC
11.484   G100 sends SYN-ACK
 [ tcp conversation doing POST /run.cgi follows ]
11.629  ends.
19.931   PC sends UDP discover to 255.255.255.255 port 13887
20.368   PC sends UDP discover to 255.255.255.255 port 13887
20.806   PC sends UDP discover to 255.255.255.255 port 13887


21.572  PC sends SYN
24.540   PC re-sends SYN
24.853   G100 sends reply to all packets!


It appears to me that the mach plugin stops listening for discovery replies as soon as it starts trying to send the POST /run.cgi

Basically, the G100 takes 7 seconds to start replying to the discovery packets (still going through boot-up?)  but mach gives up a few seconds too soon.

Title: Re: G100 crashing?
Post by: Chaoticone on July 26, 2007, 11:52:04 AM
I have sent you a personal message.

Brett
Title: Re: G100 crashing?
Post by: 123cnc on July 30, 2007, 03:16:06 PM
ChaoticOne,

Could you post or reply with some light on this problem.

I'm having identical trouble with my G-Rex and had posted on CNCzone with no updates.

My sentiment, similar to moreil, is also nearing the end of what I thought was a great buy with the G-Rex, but it seems like a really bad relationship, suffering from poor communication.

Title: Re: G100 crashing?
Post by: moreil on July 30, 2007, 03:59:39 PM
The outcome is no outcome.

There's no configuration problem, the network works fine. It's fairly clear from the sniffer traces and the G100 documentation that the pluginhas a bug of some sort.

Basically, the communication between the G100 and the plugin live-locks: The plugin is not waiting long enough for the reply back from the G100 and so attempts to reset the G100 even as it's replying.

If I leave it for a few hours, it eventually sync's up, so the timing is close, but not quite.

Very very annoying.

I suspect it's time to give up on the g100. I'm currently looking at the EMC2 software and the Mesa Electronics M5I20 board. It a PCI board that plugs into the PC, but at least it's stable!
Title: Re: G100 crashing?
Post by: Brian Barker on July 30, 2007, 05:28:12 PM
I have sent this to Steve (the firmware guy for the G100) we will see what he thinks about his and if he can get it working for you.

Thanks
Brian
Title: Re: G100 crashing?
Post by: Brian Barker on July 30, 2007, 05:39:43 PM
Also I have had this problem in the past.... what do you have for network connections? Do you have more then one network setup on that computer? Do you have it plugged in where there is a router that is not liking the IP range? I have found that I have to turn off all other networks on the Host computer to get the device to upload the firmware.... Please try a few things and als know that it takes time to make things better and we are here to help you (I feel your pain)


Thanks
Brian
Title: Re: G100 crashing?
Post by: Chaoticone on July 30, 2007, 05:46:21 PM
I don't know what to say, mine works fine. Moreli, did you follow the manual to a t? When you tried to install the firm ware the second time, did you change your auto boot time to 0, load the xsvf file, change the boot time to 10 only after you load the bin file. I have attaced what I have of the set-up guide so far.

Brett
Title: Re: G100 crashing?
Post by: moreil on August 01, 2007, 06:43:31 PM
Also I have had this problem in the past.... what do you have for network connections? Do you have more then one network setup on that computer? Do you have it plugged in where there is a router that is not liking the IP range? I have found that I have to turn off all other networks on the Host computer to get the device to upload the firmware.... Please try a few things and als know that it takes time to make things better and we are here to help you (I feel your pain)

I posted the details to this earlier. I have full communication with the g100 outside of mach3. I can ping it, I can bring up the download web page. I have quadruple checked that the firmware is correct, and can confirm that the size listed in the download manager matches the byte size in the zip file, and matches the screen-shot in the manual.

There is no network, router or switch anywhere. The computer is off the LAN and is plugged directly into the G100 with a cross-over cable. There are no other network interfaces on the computer.

As I said previously, it does eventually sync up if I leave it for a few hours, and once it's sync'ed, I can drive the machine with no apparent problems. This seems to me to rule out network problems.

The network sniff trace I posted earlier seems to show fairly clearly to me that the mach3 plugin simply has the timeout's wrong: It's just not waiting long enough for the G100 replies.
Title: Re: G100 crashing?
Post by: moreil on August 01, 2007, 06:48:50 PM
I don't know what to say, mine works fine. Moreli, did you follow the manual to a t? When you tried to install the firm ware the second time, did you change your auto boot time to 0, load the xsvf file, change the boot time to 10 only after you load the bin file. I have attaced what I have of the set-up guide so far.

I'm happy that yours works for you. :) Mine doesn't. :(

I've done every step in the manual to a T, multiple times to make sure. As I said, it does eventually sync up, which it wouldn't do if the firmware or FPGA load wasn't working. The boot time is set to 10 seconds as per the manual.

On my setup, I can see the PC send 'POST /run.cgi' to the G100 to start the firmware. The PC then waits 10 seconds, then tries to get the config from the G100. 10 seconds later it gives up and goes back to trying to reset the G100, 3 seconds after that the first reply comes back from the G100 which the PC ignores because it's too late.

I wouldn't mind seeing a network sniff of a working connection to see what the difference in timing is.  I'm a bit puzzled about how my setup differs to anyone else. Normal PC running XP SP2, direct cross over connection to G100. What's different?
Title: Re: G100 crashing?
Post by: Chaoticone on August 01, 2007, 07:06:07 PM
Did you set the auto boot time to 0 the second time you tried to load the firmware?

Brett
Title: Re: G100 crashing?
Post by: Brian Barker on August 01, 2007, 11:38:14 PM
There is a new plugin on the web and it has much more time biult in... tell me if that helps
Thanks
Brian
Title: Re: G100 crashing?
Post by: moreil on August 02, 2007, 03:34:28 AM
Yay! Come up straight away. And it still works when connected to the ethernet switch. excellent!

Much happier.

Many thanks,
Michael.
Title: Re: G100 crashing?
Post by: Brian Barker on August 02, 2007, 07:01:25 AM
That is what we like to hear!

If you ever have a problem and I don't post it is becuase I didn't see the thread :( I don't look at every post but if you send me a PM I will have a look at the thread.
Sorry it was not fixed sooner..

Brian
Title: Re: G100 crashing?
Post by: moreil on August 03, 2007, 04:57:45 AM
Spoke too soon. :(

It's back to it's old tricks. mach3 endlessly looping around 'Found G100', 'init G100', etc etc.

It's pretty random as to if it connects or not. Sometimes it connects as soon as I start mach3, sometimes it goes into the loop. When it goes into the loop I can't get it to stop it. Exiting mach3 and power cycling the G100 doesn't seem to help. I've going back to use a direct cross-over ethernet cable without any change.

The network trace is slightly different: I see mach3 send a g100 discovery packet to 255.255.255.255 and the G100 sends a reply (also to 255.255.255.255 amusingly enough!). But mach3 is still looping.

:(

Any other ideas? Sending me a snippet of the code that handles the G100 sync would really help: It used to be part of the plugin SDK it seems, but I can't find it anymore (and it would be out of date anyway it seems).

Are there any debugging flags I can turn on to have the g100 plugin log what it's trying to do?
Title: Re: G100 crashing?
Post by: moreil on August 03, 2007, 04:58:57 AM
One other question: When I click on 'test' on the G100 download manager page for the FPGA software, the run light goes out. But when the mach3 plugin sends POST /run.cgi, the run light on the G100 stays lit. What's the behaviour supposed to be here?

Many thanks,
Michael.
Title: Re: G100 crashing?
Post by: moreil on August 03, 2007, 05:59:47 AM
Ok. Some more information. I found the protocol_g200x.h and decoded some of the packets. The WTconfig structure that comes back from the G100 has version set to VERSION_DLM (0x00) constantly. In a trace lasting for 47 minutes, no packet ever came back from the G100 with version set to VERSION (0x01).

This seems to indicate that the loaded firmware is immediately crashing in some fashion? Certainly if I click on the 'Run' button in the DLM, it came back to the DLM in about 15 seconds. Is that normal? or should the DLM web page stay unavailable after the firmware is started?

I've re-loaded the firmware about 20 times now. I've checked the size in flash versus the file size and versus the size listed in the screenshot in the manual many times. I'm using the firmware that comes with the G100 plugin.

Is my G100 simply broken??
Title: Re: G100 crashing?
Post by: moreil on August 03, 2007, 06:30:23 PM
Still beavering away on this: When I click on 'Test' in the G100 download manager, I get

<6>Feb 18 19:02:18 [17]: XSVF loader completed
<6>Feb 18 19:02:18 [18]: Programming time = 1046ms
<6>Feb 18 19:02:18 [19]: Config completed
<6>Feb 18 19:02:18 [20]: FPGA version register (addr 0x01) reads 0x08
<6>Feb 18 19:02:18 [21]: FPGA vers.release = 0.8
<6>Feb 18 19:02:18 [22]: FPGA status register (addr 0x00) reads 0x94

#1. Is that the right version? 0.8?

#2. The G100 firmware programming manual indicates that bit 7 is the main FAULT output. It's set if there's a watchdog timeout. Is that to be expected when using test? What do output do other people get from the test button?

Many thanks,
michael.
Title: Re: G100 crashing?
Post by: 123cnc on August 06, 2007, 02:04:06 PM
My experience strongly links this behavior to a hardware issue, my gRex really doesn't like temperatures above 92 F. If ambient air is slightly below 90 F, I will likely lose communication once the gRex warms up, if ambient is above 92 F, then communication is never established. It does the infinite loop of initializing, reboot, reconnect...etc., Below 90 F and all is well, can run and run, stop and restart, no problems.

Luckily the weather finally cooled down here a bit to get some run time and not have to wait til the wee morning hours.  I'm located just a few miles from Xarach which isn't too far from Davis CA where the Rabbit leaves the hat.
Title: Re: G100 crashing?
Post by: moreil on August 06, 2007, 05:22:17 PM
I suspect I'm seeing something similar. The ambient here is in the 16 - 22 celsius range (60 - 71 F), but the behaviour of leaving it on for a long and it eventually come good matches a temperature response.

Seems very weird for what's supposed to be production hardware. :(

Did you find a workaround other than warming or cooling the box?
Title: Re: G100 crashing?
Post by: 123cnc on August 07, 2007, 02:29:45 PM
I contacted Gecko Drive to get their recommendations. They have seen approximately 5% of the Rabbit Semiconductors module RCM3720 with networking trouble. I'm going to order a new one and I'll post my results.
Title: Re: G100 crashing?
Post by: moreil on August 07, 2007, 07:25:23 PM
I got no reply to my query. :(

Are they replacing it or are you buying a new one?
Title: Re: G100 crashing?
Post by: mhdale on August 09, 2007, 11:56:23 AM
Hi guys, I had a similar problem, my Grex was working fine for a while and then this reboot thing started happening to me as well. If I shutdown and restarted it would clearup most of the time. I struggled with it untill the direction output went on the A Axis and then I sent it back to Gecko. The spare we installed in the system has worked flawlessly, never had any issues or problems with it. I remember when I first got the first GREX i struggled with getting the firmware loaded getting connected etc. Of course the second time around I was more informed but to me it seemed like the newer unit had fewer "issues". Perhaps some of the earlier versions had some networking issues which have since been resolved. My G-100 is currently at Gecko being "examined" if I learn anything more I will post it here.
Cheers
Title: Re: G100 crashing?
Post by: mhdale on August 09, 2007, 02:05:43 PM
I Just heard back from Gecko, the G100 had a weak oscillator (Probably a bad crystal) which would cause it to not start up reliably, It may be possibel that this could be related to the Init - run - Init - Run Loop we seem to have all experienced. If so perhaps it was a bad batch or something. I realize that not all similar problems have the same solution but there you have it.
Cheers and good luck!
Title: Re: G100 crashing?
Post by: Chaoticone on August 09, 2007, 05:03:18 PM
Thanks for the report mhdale. Little things like that helps us all a lot.  ;)

Brett
Title: Re: G100 crashing?
Post by: wnevels on August 10, 2007, 04:11:29 PM
Does the G100 ever do anything but crash.   I read these threads and really have to wonder.

Wes
Title: Re: G100 crashing?
Post by: Chaoticone on August 10, 2007, 08:22:52 PM
Hey Wes,
    Mine works pretty well. There are still a few things I would like to see worked out, and may be now, not certain though. The homing routine seems finiky for some, mine works pretty well with only an ocassional hick-up.


Brett
Title: Re: G100 crashing?
Post by: 123cnc on August 28, 2007, 03:19:46 PM
I replaced my Rabbit RCM and now the world seems right.

Wes, it may seem like the GRex can be a load of trouble, but I think you would generally find most people happy with the outcome. Unfortunately it is when we are deep in despair and trouble that we seem to speek the loudest. Boards such as these help to find a fix and keep the relationship going strong. After getting past this 'failure to communicate' problem, I've already committed to another GRex, and will likely again. Of course my enthusiasm lies strongly with the continued efforts and development at both Mach and Gecko. You would have difficulty to find two companies that are as dedicated and as friendly in supporting there users.
Title: Re: G100 crashing?
Post by: Chaoticone on August 28, 2007, 04:31:55 PM
Thanks for the kind words 123cnc, :)
    This is a great group and everyone works hard to get things done. I don't see how Art and Brian keep up with everything as fast as it moves. No one is perfect, but Artsoft and Gecko really stand behind their products. Everyone screws up sometimes, it is how they handle it that makes the difference. 


Brett
Title: Re: G100 crashing?
Post by: 123cnc on October 07, 2008, 12:52:38 PM
Just to be sure...  I know that the Firmware (5420 or 5720) has a free trying license time of 1 hour.  Can I presume that my bug will be ok after buying the 50$ license or not! 
I say that because the only time that I'm able to connect to the G-100 and work my drives, its after clearing everything and restart from origin.  It work a little time, restart, work again and after that I loose it again and after loosing it, I cannot connect to it again...
Please, tell me, from your experiences, if its can be a licensing bug... Kind off...
Title: Re: G100 crashing?
Post by: Chaoticone on October 07, 2008, 01:16:00 PM
I have never seen those issues from a licensing bug but it has been a very long time since I messed with the G100. Also at the time I was installing it, I had allready bought the licenses so can't say either way.  :(

Brett
Title: Re: G100 crashing?
Post by: 123cnc on October 07, 2008, 07:57:04 PM
Ok,

Just to be sure... Its in a text file that come with the Firmware...
The Firmware File Name: G100_DLP_3720_770402.bin
The Firmware "presumed" license file: G100Lic.dat "That I dont already have in this installation"

I was thinked that the g-100 disconnecting bug come from this point.... 

Thx, :)