Adventures with GPRS

Over the summer I purchased a GPRS Shield for Arduino, which is shield that allows the Arduino to communicate over the cell phone network using the General Packet Radio Service (GPRS).  The shield I purchased is based on a SIM900 module, which seems to be a widely-available component for this purpose.  I purchased my shield on eBay, but they are available from other source, too.  I'll explain more later, but I strongly suggest that you proceed carefully if you do decide to order from an eBay seller, as I had a rather unexpected issue pop up that made trying to use the shield more than a little frustrating.

My GPRS Shield

Step 1: Purchase a Prepaid SIM Card

Before I could use the modem, I needed to purchase a SIM Card and a Matching Service plan.  I didn;t want to wind up with a new monthly bill for this thing, as I was planning to use it only occasionally for a very special purpose.  So, I had hoped to find some kind of "pay as you use it" service plan.  After a lot of research, I found a plan on Amazon that was offered by T-Mobile called the "T-Mobile Sim Card Prepaid Kit."  It cost only $5.98 to purchase the plan and, by carefully selecting my options when I signed up, I was able to get a plan that costs me only $2/day and charges this only when I use the device.  Surprisingly, once I signed up I found I had a $30 credit included with the plan.  The sign up process seemed to require me to add additional money to activate it, so I added $10 and would up with $40 of service credit for a total cost of only $15.98.  Unfortunately, the plan came with a Micro size SIM and not the Standard size SIM my shield needed, so I was forced to make a trip to the T_Mobile store where they sold me a thin piece of cut plastic (a SIM adapter) for $5...

Step 2: Insert SIM Card and Get Frustrated

According to the instructions, I had to first insert my SIM card to activate the GPRS shield so it could identify itself to the network.  This seemed like a simple step but, after spending several hours trying to get the LED that's supposed indicate connection status to blink in the expected way, I was no closer to making anything work.  I was starting to wonder if the card was broken, or if perhaps my bargain service plan was not such a bargain after all.  So, I did what any good engineer does when faced with a problem like this, I started to scour the internet for advice.  There seemed to be a lot of posts suggesting that I needed to add a capacitor to the pins that supply power to the SIM.  So, using a diagram that showed proper SIM pinout, I went probing around to find how to do this.  However, after about 30 minutes of head scratching, I had another mystery on my hands.  The voltage to the SIM was not showing up on the expected pin.  I sat staring at the diagram for more than a few minutes until it suddenly hit me ... the SIM hard holder on the PCB was installed incorrectly.  It was reversed!  Pulling up a photo of a nearly identical shield on the Internet, I was quickly able to confirm this.  And, after a little de-soldering and re-soldering to rotate the SIM holder 180 degrees, I finally had a working GPRS Shield.  Note: insert snide comment here about the dangers of using cheap, imported crap purchased via eBay.

The incorrectly installed SIM Holder

Step 3: Look for Compatible Software

The SIM900 module, which is the heart of the shield, is a serial device that's intended to be commanded by Hayes-style AT commands.  So, I started out by programming the Arduino to simply pass through commands typed into the Arduino's Serial Monitor page.  Using these I was able to send an SMS (TXT message) to my phone, which was a real confidence booster.  Unfortunately, things started to downhill again when I tried to use a GPRS Socket connection to read data from a web server.  Rather than bore you with the long version of this story, here's a list of things I learned the hard way over the next several days:

• • The SIM900 is a twitchy beast and you have to send it commands in a certain, exact way and handle all error responses properly or it can drive you mad with inconsistent results

  • Avoid using SoftwareSerial.  I switched to using an Arduino Leonardo (so I could use the hardware serial port to talk to the SIM900) and this stabilized things greatly.  SoftwareSerial, especially above 4800 baud, seemed to garble characters sent to the module fairly often.

  • You *must* power the Arduino from an external power supply (not from USB) that can supply at least 9 volts at 2 Amps.  The SIM900 uses a lot of power when it transmits (if only for a few microseconds) and this can glitch the power and cause a reset, or other weird response.  Note: some GPRS shields based on the SIM900 have a power socket on the shield.

  • Even when you send commands in the proper order, there are places where a short, 100 ms timing delay is needed after processing the response before you can send the next command.

  • Even through written for Netduino, the following post was a great help http://forums.netduino.com/index.php?/topic/8637-need-for-a-good-gsmsmsgprs-library/

  • What software is available for GPRS shields is mostly crap that fails to perform reliably in many situations.

Step 4: Write my Own Software

After wasting a lot of time trying to get other people software to operate in a consistent fashion, I decided to write a simple piece of "test bench" software I could use to explore the effects of the different AT commands and find out why they seemed to work one minute only to fail the next time I tried the same sequence.  The result is attached below as Sim900TestBench.ino.  I'm not going to hold this code up as an example of great code design, but it should help you understand the sequence of commands needed to use GPRS to fetch a web page using the HTTP protocol.  The code is driven by a set of single letter commands that select and run different AT command sequences.  To send a command, type a letter into the Serial Monitor's text box and press "Send".  The set of supported letter commands is, as follows:

• • Send '-' to power down the SIM900 module

  • Send '+' to power up the SIM900 module (wait at least 30 seconds before issuing any other commands.)

  • Send '.' (period) to activate the GPRS radio subsystem and fetch a web page (Note: Wait until "CLOSED" and "HTTP Timeout" display before issuing another command)

  • Send 'r' to repeat the HTTP request without reactivating the GPRS subsystem

  • Send 's' to shutdown the GPRS subsystem

  • Send 'c' to close the last HTTP Socket connection (not normally needed as socket is closed automatically in most cases)

Note: the code contains debug statements that echo the commands and responses sent to and from the SIM900 module.  Here's an example of what's displayed in response to the '.' (period) command in the above list with all debugging features enabled:

Send HTTP Request

SND: ATE0\r

RSP: \r\nOK\r\n

SND: AT+CGATT?\r

RSP: \r\n+CGATT: 0\r\n\r\nOK\r\n

SND: AT+CGATT=1\r

RSP: \r\nOK\r\n

SND: AT+CSTT="epc.tmobile.com"\r

RSP: \r\nOK\r\n

SND: AT+CIICR\r

RSP: \r\nOK\r\n

SND: AT+CIFSR\r

RSP: \r\n100.129.140.152\r\n

SND: AT+CIPSTART="TCP","www.play-hookey.com","80"\r

RSP: \r\nOK\r\n\r\nCONNECT OK\r\n

SND: AT+CIPSEND=27\r

RSP: \r\n>

SEND OK

HTTP/1.1 200 OK

Date: Sun, 09 Feb 2014 04:10:08 GMT

Server: Apache

Content-Type: text/html

Connection: close

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Frameset//EN">

<html>

<head>

<title>Demonstration HTML Test Page</title>

</head>

<frameset rows="115, *"> 

<frame name="Header" src="header.html" noresize marginwidth="4" marginheight="0" scrolling="no" id="Header">

<frameset cols="50%,*"> 

<frame name="Source" src="source.html" id="Source">

<frame name="Result" src="result.html" id="Result">

</frameset>

<noframes> 

<body bgcolor="#FFCCCC">

<h1 style="align:center">

Frames and JavaScript Required!

</h1>

<p style="text-indent:20px">

This page uses both frames and JavaScript, to enable users to type in their own HTML and then display it

side by side with the source. You will need Netscape 3 or later, or IE4 or later (or a compatible browser

that recognizes both frames and JavaScript) in order to make use of this page.

</p>

</body>

</noframes> </frameset>

</html>

CLOSED

HTTP timeout

Special Notes about the Code

If you look closely at the example code, you see places where the #else case of the SHOW_RSP code contains a call to delay(100).  I'm not sure why, but this delay seems to be needed to get a consistent response from the commands.  If you remove the delay the code works sometimes, but not others.

Also, you can issue a '.' command after a power up and this will fetch the web page, but leave the GPRS subsystem power up.  You can then use the 'r' command to fetch the same page again and again for as long as you like.  However, after a substantial delay (unsure as to the exact time needed) you will need to use the 's' (shutdown) command to close down the GPRS connection before you can use the '.' (period) command again.  You can also use the 's' command after an error to put the system back into a known state (or power down, the power up again).

Unresolved Issues

The code documented here is only a test bench, but does demonstrate the steps needed to fetch data via an HTTP GET request.  It should be fairly easy to extend this to perform other network protocols, such as an HTTP POST operation, but I have yet to try this.  One issue that bugs me, however, is that it's difficult to handle GET requests that don't return an indication of how many bytes the code should expect to read.  The HTTP protocol does provide the content-length header element to handle this, but this is optional and, as you can see, one is not returned by GET request shown above.  This means that I had to use a rather long timeout to be sure I'd read all the bytes.  The SIP900 does return a "CLOSED" response in the text stream that could be used for this purpose, but that poor practice from a coding standpoint because the word CLOSED could be included as part of the data being returned.  However, even using a timeout to detect the end of data is problematic, as the return data could be delayed by a variety of network issues such as a poor cellular connection, or a server that responds slowly due to congestion, or other factors.  Ideally, the best solution is to have control of the server side, too, and make sure that a proper content-length value is included in the response header.  In that case, the connection can then be terminated on the receiving side by issuing a "AT+CIPCLOSE" command in order to turn around the socket connection more quickly.  Note: as I've yet to fully explore all the available AT commands for the SIM900, it's possible that there are commands, or modes that may help me solve these problems.  If so, I'll try to update this page when I've discovered a new solution, or technique.

The AT+CIPHEAD Command

Well, the digital ink was hardly dry on this web page when I discovered a possible solution to detecting how many bytes to read on an incoming socket connection.  The trick is to use the AT+CIPHEAD=1 command to turn on a special mode that adds an IP header prefix to chunks of incoming data that indicates how many bytes of data follow.  For example, if I add the following line of code just after the "case 'r':" case statement:

sendWaitOk("AT+CIPHEAD=1\r"); 

and then rerun the '.' (period) command the returned data will now look like this:

SND: AT+CIPSEND=27\r

RSP: \r\n>

SEND OK

+IPD,931:HTTP/1.1 200 OK

Date: Sun, 09 Feb 2014 22:30:36 GMT

Server: Apache

Content-Type: text/html

Connection: close

<html>

... truncated for brevity

As you can see, the beginning of the HTTP response sent back from the server is now prefixed with the text "+IPD,931:", which indicates that 931 bytes of data will follow just after the ":".  In this case, the entire response from the web server fits into a single, 931 byte chunk.  To get more data, you can change the URL to point to a different server, such as www.google.com by making a few changes to the code, such as:

    if (!sendWaitSeq("AT+CIPSTART=\"TCP\",\"www.google.com\",\"80\"\r", "CONNECT OK\r\n"))

      break;

    // Note: 27 is the length of the GET command string written after this command

    if (!sendWaitSeq("AT+CIPSEND=28\r", "\r\n>"))

      break;

    delay(100);

    Serial1.print("GET /index.html HTTP/1.0\r\n\r\n");

Ignoring the data that comes back, you should see a series of blocks of "+IPD" prefixes, like this:

+IPD,1400:

+IPD,1400:

+IPD,1400:

+IPD,1460:

+IPD,1460:

... several more blocks of 1460 bytes

+IPD,814:

Now, I have no idea why the first three blocks are 1400 bytes in size and remaining full blocks are all 1460 bytes in size, but this does suggest a possible way to detect when all the data has beens sent by the server.  That is, look for the block that has a size <1400 and assume that this is the last block of data.  But, there are a few problems with this.  First, the code will need to made more complex in order to parse and strip out the "+IPD" prefix blocks.  Also, what if the last block of data is exactly 1460 (or 1400) bytes long?  How do we know this is the last block?  Hmm, more experiments are needed here.  Stay tuned...