USING A SD CARD WITH FAT16, PIC18F2550 and Bｌuetooth

I bet some day we have needed to or wanted to save a large amount of data using the common SD cards and some microcontroller.
For example to save samples of temperature every hour, or every minute, to read pictures and show them in a full color display, or maybe to make an mp3 player. Well the possibilities are huge.
This time I wanted to save EMG signals (Electromyography) from my active EMG sensors. For this reason I made a circuit and a program for the PIC18F2550 microcontroller.
Basically I have to meet two requirements.

1. High storing capacity (Enough to save several hours of data at 1kHz-12bit per sample)
2. High sampling frequency (Around 1kHz) 

The transmission by SPI is  the easier way to communicate a microcontroller with a SD card. According to wikipedia the lowest speed for SD Cards is 2Mbps, so I thought it was enough speed for my goal.
CLUSTERS and SECTORS
Unfortunately, the SD cards are divided into Clusters and the clusters divided in sectors. A sector id the minimum quantity of memory that must be recorded each time you access to the SD card.  One sector is 512 bytes large, it means that every time we access to  the SD card we MUST to write at least 512 bytes. Is NOT possible to store only 256 bytes for example and then the next time we access the another 256 bytes.
In order to use all the memory we must to save the 512 bytes in a buffer and then, once filled the buffer save the data into the SD card.
DO YOU WANT TO SEE YOUR DATA IN WINDOWS?
The other thing I wanted to do was to be able reading the stored data in my computer using windows. To achieve that is necessary to save the data in a format known as FAT16 or 32, maybe FAT12 is also possible.  The FAT system is a way of saving data, enabling to the systems that have implemented the same system to read the data stored.
Implementing FAT16 in a microcontroller with limited resources (memory, speed of processor, etc) is a pain. What I do is to search for libraries already done for someone else and then, only use  them in my main program.
I tried the libraries that accompanied the CCS compiler which is the compiler used here in the lab. Unfortunately it seems this compiler has many bugs and are difficult to implement in a microcontroller different of that used in the example of the compiler furthermore, these libraries are large and high consuming memort and resources, after try several times I quit and looked for another option.  
I find a very good option, the libraries made by SUKY, an Argentine forum participant who kindly is sharing his work.
The libraries are
FAT16_SDCard.c  :Implements the FAT16 format
SDCard_hard.c  : Initialize the SDcard, write and read the SDCard by sectors.
You can download the latest version here or the files that I used by clicking on the name of each  file above. I recommend you go to the forum (it’s in Spanish) for the latest news from the author.
I used a very simple circuit using the PIC18F2550, a Bluetooth module KC21 and a card adapter. The figure below illustrate the connection SD card-Microcontroller-Bluetooth. Also used the channel 0 of the ADC to read the analog data and save the conversions in ASCII into the SD card.

The next figure shows my circuit in breadboard. As you can see is a very simple and ugly circuit but it works!!

The card is a 512MB DigEcam without any more information. A very cheap and generic SD card.
You can download the .c file of my program here or just copy directly from the next box. (My program is based in one example of the SUKY´s programs, the maker of the SD library)

#include <18F2550.h>            //Choose the microtroller to use
#device adc=10                  //ADC output 10 bits 
#fuses HSPLL,NOWDT,NOPROTECT,USBDIV,PLL5, CPUDIV1,NOLVP      //Hig frequency, no WDT, No code protection, No low voltage programming
#use delay(clock=48000000)         //Xtal frequency = 20MHz
#use rs232(baud=115200, xmit=PIN_B5, rcv=PIN_B4, FORCE_SW)   //Set-up RS232 module, 11.5kbps
#include <SDCard_hard.c>
#include <FAT16_SDCard.c>
#include <string.h>        //Library for strings management
#define ARRAY_SIZE 512     //Array size according to the sector of SDcard size
//**************************************************************************************
void Save_ADC_sector (void);
//**************************************************************************************
char Texto1[50]={"******ADC Data**********\r\n"};
char Data_ADC[ARRAY_SIZE];//
/*******************************************************************************/
//Read and ADC, convert values to ASCII and store them in an ARRAY 512 bytes size 
//That ARRAY will be stored in one sector of the SD
void Save_ADC_sector (void){
int16 len;      //Variable for controlling the temporal length of the conversion
int m;         //Variable for counting the number of samples
int16 value;   //Store the ADC reads
float voltaje; //Store the ADC conversion into voltaje   
char Temp_ADC[10]; //Store the ASCII data of each conversion
int16 totalLen =0; //Counting the total length of the intended 512 bytes size ARRAY
Data_ADC [0]=0;
for (m=0;m<63;m++)  //Number of readings to fill the ARRAY 512
{
delay_ms(10);        //Delay between samples
value=read_adc();    //Read the ADc value
voltaje=3.3*value/1024; //Converting read into voltage
sprintf (Temp_ADC,"\r\n%1.3f",voltaje); //Creating a miniarray with the data in ASCII including \r\n
printf("\r\n %1.3f \r\n ",voltaje);    //Sending the converted value to a PC via RS232, just for control
len=strlen(Temp_ADC);            //Length of temporal array
//   printf("Length Temp_ADC %lu\r\n", len);//showing the length of the array
//   printf("Total length %lu\r\n", totalLen); //Showing the length of the ARRAY 512
if ((totalLen+len+1)<ARRAY_SIZE)      //Controlling the total length, no more than 512
{
//       strcpy (&Data_ADC [totalLen],Temp_ADC);     
totalLen+=(len+1);
strcat (Data_ADC, Temp_ADC);
}
else
{
printf("Length before out %lu\r\n", totalLen);
break;
}
}
printf("Length before out %lu\r\n", totalLen);
}
//***********************************************************************//
//Main program
//Creates a file "Datos.txt" into a file named LOGGER already in the SDcard
//Call a function to read the data form the ADC until fill an 512 bytes ARRAY.
//Add the ARRAY to the created file 10 times
void main(){
char NombreArchivo[13]="Datos.txt";  //Name of the file to create
char Carpetas[20]={"/LOGGER"}; //Name of the file existent in the SDcard
int16 UbicacionFolder;
//Setting up the ADC module
Unsigned int16 value;      //Defines the variable to save the data from the ADC
setup_adc_ports (AN0);     //Defines which ports will be used for the ADC module
setup_adc(ADC_CLOCK_DIV_32);//Selects the TAD=1.6uS (XTAL=20MHZ) total adq time=17.6us/10bits
set_adc_channel(0);        //Selects the channel from where the measures will be taken
setup_wdt(WDT_OFF);
setup_timer_1(T1_DISABLED|T1_DIV_BY_1);
setup_timer_2(T2_DISABLED,0,1);
setup_timer_3(T3_DISABLED|T3_DIV_BY_1);
setup_comparator(NC_NC_NC_NC);
delay_ms(1000);
printf("Starting SD card\r\n");
if(SDCard_init()==0){
printf("Error initializing the card\r\n");
}else{
printf("The SD card has been correctly initialized!!!\r\n\r\n");
FAT_init();
if(FAT_FindDirectory(Carpetas,DirectorioRaiz,UbicacionFolder)==0){
printf("Error, the file CARPETA can not be found\r\n");
}else{
printf("Creating the TXT file\r\n");
if(FAT_CreateFile(NombreArchivo,NombreArchivo,UbicacionFolder,Texto1)==0){
printf("Error creating the TXT file\r\n");
break;
}
int16 r;
for (r=0;r<100;r++)
{
Save_ADC_sector();
printf("Appending data: %lu \r\n",r);
output_high (PIN_B7);
if(FAT_OpenAddFile(NombreArchivo,UbicacionFolder,Data_ADC)==0)
{
printf("Error the file can't be found");
output_high (PIN_B6);
break;
}
output_low(PIN_B7);
//            delay_ms(10);
}
}
}
while(1){
delay_ms(200);
output_high (PIN_B6);
delay_ms(200);
output_low (PIN_B6);
}  //Infinite loop doing nothing
}

Using that you will be able to create the file that you can see in the next figure. In the program I’m only saving 51200 bytes (100 sectors), around 6300 samples.
 
The big problem
Well, the program save correctly the data, The microcontroller is able to sample at relatively high rate, more than 10kHz, I think, but unfortunately, the memory is very slow and it takes hundreds of milliseconds to save 512 bytes.
Actually, because the fragmentation of the memory card, the required time for saving data increases exponentially. I measured the shortest time (SD card clean, just formatted) in 164ms for each recording of sector.  After around 50 sectors the time required was almost the double (292ms) and so on. The last time I measured was 468mS (I was recording 10000 sectors but I don’t know how many sectors were actually recorded).
It means, that while the memory is being recorded, the microcontroller can NOT read the ADC, the we will miss a lot of samples.
Probably, using interruptions, we could be able to avoid this problem but it requires a lot of time and thinking to ensure we will not miss any data when attending the interruption. 
But for now, I share this information, I think is very useful if you want to save data at low sampling rate, around 30 samples per minute and lower I think is very useful.
Just modify the program according to your requirements and enjoy saving a lot of data! :-)
NOTE: I´m using the Bluetooth to send data to the PC in real time but just for debugging purposes. You can avoid that or use it also as a redundant system to save data in the PC, The BT module communicates with the PIC using the general RS232 protocol, it means you can substitute the BT module with a RS232 cable.
I highly recommend you to visit the forum where the original libraries are and to read more about the SD card and FAT16 themes.