Deciphering the SHP format...

[MarkSheppard]

I have some of fred Chlanda's old notes that came packed with SHPEd, plus the source code for SHPEd v0.50 (see attached).

Here are my notes:

Main File Header (Offset 0) (4 Bytes)
The SHP file itself starts with four bytes that are always "1.10".

SHP Image Count (Offset 4) (4 Bytes)
4 Byte variable giving the number of SHP images in the file.

Table List (Begins at Offset 8)

This lists the address(es) where each image begins and the address of the palette information for that image (this part is generally ignored.)

Essentially, it is a repeating table of two values:

SHP Offset (four byte long integer)
SHP Palette Offset (four byte long integer)

If the SHP file has 10 images, there would be 10 sets of the above, for 20 total values.

Here's an example from a SHP:

256
0
7930
0
10757
0
10853
0

Basically, SHP #1 begins at offset 256, SHP #2 at offset 7930, etc.

The Table List offsets that concern us for Steel Panthers are:

Image #1 Address Offset: 8
Image #2 Address Offset: 16
Image #3 Address Offset: 24
Image #4 Address Offset: 32
…
etc.

A simple equation to calculate where to get the offset from is this:

8 * ImageNumber = Offset Location

Thus, Image 35 would be 8 * 35 = 280 offset.

Image Information (Begins at offset indicated in table for that slot)

Each SHP image begins with a header which is 24 bytes long and stores various parameters such as the image size.

The header is uncompressed and is arranged in the following order:

Height (aka lines) (two byte integer) (this is really the number of lines -1)
Width (two byte integer)
var1 (four byte long integer)
xstart (four byte long integer)
ystart (four byte long integer)
xend (four byte long integer)
yend (four byte long integer)

The header is followed by the actual image data. These headers are very easy “tells” by which to locate SHP data; especially if all the icons in a SHP file are the same standard size: 88 x 88 – then you'd look for the following sequence that indicates the start of a new SHP Icon+Header: 88 0 88 0

*************

This is where I get to the tricky part. SHP raw image data is stored in a compressed format (I think it's Run Length Encoding); same as most other data in Steel Panthers, going back to SP1 itself.

Fred's SHP Decompression/Image Write algorithm in Main.cpp is:

//-----------------------------------------------------------
// read the shp data (for icon n) into the pseudo image
void __fastcall read_shp(FILE *inf, int n)
{
int l; // first line the counter
int lf; // last line
int ch,b,r,i;


fseek(inf,icon_add[n]+HEADER_SIZE,0);
if (header.ystart<0)
{
// a trial
// lf=header.yend+(header.var1>>16);
// l=header.ystart+(header.var1>>16);
l=0; lf=header.yend+abs(header.ystart);
}
else
{
l=header.ystart;
lf=header.yend;
}
if (header.xstart<0)
// a trial
// pix_pos=header.xstart+(header.var1&0xFFF);
pix_pos=0;
else
pix_pos=header.xstart;
do
{
// skip to xstart
// for (i=0; i // put_pix(l,BACK_COLOR);

// read data and decode
ch=fgetc(inf);
r=ch%2;
b=ch/2;
if (b==0 && r==1) // a skip over
{
ch=fgetc(inf);
for (i=0; i put_pix(l,BACK_COLOR);
}
else if (b==0) // end of line
{
++l;
if (header.xstart<0)
{
pix_pos=0;
// a trial
// pix_pos=header.xstart+(header.var1&0xFFF);
}
else pix_pos=header.xstart;
}
else if (r==0) // a run of bytes
{
ch=fgetc(inf); // the color #
for (i=0; i put_pix(l,ch);
}
else // b!0 and r==1 ... read the next b bytes as color #'s
{
for (i=0; i {
ch=fgetc(inf);
put_pix(l,ch);
}
}
} while (l<=lf);
}

I think it works by:

Gets one CHARACTER from the raw SHP file.

Subjects that CHARACTER to a series of tests:

R = CHARACTER Modulus of 2.
B = CHARACTER divided by 2.

Then runs a series of logical tests:

If B = 0 AND R = 1: (Paint one pixel the background color.)

ELSE IF B = 0: (End of that Line of pixels?)

ELSE IF R = 0: (Single Pixel paint)

It gets a bit confusing past that point, as I'm not familiar with C++ and Fred nested things a bit deeply...

[MarkSheppard]

I think the decompression code works out as this:

Read Character

R = Character % 2
B = Character / 2

###########

EOF Test:
If Character EQUAL TO -1 break, this is End of SHP.

##########

TEST ONE:

IF B equals 0 AND R equals 1
(I think Char = 1, which returns B = 0.5 and R = 1)

We have a huge string of blank pixels coming up.

Read the next character in the file. That character is the number of pixels that we paint as the transparent background color.

############

TEST TWO:

If B equals 0

End of SHP Line -- I think this used to mark the end of each row in the SHP file, i.e. incremeting from y = 0 to y =1 and so on.

###########

TEST THREE:

IF R equals 0

Read the next character in the file; that character is the color index number from the palette.

We then paint a number of pixels equal to "B" with that color we just grabbed.

##############

TEST FOUR:

IF B NOT ZERO and R EQUAL ONE

Read the next [b] bytes as color numbers?

Basically this says that the next B characters are direct image colors; probably used to save space when you have a lot of different characters, to avoid the use of space wasting couplets.

//---------------------------------------------------------------------------
// read the shp data (for icon n) into the pseudo image
// this does not use the header info for lines, but reads
// until it finds KEY BYTE 88 Next byte 0.
void __fastcall read_SPicon_shp(FILE *inf, int n)
{
   int ch,b,r,i,l=0;
   long fpos;

   fseek(inf,icon_add[n]+HEADER_SIZE,0);
   pix_pos=0;
   far_right=0;
   far_bottom=0;
   do
   {
     // read data  and decode
     ch=fgetc(inf);

     //==== this section checks for end =====
     if (ch==-1) break;// for last image this is end of shp
     fgetpos(inf,&fpos);
     if ((n<(icons-1)) && fpos>icon_add[n+1]) break;
     // ==== end of end check ===========
     r=ch%2;
     b=ch/2;
	 
     if (b==0 && r==1) // a skip over
     {
        ch=fgetc(inf);
        for (i=0; ifar_right) far_right=pix_pos;
     }
	 
     else if (b==0)   // end of line
     {
       ++l;
       pix_pos=0;
     }
     else if (r==0) // a run of bytes
     {
       ch=fgetc(inf); // the color #
       for (i=0; ifar_right) far_right=pix_pos;
     }
     else // b!0 and r==1 ... read the next b bytes as color #'s
     {
       for (i=0; ifar_right) far_right=pix_pos;
     }
   } while (true); //exit this loop with break
   far_bottom=l;
}