yeah the data is chunked, if your java code can download chunked transfers than it would definitely help

HTTP/1.1 200 OK
Cache-Control: private, max-age=0
Date: Wed, 22 Jul 2009 18:14:11 GMT
Expires: -1
Content-Type: text/html; charset=ISO-8859-1
Set-Cookie: PREF=ID=4620d9e95c8c2ef6:TM=1248286451:LM=1248286451:S=gjw09cMxpIa8uxrZ; expires=Fri, 22-Jul-2011 18:14:11 GMT; path=/; domain=.google.co.uk
1248286451
Server: gws
Transfer-Encoding: chunked

I suppose this continuation question is still relevant since its about pointers:

How the hell is line nonsensical, i have a pointer and i want to check if the string it points to is EXACTLY the same as "Transfer-Encoding: chunked"

if(*line == "Transfer-Encoding: chunked")

oh an i figured out how to download in chunks now [www.tcpipguide.com]

*line deferences the pointer but *line would equal 'T'. You'd need to strcmp,strncmp, memcmp or loop though checking each char.

Also, just comparing a pointer with a string literal is equally useful as comparing two pointers, where the later is an undefined value.

hmmm, thats interesting, i might just try out a few things on my pc before moving to wii

Quote
SteelSLasher

if(*line == "Transfer-Encoding: chunked")

You would be comparing 'T' to a pointer to wherevever "Transfer-Encoding: chunked" is stored.
Don't use == against string literals

[c-faq.com]

i tried some things out and i cant figure out how to retrieve chunked, i know how they are chunked but using the tcp_readLn() just gives out a load of blank lines and tcp_read() gives fef

[chunk 1 size]
[chunk 1 data]
[chunk 2 size]
[chunk 2 data]
...

i made this small piece of code for testing, i wanted to see if i could get some chunked data to work with but for some reason it just gives "fef" each time

if(chunk)
{
char *line = tcp_readln (s, 0xff, gettime(), HTTP_TIMEOUT);
int i = 0;
for(i=0;i<32;i++)
{
Debugs(line);
}
}

Edited 1 time(s). Last edit at 07/24/2009 01:30PM by SteelSLasher.

i figure out how to remove the transfer-encoding, by downgrading to http 1.0, it may not be upto date but i have to start somewhere. This also means its not my fault if transfer encoding is forced and doesnt work.

Not all servers honor your request when you ask for http1.0, they send http1.1 anyhow. Some even send http1.1 and then don't send chunked encoding or length nor close the connection when all data is recieved. To get 100% compatible with all servers is a hell. But chunked and content-length support are both a must, else you cannot talk to half of the servers out there.

Here is the java code for http downloads I promised: [pastebin.com]

had a look at the java and tbh my java isnt great but i got the sense that it was reading each line and appending the lines with content

this is pseudo code so dont go on about compile errors, i found it here [www.greenbytes.de]

    length := 0
    read chunk-size, chunk-extension (if any) and CRLF \\assuming i just read the next line
    while (chunk-size > 0) { \\if 0 responses ends
       read chunk-data and CRLF \\assuming i read next line
       append chunk-data to entity-body \\quite obvious
       length := length + chunk-size \\to check validity
       read chunk-size and CRLF \\next line
    }
    read entity-header \\chunk encoding includes footers which are like headers but at the end, read next line
    while (entity-header not empty) { \\if empty, end
       append entity-header to existing header fields \\duh...
       read entity-header \\next line
    }
    Content-Length := length \\ummm... cant think of a comment for this one
    Remove "chunked" from Transfer-Encoding \\not sure about this

the pseudo-code is probably better than my interpretation of the java but should be theoretically correct

here is the rerite of the pseudo-code, content_length has already been initialised and used. footers are not taken into consideration, just want to check if this is "alright"

if(chunk)
{
char *html;
int length = 0;
char *line = tcp_readln (s, 0xff, gettime(), HTTP_TIMEOUT);
unsigned int *result = 0;
if(!xtoi(line, result))
{
while(result > 0)
{
length = length + *result;
char *line = tcp_readln (s, 0xff, gettime(), HTTP_TIMEOUT);
strcat ( html, line);
free(line);
line = tcp_readln (s, 0xff, gettime(), HTTP_TIMEOUT);
xtoi(line, result);
}
content_length = length;
}
}

HOLY SHIT!!! I THINK I GET IT NOW!!!

I downloaded a http debugger and found how chunked data is recieved and the http lib is wrong for chunked and needs to be changed (not my coding but bushings coding) so it should be fixed soon