Java-Gaming.org    
Featured games (81)
games approved by the League of Dukes
Games in Showcase (495)
Games in Android Showcase (114)
games submitted by our members
Games in WIP (563)
games currently in development
News: Read the Java Gaming Resources, or peek at the official Java tutorials
 
    Home     Help   Search   Login   Register   
Pages: [1]
  ignore  |  Print  
  Download Cache and Corruption with Applets  (Read 2878 times)
0 Members and 1 Guest are viewing this topic.
Offline kappa
« League of Dukes »

JGO Kernel


Medals: 77
Projects: 15


★★★★★


« Posted 2009-09-09 15:15:19 »

I am downloading files with a Java Applet and running into problems with caching and file corruption.

I download files using a URLConnection by opening an InputStream to it, something like this

1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
urlconnection = url.openConnection();
         
InputStream inputstream = getInputStream(currentFile, urlconnection);
FileOutputStream fos = new FileOutputStream(path + currentFile);
     
int bufferSize;
byte buffer[] = new byte[65536];
         
while ((bufferSize = inputstream.read(buffer, 0, buffer.length)) != -1) {
   fos.write(buffer, 0, bufferSize);
   ...
}
         
inputstream.close();
fos.close();



On the rare occasion I download a file it turns out corrupt, I understand this is normal? (or is it that the buffer size is too big or something?), anyway what is annoying me is that on refreshing the applet the file should redownload right? but the file keeps turning out corrupt until the java cache is cleared from the java control panel. I've tried setting urlconnection.setUseCaches(false); and urlconnection.setDefaultUseCaches(false); but the problem still occurs, It still keeps downloading a corrupt file and only goes away when the java cache is cleared from the control panel. This leads me to believe that the file is still cached somewhere and is used on refresh instead of redownloading, it does seem like its redownloading though.

I am all out of idea's now and not sure how or where its managing to cache the corrupt file (inputstream? browser cache?), any ideas or suggestions welcome.

thanks.
Offline ddyer

Senior Member


Medals: 5



« Reply #1 - Posted 2009-09-09 16:48:03 »

Browser caches are a plague if your content is changing.   You ought to
be able to defeat caches for individual files by adding meaningless arguments
to the URL to make it unique.  For example file.txt?attempt=2

In general, updating content reliably is very difficult - there is no file
system operation that will replace file "a" with file "b" as an atomic
operation, such that no outside party will ever see an intermediate
state.  So just replacing a file, however carefully, is not adequate.
One way to deal with this is to use a file reading proxy instead of
direct file system access = url=file-reader.cgi?file=xxxx, but be
careful you don't create a url that will copy any file!   The file reader
proxy doesn'thave to read any actual files - it can generate the appropriate
content on the fly from a database or whatever.

Also, when I update my java classes, I never, ever, update them in place.
I create a new root, and create a complete new hierarchy with the modified
content.  This is the only way to be sure that no browser sees and caches
a partically updated hierarchy.
Online Riven
« League of Dukes »

JGO Overlord


Medals: 798
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #2 - Posted 2009-09-09 23:07:18 »

Quote
On the rare occasion I download a file it turns out corrupt, I understand this is normal?

This is definitely NOT normal. How often does a download in your browser fail?? Once a year?

How often does a Java download fail? Every other day? I think this is the root cause of all JavaWebStart problems. I'm starting to get more and more convinced that URLConnection is bugged. I should dive into the code some day.

Some time ago, I had this applet that loaded a bunch of thumbnails from a server. It was quite normal that a few wouldn't get loaded.


So... on to the problem:

HTTP isn't that hard. Just do your own HTTP handling. Add support for "Transfer-Encoding: chunked" and you instantly support 99.99% of all webservers. Cache nothing. Worked for me. The guarantee that all bugs are in *your* code is worth gold.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline DzzD
« Reply #3 - Posted 2009-09-10 00:41:45 »

This is definitely NOT normal. How often does a download in your browser fail?? Once a year?

How often does a Java download fail? Every other day? I think this is the root cause of all JavaWebStart problems. I'm starting to get more and more convinced that URLConnection is bugged. I should dive into the code some day.

Some time ago, I had this applet that loaded a bunch of thumbnails from a server. It was quite normal that a few wouldn't get loaded.


So... on to the problem:

HTTP isn't that hard. Just do your own HTTP handling. Add support for "Transfer-Encoding: chunked" and you instantly support 99.99% of all webservers. Cache nothing. Worked for me. The guarantee that all bugs are in *your* code is worth gold.

Absolutly right URLConnection is Buuugged and.... act differently on different JVM ... borring....


Unfortunatly it have a usefull feature that cannot be made using a TCP socket, it is allowed to read browser configuration and especially proxy setting including login, password and IP.

You can go througt HTTP proxy with TCP socket by using Proxy-Authorization header but you will have to ask the user to enter it login/password and the proxy IP,  but finally that may not be a big issue.


if you dont want to implement chunked exchange (or other complexe/heavy HTTP 1.1) features you can make your request with HTTP/1.0 rather than HTTP/1.1


but... another but.... there are several possible issue when implementing HTTP protocole partially, you should especially take care of the following :

Request headers for HTTP 1.1 :
Host (requiered for 1.1 request)

Response code for both 1.0 / 1.1:
407 : proxy authentication requiered
307/301 : moved => you must do another call to the returned URI/location


I have to deal with Applet/Server communication at work, I choose to make my own HTTP implementation using Socket as mentioned by Riven (the main first reason was a bugged keep-alive implementation on URLConnection), but my Class will keep a fall back to URLConnection and switch to it in case of something go wrong with the direct TCP Socket connection.


finnaly I would recommend the following : connect using your home made HTTP connector and if it fail retry with URLConnection


Offline kappa
« League of Dukes »

JGO Kernel


Medals: 77
Projects: 15


★★★★★


« Reply #4 - Posted 2009-09-11 00:24:47 »

thx for the suggestions.

I'm not really sure a custom http connector is really possible here, since I'm limited to a single class and gotta keep the size small, roughly how much code are we talking about here to implement a custom http connector?

The corruption is a pretty serious issue because from what I've seen so far there is roughly a 1/20 chance that a download will be corrupt, which is pretty high.

Aside from the download becoming corrupt, any idea's where and how the corrupt file is getting cached and stored? If I could at least get the caching to stop it should be good enough for now.
Online Riven
« League of Dukes »

JGO Overlord


Medals: 798
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #5 - Posted 2009-09-11 06:11:23 »

thx for the suggestions.

I'm not really sure a custom http connector is really possible here, since I'm limited to a single class and gotta keep the size small, roughly how much code are we talking about here to implement a custom http connector?


The only problem is that HTTP is both text based any binary. So you can't simply use BufferedReader.readLine() because it will buffer beyond the "\r\n" and screwup your binary content (if any).

So just use your average BufferedInputStream, probably throw a PushbackInputStream into the mix, and wait for that "\r\n" until you read an empty line (end of header). Now parse all header-lines which gives you a clue on how to 'decode' the content.

Maybe a few hundred lines.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Pages: [1]
  ignore  |  Print  
 
 
You cannot reply to this message, because it is very, very old.

 

Add your game by posting it in the WIP section,
or publish it in Showcase.

The first screenshot will be displayed as a thumbnail.

Dwinin (28 views)
2014-09-12 09:08:26

Norakomi (57 views)
2014-09-10 13:57:51

TehJavaDev (75 views)
2014-09-10 06:39:09

Tekkerue (38 views)
2014-09-09 02:24:56

mitcheeb (57 views)
2014-09-08 06:06:29

BurntPizza (45 views)
2014-09-07 01:13:42

Longarmx (28 views)
2014-09-07 01:12:14

Longarmx (34 views)
2014-09-07 01:11:22

Longarmx (35 views)
2014-09-07 01:10:19

mitcheeb (40 views)
2014-09-04 23:08:59
List of Learning Resources
by Longor1996
2014-08-16 10:40:00

List of Learning Resources
by SilverTiger
2014-08-05 19:33:27

Resources for WIP games
by CogWheelz
2014-08-01 16:20:17

Resources for WIP games
by CogWheelz
2014-08-01 16:19:50

List of Learning Resources
by SilverTiger
2014-07-31 16:29:50

List of Learning Resources
by SilverTiger
2014-07-31 16:26:06

List of Learning Resources
by SilverTiger
2014-07-31 11:54:12

HotSpot Options
by dleskov
2014-07-08 01:59:08
java-gaming.org is not responsible for the content posted by its members, including references to external websites, and other references that may or may not have a relation with our primarily gaming and game production oriented community. inquiries and complaints can be sent via email to the info‑account of the company managing the website of java‑gaming.org
Powered by MySQL Powered by PHP Powered by SMF 1.1.18 | SMF © 2013, Simple Machines | Managed by Enhanced Four Valid XHTML 1.0! Valid CSS!