Java-Gaming.org    
Featured games (79)
games approved by the League of Dukes
Games in Showcase (477)
Games in Android Showcase (107)
games submitted by our members
Games in WIP (535)
games currently in development
News: Read the Java Gaming Resources, or peek at the official Java tutorials
 
    Home     Help   Search   Login   Register   
Pages: [1]
  ignore  |  Print  
  java.lang.Runtime.exec and Process == hang  (Read 1786 times)
0 Members and 1 Guest are viewing this topic.
Offline blahblahblahh

JGO Coder


Medals: 1


http://t-machine.org


« Posted 2004-12-12 09:17:25 »

Neat program called HTML Tidy, been around for years, open source (http://tidy.sf.net). Works perfectly: you pump in an HTML file to stdin, it prints out a "tidied" version on stdout.

So, I use this:
1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
19  
20  
21  
22  
23  
24  
25  
26  
27  
28  
29  
/*
* Launch the external process, a single-file linux binary sitting in the current directory
*/


      Process p = Runtime.getRuntime().exec( new String[] { "./tidy", "-i", "--doctype", "omit", "--show-body-only", "true", "-f", "htmltidy-error.log" } );

/*
 * attach to the stdin, and send the HTML file to Tidy
 */

      PrintWriter htmlTidyWriter = new PrintWriter( p.getOutputStream() );
      htmlTidyWriter.print( fragment );
      htmlTidyWriter.flush();
      htmlTidyWriter.close();
     
/*
 * attach to stdout, and read the result back from Tidy
 */
           
      logger.info( "Reading data from external linux process = \"tidy\"");
     
      BufferedReader br = new BufferedReader( new InputStreamReader( p.getInputStream() ) );
      String line = null;
      while( (line = br.readLine()) != null )
      {
            logger.debug( "read line from Tidy: "+line );
            result.append( line+"\n" );
      }
      br.close();
     
      logger.info( "Completed discussion with external linux process = \"tidy\"; kill'ing it since java has no wait-for-exit command");


And, when run locally, it *always* works.

But...when run on the JGF server (same OS, almost exactly the same kernel, same software installed, same JVM - 1.4.2_05), most of the time it hangs. And...it hangs on the first attempted read from the BufferedReader.

i.e. most of the time, either the JVM is failing to launch this process, or the process itself is hanging. Given how widely used HTMLTidy is, and how mature, I find it hard to believe that anyone else experiences the same problem Smiley.

In the API docs, it mentions something about how the process could deadlock if you read/write to it at the wrong times. This seems strange to me - block, sure. Deadlock? Shocked. Maybe that's what's happneing? Especially given it never happens locally but often (not always) happens remotely (i.e. it's probably a race condition of some sort)...

Other than that I'm stumped. And it's kind of hard to put out a piece of software and say "hey, it hangs most of the time, but I have no idea why. Just keep trying and it will work eventually".

PS: no, I can't use the so-called java port: the maintainers stopped maintaining it *3 years ago* and it lacks most of the features of the real HTMLTidy, certainly at least 3 I cannot do without. Sob.

malloc will be first against the wall when the revolution comes...
Offline hawkwind

Junior Member




Java games rock!


« Reply #1 - Posted 2004-12-12 17:55:48 »

I would mod the script to handle the error out stream of the proces also.  I have had Process hang when its dumping to both the output and th error stream and I wasn't reading both.
Offline blahblahblahh

JGO Coder


Medals: 1


http://t-machine.org


« Reply #2 - Posted 2004-12-12 18:11:24 »

Good point. However...

The official docs for "tidy" are that nothing will be output to stderr if you divert errors to a log file (which are what the last two parameters do).

I've verified that it is indeed writing it's errors to that error file, and it does this on every launch.

When it hangs, and when it doesn't, is on the exact same input data, and all input data has at least one error - yet sometimes it Just Works. So, I fear that "not reading from stderr" isn't the cause of this problem Sad.

malloc will be first against the wall when the revolution comes...
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline blahblahblahh

JGO Coder


Medals: 1


http://t-machine.org


« Reply #3 - Posted 2004-12-13 19:08:38 »

Crap.

Seems that Runtime.exec(...) is basically just buggy and unloved by Sun, with outstanding unfixed bugs going back as far as 1.1.x:

http://bugs.sun.com/bugdatabase/view_bug.do;:WuuT?bug_id=4103432

c.f. the comment on  xxxxx@xxxxx  2004-03-29:
1  
2  
3  
4  
5  
6  
The evaulator states tthat the program "test" should work. So it seems
this bug report shouldn't be closed as "not a bug" since the original and
main focus of the bug report is the program "test" and that is still
reproducible with JDK 1.5 beta2 (build 43) on Windows 98/SE. It hangs there.
But as the previous evaluator notes, it exits properly on Windows 2000 and XP.
This bug was the cause of 5007388 which also manifests only on windows 9X.


But...some digging on the net threw up that other people were finding the precise same code on java 1.4.x will "never hang with server VM, and randomly hang with client VM", so...I'm off to check whether I get a difference with different VMs.

malloc will be first against the wall when the revolution comes...
Offline jbanes

JGO Coder


Projects: 1


"Java Games? Incredible! Mr. Incredible, that is!"


« Reply #4 - Posted 2004-12-13 19:12:10 »

Just an odd thought:

http://jtidy.sourceforge.net/

Java Game Console Project
Last Journal Entry: 12/17/04
Offline blahblahblahh

JGO Coder


Medals: 1


http://t-machine.org


« Reply #5 - Posted 2004-12-13 19:20:52 »

Quote
PS: no, I can't use the so-called java port: the maintainers stopped maintaining it *3 years ago* and it lacks most of the features of the real HTMLTidy, certainly at least 3 I cannot do without. Sob.


I tested it. The most important of the config options it outputs from it's own built-in help DON'T ACTUALLY WORK, and generate errors telling you to see the help for supported options. Stupid, stupid, stupid.

malloc will be first against the wall when the revolution comes...
Offline jbanes

JGO Coder


Projects: 1


"Java Games? Incredible! Mr. Incredible, that is!"


« Reply #6 - Posted 2004-12-13 19:37:47 »

Sorry, I missed your "P.S." clause.  Tongue

FYI, while the last stable release was three years ago, they've nearly got a new release finished. Go to the homepage, and you should find links to their nightly builds. You may find that the nightlies greatly improve upon that three-year-old version. :-)

Java Game Console Project
Last Journal Entry: 12/17/04
Offline blahblahblahh

JGO Coder


Medals: 1


http://t-machine.org


« Reply #7 - Posted 2004-12-13 23:02:03 »

Quote
Sorry, I missed your "P.S." clause.  Tongue

FYI, while the last stable release was three years ago, they've nearly got a new release finished. Go to the homepage, and you should find links to their nightly builds. You may find that the nightlies greatly improve upon that three-year-old version. :-)


/me drools

thanks for that, although I looked around extenstively, through the forums etc, all I found were long threads from 2003 lamenting the fact that it was no longer maintained.

Looked dead as a dodo. But, clearly, I need to re-examine.

malloc will be first against the wall when the revolution comes...
Offline blahblahblahh

JGO Coder


Medals: 1


http://t-machine.org


« Reply #8 - Posted 2004-12-14 06:34:30 »

If the unit tests are to be believed, there are only two bugs that make it unusable.

Unfortunately, one of these is that it deletes the content from textareas - which is rather a big problem for a CMS, where almost every page has a textarea Sad.

malloc will be first against the wall when the revolution comes...
Offline jbanes

JGO Coder


Projects: 1


"Java Games? Incredible! Mr. Incredible, that is!"


« Reply #9 - Posted 2004-12-14 13:19:04 »

Quote
Unfortunately, one of these is that it deletes the content from textareas - which is rather a big problem for a CMS, where almost every page has a textarea Sad.


Q: Is the CMS itself going to be tidied on the fly, or is it the CMS *content* that is to be tidied? If it's the later, is this actually an issue? If it's the former, then perhaps you could JavaScript around it. e.g.:

1  
2  
3  
4  
5  
<script>
    var text="Lots of CMS text right here, for your viewing pleasure!";
    
    document.forms[0].mytextbox.value = text;
</script>


Alternatively, you could dive into the JTidy code and fix the bug yourself. :-)

Java Game Console Project
Last Journal Entry: 12/17/04
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline blahblahblahh

JGO Coder


Medals: 1


http://t-machine.org


« Reply #10 - Posted 2004-12-14 13:46:01 »

Quote


Q: Is the CMS itself going to be tidied on the fly, or is it the CMS *content* that is to be tidied?


Good question. It's just the content that's being tidied at the moment (although, eventually, when online editing of the CMS code goes live we'll obviously want to tidy that too - for pure convenience of makign it easier to edit! At least I, personally, am not going to edit raw HTML templates without some tidying Wink).

So, you're right: the textarea bug is not an issue right now. Gah. Stressed, tired. I'm making stupid mistakes!

Anyway, FYI, tidying is currently only necessary for these use-cases:

1. User who tries to insert malicious HTTP code; need to prune or nullify nasty tags
2. User who tries to insert undesirable but non malicious HTTP code; use of IMG or A HREF tags in places where the only possible rason for them is to promote spam etc

...in both these cases, the tidying can be intensive. In both these cases, the check is necessary "once only", but the output will be read thousands if not millions of times.

Hence, check *must* be on the server, and needs to be prior to storing in DB. This assumes that pages are read a lot more often than they are modified (which seems fair)

3. Nice user who is crap at writing HTML
4. Nice user who copies/pastes HTML from a word document

...only difference is that it would be "safe" to do the check on the client browser since these cases the user does not want to circumvent it

5. Nice users who have javascript disabled or broken

...we're back to cases 1 and 2 again, as far as "where" the tidying can be done

---

So, tidying needs to be:
- server-side
- at form-submit time
- good at removing nasties

At the moment, we have a custom set of filters written using REGEXPs (because they're so damn good at filtering nasty HTML; very easy to cover all the random deliberate attempts to fool HTML parsers WITHOUT having to construct full HTML parse trees) and attempting to run Tidy as a final-stage process that does most of the "co-ercing into well-formed HTML, using only modern tags, and undoind any Word/etc nastiness".

Although Tidy does the full tree parse I don't trust the security of the HTML to it. Yet. Maybe I will in the future; at any rate, the regexp security parsing was added before starting with Tidy, so we might as well keep it for now.

malloc will be first against the wall when the revolution comes...
Pages: [1]
  ignore  |  Print  
 
 
You cannot reply to this message, because it is very, very old.

 

Add your game by posting it in the WIP section,
or publish it in Showcase.

The first screenshot will be displayed as a thumbnail.

E.R. Fleming (20 views)
2014-07-29 03:07:13

E.R. Fleming (8 views)
2014-07-29 03:06:25

pw (39 views)
2014-07-24 01:59:36

Riven (39 views)
2014-07-23 21:16:32

Riven (26 views)
2014-07-23 21:07:15

Riven (28 views)
2014-07-23 20:56:16

ctomni231 (59 views)
2014-07-18 06:55:21

Zero Volt (50 views)
2014-07-17 23:47:54

danieldean (42 views)
2014-07-17 23:41:23

MustardPeter (45 views)
2014-07-16 23:30:00
HotSpot Options
by dleskov
2014-07-08 03:59:08

Java and Game Development Tutorials
by SwordsMiner
2014-06-14 00:58:24

Java and Game Development Tutorials
by SwordsMiner
2014-06-14 00:47:22

How do I start Java Game Development?
by ra4king
2014-05-17 11:13:37

HotSpot Options
by Roquen
2014-05-15 09:59:54

HotSpot Options
by Roquen
2014-05-06 15:03:10

Escape Analysis
by Roquen
2014-04-29 22:16:43

Experimental Toys
by Roquen
2014-04-28 13:24:22
java-gaming.org is not responsible for the content posted by its members, including references to external websites, and other references that may or may not have a relation with our primarily gaming and game production oriented community. inquiries and complaints can be sent via email to the info‑account of the company managing the website of java‑gaming.org
Powered by MySQL Powered by PHP Powered by SMF 1.1.18 | SMF © 2013, Simple Machines | Managed by Enhanced Four Valid XHTML 1.0! Valid CSS!