Java-Gaming.org Hi !
Featured games (91)
games approved by the League of Dukes
Games in Showcase (804)
Games in Android Showcase (239)
games submitted by our members
Games in WIP (868)
games currently in development
News: Read the Java Gaming Resources, or peek at the official Java tutorials
 
    Home     Help   Search   Login   Register   
Pages: [1] 2
  ignore  |  Print  
  Unit Testing: Waste Of Time? Discuss  (Read 16422 times)
0 Members and 1 Guest are viewing this topic.
Offline princec

« JGO Spiffy Duke »


Medals: 1146
Projects: 3
Exp: 20 years


Eh? Who? What? ... Me?


« Posted 2015-02-05 12:12:19 »

As the title says.

Apropros unit testing. Is it increasing software quality? It it decreasing time-to-market? Is it having the opposite effects? Is it "best practice" (and therefore doomed to fail 90% of the time)? What are the alternatives? How do they compare? All this and more to discuss in this new monster argumentdiscussion thread!

Cas Smiley

Offline jmguillemette
« Reply #1 - Posted 2015-02-05 12:16:49 »

in the business software i write as a day job unit tests are invaluable.
They provide both reality checks as the projects get bigger or aged... and allow new developers to come in and make changes with the confidence that they have not broken existing functionality.

In most cases we unit test our utility code and interface points..
Can you write unit tests for a games OpenGL interface? i dont know.. would be interested to know if there is a framework out there for it.

Now as a complete confession. In my game im working on .. i have not written a single test yet. Frankly the code is too in flux and im the only developer. So there is likely a reasonable argument that size of team equals greater value from unit tests.

-=Like a post.. give the author a medal!=-
Offline Riven
Administrator

« JGO Overlord »


Medals: 1371
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #2 - Posted 2015-02-05 12:33:30 »

@princec: unit tests are invaluable in LibStruct - but... that's the first project I am forced to use them to check for regressions. There is simply too much that can go wrong.

In all my other projects, I couldn't care less about proper unit tests, let alone decent coverage.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings!
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline CommanderKeith
« Reply #3 - Posted 2015-02-05 12:52:10 »

I've never used unit-testing before, but it sounds useful for those reasons jmguillemette mentioned.
Most of the bugs that I make for myself are due to threading, and apparently unit testing threading problems is very difficult:
http://stackoverflow.com/questions/12159/how-should-i-unit-test-threaded-code

Offline kevglass

« JGO Spiffy Duke »


Medals: 319
Projects: 25
Exp: 22 years


Coder, Trainee Pixel Artist, Game Reviewer


« Reply #4 - Posted 2015-02-05 13:03:04 »

We use on in the day job on a huge enterprise scale. However, we're a rapidly moving software team so the problem is fragility and false negatives. That along with the worth in regression is pretty suspect, looking at the statistics across bugs coming back in, support requests vs the coverage in the code there isn't any correlation between high coverage = low bugs/support.

Cheers,

Kev

Offline princec

« JGO Spiffy Duke »


Medals: 1146
Projects: 3
Exp: 20 years


Eh? Who? What? ... Me?


« Reply #5 - Posted 2015-02-05 13:09:01 »

This was my finding too: no correlation between coverage and reported issues.

What I did discover was a very high correlation between tests written and time it took to get the software released. Unit testing at least doubled time-to-market but found virtually no bugs that trivial QA didn't find and never found bugs that weren't suitable for unit testing (which is most of them).

My conclusion is that TDD is a self-licking lollipop which falls into the category "best practices" (this is worth a read). It makes managers look like they've got their arses covered; it makes developers look like they're busy; it makes all the right "noises". What it doesn't seem to be doing is getting software out quicker, or particularly any higher quality than by any other means.

What those other means might be I leave to further commentary Smiley

@Riven - there's a time and a place for unit tests and I think you'd be one of those who knows when and where they're needed Wink

Cas Smiley

Offline BurntPizza

« JGO Bitwise Duke »


Medals: 486
Exp: 7 years



« Reply #6 - Posted 2015-02-05 13:20:58 »

I think unit testing suddenly becomes important in a lot of cases when using dynamic languages, which are popular, thus leading to popularity of practices like TDD etc. Here's one such post: http://williamedwardscoder.tumblr.com/post/54327549368/dynamic-languages-are-unmaintainable-and-unit

I agree that it quickly becomes much less useful the more the language itself checks everything for you, disallows you from doing a lot of things that don't make sense, and provides inline documentation of itself.
Offline Roquen

JGO Kernel


Medals: 518



« Reply #7 - Posted 2015-02-05 13:30:55 »

I write test units (like things) in one and exactly one situation and never than other.  That situation generalizes to code that writes code.
Offline kappa
« League of Dukes »

JGO Kernel


Medals: 123
Projects: 15


★★★★★


« Reply #8 - Posted 2015-02-05 13:57:18 »

Another argument for techniques like TDD which is usually pushed is that they help you design cleaner and better public API's but personally not seen any advantage here over manually doing so (in fact its faster to do so manually).

When it comes to writing games (as opposed to libraries and engine code), unit testing is a complete waste of time. Jonathan Blow sums up the problem pretty nicely (including arguing pretty strongly against spending time commenting code and most types of optimisations).
Offline Stranger
« Reply #9 - Posted 2015-02-05 14:00:04 »

Frankly, I often forget what is the difference between unit and functional tests, so I have to recall that permanently... Undecided Smiley  Can anybody also mention about mock frameworks. Nice thing in the unit testing, isn't it?

Anton
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline princec

« JGO Spiffy Duke »


Medals: 1146
Projects: 3
Exp: 20 years


Eh? Who? What? ... Me?


« Reply #10 - Posted 2015-02-05 14:11:32 »

I've seen one area where it has a vague positive, which is that if you're doing TDD effectively, you'll have a separate programmer writing the tests against the APIs he's given. Most code instantly fails at the first hurdle by simply having no specification or documentation to speak of, and the tester can simply throw it back and say, "I can't test this as I don't know what you think it's supposed to do."

Of course, largely totally irrelevant outside of fully reusable, releasable, versionable library code... which is most code.

Cas Smiley

Offline divxdede
« Reply #11 - Posted 2015-02-05 18:44:55 »

You can use "unit tests" without going completely in a TDD process.
I try to have a pragmatic approach in my job or for personnal projects that is the following:
  - Wrote unit test during the developpement process for parts that i found critical of difficult.
 
Critical because i haven't the right to introduce a bug
Difficult because i will have difficulty to enhance theses parts without taking some risks to break something.

In this stage, i not write a billions of unit tests and i try to keep reasonnable.

After this developpement process, become the maintenance time when a bug is found.
When a bug is found in production, i consider we never see it in all stages we can have (dev, internal test, client test)
So it's not a "simple bug", and in this case, we wrote a unit test that can check this precise bug that protect us to have it again.

With this approach, i can reduce costs of unit tests (in term of time and effort) and trying to maximize theses benefits.

Best regards,
Séb.




Offline ags1

JGO Kernel


Medals: 367
Projects: 7


Make code not war!


« Reply #12 - Posted 2015-02-05 19:49:58 »

I find unit tests helpful in my day job where I build frameworks or utilities for other people. The other side of my job is developer support and we have tests there too, to ensure our samples still work with the latest server code and API changes (which are always "backwards compatible"). Hell, I even have some unit tests for my user guides, just to check the code snippets compile and run without errors.

I find it very hard to write unit tests at home. In my games I have almost no unit tests, except for set pieces like AI, collision detection and so on.

Offline EgonOlsen
« Reply #13 - Posted 2015-02-05 19:57:57 »

I'm doing a lot of server side development for my day job that usually runs as a kind of job or web service or is used by others to access some remote data. For these things, i write unit tests to see if my code does what it's supposed to do during development and they also help others to understand how to use that stuff. You could easily accomplish this with some test classes with simple main-methods as well. But using unit tests instead makes it easier for others to identify them in the project.
But it almost never happens that they unveil some bug that otherwise would have gone unnoticed...well, except for today, but that doesn't count... Wink TDD as a pure doctrine is rather pointless IMHO.

Offline Roquen

JGO Kernel


Medals: 518



« Reply #14 - Posted 2015-02-05 20:26:43 »

See: white box vs. black box testing.  White box is good.  Black box is usually a waste of time.
Offline HeroesGraveDev

JGO Kernel


Medals: 383
Projects: 11
Exp: 4 years


┬─┬ノ(ಠ_ಠノ)(╯°□°)╯︵ ┻━┻


« Reply #15 - Posted 2015-02-05 21:47:53 »

The Rust language relies very heavily on unit and integration testing. Countless bugs have been caught and fixed before being merged into the master branch. Because of this it lowers the barrier for contribution considerably, as your changes can be easily verified without wasting enyone's time. The same bug very rarely happens twice, and when it does you can be sure it's never the same cause (because regression tests).

In fact, unit testing, integration testing, and benchmarking functionality has been well-defined and built into the compiler to avoid the bloat and fragmentation among external frameworks  (and it also allows tests to be written alongside implementations without making a mess). Even code examples in the documentation are built and run to make sure that it all stays up to date.

I'd say it's a very important practice in large open source projects to lower the barrier for contributions, but for small projects with only a handful of people who understand most of the system, it won't catch enough bugs to be worth the time it takes to write all the tests.

Offline princec

« JGO Spiffy Duke »


Medals: 1146
Projects: 3
Exp: 20 years


Eh? Who? What? ... Me?


« Reply #16 - Posted 2015-02-05 22:54:00 »

... or even rewrite the tests.

IME tests take about as long as the original code to develop... and maintain. Is that really worth it?

Cas Smiley

Offline Riven
Administrator

« JGO Overlord »


Medals: 1371
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #17 - Posted 2015-02-05 22:55:09 »

In environments where regressions cost thousands or millions, I think I'd unit-test every nook and cranny.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings!
Offline princec

« JGO Spiffy Duke »


Medals: 1146
Projects: 3
Exp: 20 years


Eh? Who? What? ... Me?


« Reply #18 - Posted 2015-02-05 23:04:41 »

I think I'd be using a language it was harder to make mistakes in, too Smiley

Cas Smiley

Offline Riven
Administrator

« JGO Overlord »


Medals: 1371
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #19 - Posted 2015-02-05 23:21:53 »

I used to work for small companies that tested their code by clicking though a web UI, filling out forms, observing what happened. Slowly but surely, almost every piece of business logic (at least subtily) broke.
It involved administration of the healthcare of about a million people, so it... mattered, although lives where not at stake. Ever since I don't quite take the reasoning seriously that unit tests are too much of a burden because they merely catch dirt stupid, easily discovered bugs. These silly bugs won't be discovered until either the app falls apart or some bulk operation went bezerk and did real damage.

Things will always break in complex ways, but that doesn't mean we shouldn't safeguard the simple logic from off by one errors and the like.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings!
Offline princec

« JGO Spiffy Duke »


Medals: 1146
Projects: 3
Exp: 20 years


Eh? Who? What? ... Me?


« Reply #20 - Posted 2015-02-05 23:39:49 »

I'm sure it has its time and place... it seems to me though that it is mainly being used at the wrong time in the wrong places.

Cas Smiley

Offline Riven
Administrator

« JGO Overlord »


Medals: 1371
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #21 - Posted 2015-02-05 23:58:44 »

It isn't much of a discussion if it tends to boil down to 'it depends'.

Having said that - who's trying to convince who of what again ? Smiley


IME tests take about as long as the original code to develop... and maintain. Is that really worth it?
it depends persecutioncomplex

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings!
Offline princec

« JGO Spiffy Duke »


Medals: 1146
Projects: 3
Exp: 20 years


Eh? Who? What? ... Me?


« Reply #22 - Posted 2015-02-06 00:01:43 »

Heh, no convincing, the topic is open discussion Smiley I was interested to hear others' anecdotes and experiences on it. I've seen it working well myself but mostly all I've seen is software that still has as many bugs in it but that takes twice as long to make. I was hoping I was wrong and I've just been unlucky because all the contracts out there are the moment are all "TDD", "Agile", "SCRUMM" etc. (what I would have called "have you actually run it", "just make it work by Friday" and "ordinary meetings with everyone else like we've always done" back in the day)

Cas Smiley

Offline junkdog
« Reply #23 - Posted 2015-02-06 01:35:50 »

I think unit testing suddenly becomes important in a lot of cases when using dynamic languages, which are popular, thus leading to popularity of practices like TDD etc. Here's one such post: http://williamedwardscoder.tumblr.com/post/54327549368/dynamic-languages-are-unmaintainable-and-unit

Also for languages with explicit memory management, unit tests can be pretty powerful when wrapped in a runtime memory analysis tool: detecting uninitialized memory, dangling memory etc.

At work, we have a somewhere around 11k unit tests last time I checked; granted, most are more system/integration tests than plain unit tests. The test environment gets deployed across a range of different configurations (OS, database vendors, application containers, JVM versions etc); it's quite common/annoying that some errors are only  reproducible in very specific environments/configurations.

Heh, many years ago, I wrote a payroll + reports application for small casino (a type of casino-in-bars thing that are quite common here) - it was my second project as a solo developer and took place during the time when I thought unit testing was mostly a tool for consultants to embellish their invoices.

Turns out that when your software is essentially dealing with cash flow, it rapidly spins out of the comfort zone every time something looks quirky. After going live and finding the first couple of bugs, every damn anomaly starts looking suspect.

Once the worst of my paranoia had passed, in parts thanks to not uncovering any really serious bugs (aside from a few embarrassing mistakes), income tax laws changes and I had to incorporate it into the software. Naturally, the application still had to be able to generate reports the old way too. And so began another stretch of weeks with too much stress and anxiety.

The above anecdote is of course on the extreme end of stupidity in regards to unit testing. Don't ask me why I didn't start writing unit tests asap upon realizing the error of my approach, but I didn't,

For code out in the wild, only artemis-odb is unit tested: I started doing it when I cleaned up the bugs from vanilla artemis; once dev on new features begun, I tended to write tests for them too. I wasn't aiming for complete coverage, but I think it's mostly complete. When refactoring, having a semi-good coverage helps a lot - it was pretty much what made some refactoring work possible without messing up (like when changing the way entity groups were identified within the framework).

artemis-odb: bugfixing and performance optimized fork of artemis ECS
Offline Roquen

JGO Kernel


Medals: 518



« Reply #24 - Posted 2015-02-06 06:30:48 »

The Rust language relies very heavily on unit...
Code that writes code.
Offline ryanm

Senior Devvie


Projects: 1
Exp: 15 years


Used to be bleb


« Reply #25 - Posted 2015-02-07 02:59:27 »

Testing software is a good idea - whenever you write a non-trivial bit of code you should write a wee main() to reassure yourself that it does what you think it does.
Writing that main() as a @Test just means that you can keep it around and run it again really easily, which is probably a good thing. As EgonOlsen said, tests can be excellent documentation of intended behaviour - passing tests will never go stale.

Unfortunately it's really easy to measure test coverage and really tricky to measure test quality. Even if there aren't explicit targets for test coverage, it's very easy for devs to fall into coverage-chasing behaviour - the compulsion for 100% coverage essentially gamifies poor-quality tests that only serve to slow down development.

To answer the question posed: It Depends. Are you testing complex things that are easily broken in subtle ways, or are you chasing 100% coverage on the jenkins dashboard?
Offline Stranger
« Reply #26 - Posted 2015-02-07 06:56:45 »

If you'd allow me to dream, then I'd tell about ML. 'Learning from examples' it's just a kind of TDD. Examples of the behavior of the target program are in fact the tests. Thus process of developing the software becomes generate-and-test process. Though, whether these ideas could be ever reified in some real, industrial-strength frameworks? I don't know. Maybe they'll be spread in other forms in some more specific technologies ... Excuse me if I brought a cloud of dust to this discussion Roll Eyes  Smiley.

Anton
Offline ziozio
« Reply #27 - Posted 2015-02-07 08:30:11 »

In principal testing is a great idea but in practise it is quite hard to do to make it really effective

  • You could write tests for every class but you aren't then testing how the application works as a whole
  • Its very hard to create every possible type of input, for a complex system, and to then have something that can test all the possible output. Opengl is a great example of it being very hard to test the output
  • if you don't maintain tests, they can very easily get out of date
  • The effort vs reward ratio is very biased towards effort


Where I work every agrees test cases are a good thing but in practise non are actually written
Offline Roquen

JGO Kernel


Medals: 518



« Reply #28 - Posted 2015-02-07 09:51:22 »

The question has nothing to do with insuring a piece of code is correct to its contract.  That's a different topic entirely. 

Unit-testing is a battery of black-box tests which attempts to insure contracts are fulfilled.  This is pretty much only useful in very narrow situations.  Code that writes code is always one (languages, meta-programming, expressing rewriting, etc.).  Some serious numeric code is another, GCs of any complexity.  The vast majority of code (even dynamic and functional) don't really need it.

The notion that one can prevent bugs by design is a fail.   
Offline princec

« JGO Spiffy Duke »


Medals: 1146
Projects: 3
Exp: 20 years


Eh? Who? What? ... Me?


« Reply #29 - Posted 2015-02-07 10:44:09 »

So I'm wondering who actually started the myth that it was a good thing to be doing, why they concluded that, why everyone has taken up the mantra and repeating it like dogma, and most importantly for software engineers why nobody seems to be questioning it?

Cas Smiley

Pages: [1] 2
  ignore  |  Print  
 
 

 
Riven (581 views)
2019-09-04 15:33:17

hadezbladez (5510 views)
2018-11-16 13:46:03

hadezbladez (2402 views)
2018-11-16 13:41:33

hadezbladez (5772 views)
2018-11-16 13:35:35

hadezbladez (1223 views)
2018-11-16 13:32:03

EgonOlsen (4661 views)
2018-06-10 19:43:48

EgonOlsen (5682 views)
2018-06-10 19:43:44

EgonOlsen (3198 views)
2018-06-10 19:43:20

DesertCoockie (4095 views)
2018-05-13 18:23:11

nelsongames (5115 views)
2018-04-24 18:15:36
A NON-ideal modular configuration for Eclipse with JavaFX
by philfrei
2019-12-19 19:35:12

Java Gaming Resources
by philfrei
2019-05-14 16:15:13

Deployment and Packaging
by philfrei
2019-05-08 15:15:36

Deployment and Packaging
by philfrei
2019-05-08 15:13:34

Deployment and Packaging
by philfrei
2019-02-17 20:25:53

Deployment and Packaging
by mudlee
2018-08-22 18:09:50

Java Gaming Resources
by gouessej
2018-08-22 08:19:41

Deployment and Packaging
by gouessej
2018-08-22 08:04:08
java-gaming.org is not responsible for the content posted by its members, including references to external websites, and other references that may or may not have a relation with our primarily gaming and game production oriented community. inquiries and complaints can be sent via email to the info‑account of the company managing the website of java‑gaming.org
Powered by MySQL Powered by PHP Powered by SMF 1.1.18 | SMF © 2013, Simple Machines | Managed by Enhanced Four Valid XHTML 1.0! Valid CSS!