Optimization?

For sharing bugs and usability issues in Outer Colony.
wafehling
Posts: 9
Joined: Sat May 13, 2017 8:54 pm

Optimization?

Postby wafehling » Sun May 14, 2017 4:02 pm

I've been trying my best to give this demo a chance, but my 6600K and 16gb of memory and GTX 970 can't even begin to make this work. The game starts to stutter and freeze within minutes, and within an hour of playing it spends half of its time hanging and freezing. This is on a micro world as well.
User avatar
Administrator
Site Admin
Posts: 315
Joined: Fri Oct 21, 2016 4:05 am
Location: Pennsylvania
Contact:

Re: Optimization?

Postby Administrator » Sun May 14, 2017 5:55 pm

wafehling wrote:I've been trying my best to give this demo a chance, but my 6600K and 16gb of memory and GTX 970 can't even begin to make this work. The game starts to stutter and freeze within minutes, and within an hour of playing it spends half of its time hanging and freezing. This is on a micro world as well.

Hi, wafehling, and thanks for giving Outer Colony a shot. I'm really sorry I wasted an hour of your time, but this post should fix your issue and get you rolling again. Your report here was very useful, and I'm so glad you reported this.

There's a 90% chance that this is being caused by the game's default memory setting. We've been semi-experimentally supporting machines with < 4GB of memory, because some of our players are running on ancient hardware. There was a point at which 2.6GB was sufficient for a micro world, but with recent adjustments to in-game physics, the memory demands have increased. I should probably remove the 2.6GB memory option entirely.

Since you've got powerful hardware at your disposal, try switching to a higher memory setting. Click the Options button on the game's splash screen, then choose the memory option highlighted in this image:
Image

Restart Outer Colony, load into your world, and let me know if this solves your problem. I think that it will.

There's a chance that something else is going on, as worlds sometimes get into unforeseen states where some aspect of the processing framework runs amok. Forum user Torvus ran into a situation where a large pack of predatory whales spawned, and their pathfinding in search of food accidentally overran its limits and caused the world to become unplayable.

We really don't have that many testers as of yet, so I expect more bugs / glitches / processing overruns moving forward. It'll take testing and time to sort them all out. I appreciate the work that early testers, like you, are putting in to help identify these problems.

You might ask yourself, "What's up with these memory modes, and why doesn't Outer Colony just use all the RAM it can?" The short answer is that it works exactly like Minecraft, where you have to specify the memory it uses at launch, and when running, it can't exceed those bounds. This is because both systems are built on Java. As an end user, it's likely you don't care about why, and just find this to be a nuisance. There might be some novel ways to analyze the amount of memory available to a system and to automatically launch Outer Colony in such a way as to take up as much RAM as feasible, but it's not something I've had a chance to toy with.

Also, it's probably idiotic that the default setting remains at 2.6GB when this clearly isn't working for you, as all other new users will have the same experience you've had, without knowing why. I will probably have to get rid of this option in the short term, default the memory mode to 4GB, and just up the system requirements to 6GB of RAM. Sorry for this inconvenience, but it's a huge help that you've reported it. I do all my general testing giving a full 16GB to the JVM, so I wouldn't have noticed this without your post. A new release should be coming shortly.

Thanks again.
User avatar
Administrator
Site Admin
Posts: 315
Joined: Fri Oct 21, 2016 4:05 am
Location: Pennsylvania
Contact:

Re: Optimization?

Postby Administrator » Sun May 14, 2017 6:56 pm

As a quick follow-up, a new version (0.5.47) is now live, and it should correct the issue described here, assuming it was being caused by the memory mode.

I've removed support for the 2.6GB, and the new update should force users to move up to a 4096MB minimum. Forum user Torvus actually encountered this behavior once before, and I wrote it off to perhaps being an artifact of the world he was using, but after your post and some more experimentation, the 2.6GB memory mode just won't work anymore.

I may be able to re-introduce it after another round of optimization, but for now, I'm just increasing the system requirements.

Thanks again for making this post, and I'm cautiously optimistic that increasing the memory mode will solve your immediate issues, as this adjustment worked for another tester.
wafehling
Posts: 9
Joined: Sat May 13, 2017 8:54 pm

Re: Optimization?

Postby wafehling » Sun May 14, 2017 11:25 pm

Afraid it's not that easy. I've had it set to 8gb since starting it, and I did restart it afterwards, so it was authorized to use that much ram. I'm monitoring it in task manager and I can see CPU spikes to rather painful levels any time it stutters (Hitting 60% total CPU usage with 4 cores, ouch). The stutters started happening within seconds of starting a new, micro world, and I can't seem to find out why.

I don't see this kind of stuttering in your tutorial videos, so it looks like some sort of issue on my end. What version of Java are you running for this?
User avatar
Administrator
Site Admin
Posts: 315
Joined: Fri Oct 21, 2016 4:05 am
Location: Pennsylvania
Contact:

Re: Optimization?

Postby Administrator » Mon May 15, 2017 4:23 am

Most intriguing. Something interesting is definitely happening here, and I'm nigh certain that the answer will reveal some kind of flaw in the engine.

wafehling wrote:Afraid it's not that easy. I've had it set to 8gb since starting it, and I did restart it afterwards, so it was authorized to use that much ram.

Right on. That's more than enough memory to run anything other than large worlds with thousands of entities.

I'm monitoring it in task manager and I can see CPU spikes to rather painful levels any time it stutters (Hitting 60% total CPU usage with 4 cores, ouch).

60% CPU usage sounds decidedly high for a micro world with only a few colonists, but spikes like that can happen as you start getting more entities or entities start hitting situations that require more or faster decision making. If two dozen colonists are locked in combat, and they all submit pathfinding jobs and replan activities due to a significant change in their environmental awareness, you will see momentary processing spikes, distributed across all CPUs.

You've just got the 8 starting colonists though, right? And there's no combat going on when you noticed the 60% load across cores?

The thing is, even constant 60% load shouldn't cause any serious stuttering. This is what the multithreaded engine is all about. Rendering is isolated from all other processing, and it's just an asynchronous job like any other. So long as you've got 1 core with 40% of its processing capacity available, that should be more than enough to keep the rendering jobs flowing without any decrease in FPS.

Do you know how long the stutters last? Are they less than 1 second in duration, or more than that? I wonder if some rogue job is allocating way too many objects and causing STW garbage collections. This would definitely cause the behavior you've described.

I don't see this kind of stuttering in your tutorial videos, so it looks like some sort of issue on my end.

Precisely. The behavior you're experiencing is atypical, especially given the hardware at your disposal. Your PC should dominate Outer Colony, and the fact that it doesn't is indicative of a problem with my software. It's just that your current situation, be it the world or some aspect of your system, is bringing it to light.

I've got a few medium-sized test worlds that I use for stress testing purposes where I've got ~300 colonists, plus satellite expeditions with populations in the 20s, plus scores of creatures and other entities roaming about, and it all runs without any appreciable stuttering.

We've got a handful of other active testers who have much lesser machines than yours, and they've been doing alright on small and micro worlds with populations in the dozens.

The main point, though, is that there's almost certainly some kind of bug here in Outer Colony, and it's only surfaced (so far) on your setup. Out of curiosity, are you running Windows 8? I have a Windows 8 laptop that I ran Outer Colony on a couple of times last year, and it inexplicably ran poorly. I was so deep in developing other things that I never looked back at it to figure out the issue.

What version of Java are you running for this?

Outer Colony bundles its own Java setup along with it, and it should be running 1.8.0, update 101. I should probably update to the latest release, but I've been doing all my testing against this update for ages, and I didn't want to destabilize things right before trying to expand our tester group.

Would you mind if I asked a few more questions about your setup? This is all invaluable help you're providing, so serious thanks again.
wafehling
Posts: 9
Joined: Sat May 13, 2017 8:54 pm

Re: Optimization?

Postby wafehling » Mon May 15, 2017 5:31 am

The spikes occur before I even land my colonists, but yeah. After landing them they get gradually worse. There's no combat going on at all, no.

Yeah, that's the confusing part, the stuttering appears in the CPU, but doesn't seem like it should be happening with the extra power available to it.

The stutters start out small, about 1/2 a second every few seconds, but as I start to set up a colony they get longer and longer, at least several seconds.

Ask away.
User avatar
Administrator
Site Admin
Posts: 315
Joined: Fri Oct 21, 2016 4:05 am
Location: Pennsylvania
Contact:

Re: Optimization?

Postby Administrator » Mon May 15, 2017 4:21 pm

Alright, let's see if we can figure this out.

My first order of business was to try to reproduce the stuttering from scratch, so I created a totally new, micro world, landed on it, and played for a few minutes. I did the normal starting-out things, setting up shelters, deploying machinery, digging a mine shaft, logging, planting, etc. I put this together:
Image

And here's a view of the world map, showing that it's pretty typical. An ocean off to the west, terrain rising to the east, a variety of biomes with different kinds of creatures, and no fresh water on this one:
Image

After getting a couple minutes into this session, I broke out the profiler and hooked it up, just to see if anything really caught my eye. The main metrics I wanted to focus on were those pertaining to garbage collections, as they're the first thing that came to mind when you described the stuttering behavior. Here's what I came away with:
Image
The first and last collections that top 100ms seem to be caused by the profiler itself, and all the meaningful values are around or below 20ms, which shouldn't be noticeable at all.

When I checked out object allocation statistics, all is as expected:
Image
The physics engine manipulates large byte arrays directly, instead of operating on objects, for the express purpose of minimizing the total number of object allocations. This is done to reduce stress on the garbage collector, and it fits with what I expected to see if everything were working as intended, which it seems to be in this case.

wafehling wrote:The spikes occur before I even land my colonists, but yeah. After landing them they get gradually worse.

It's actually semi-expected at this point that you might see a processing spike right when you zone into a world. From a user experience perspective, it's not good, but the initial spike should wear off in a matter of moments. The reason this happens is because of the pressure-based model for liquid physics. When any volume of water is created during world generation, it's uniformly initialized with water's standard density. However, water usually creates formations that span many Z levels, like oceans or lakes, with some water at the surface and some water far beneath the surface.

When you first load into a world, the physics processing takes its first pass at all the water near you, and it exerts the downward forces for water that's near the surface on water that's underneath it. This results in deeper water being compacted to higher densities, which is the whole point of the system. Thousands or millions of points of water density are updated on the first physics passes, so the density of the deep water is increased until it's brought it into equilibrium with the density of surface water, and the tiny amounts of water can stop moving through the now static bodies.

I should probably pre-compute all the water densities during world generation so that this doesn't happen on world load, but I just haven't had a chance to implement this yet. The decreased frame rate should wear off in < 5 seconds, so I have it listed as a low priority feature request.

So that might explain what you experience right as you zone in, but after that...

There's no combat going on at all, no.

Alright, that axes off one theory.

The stutters start out small, about 1/2 a second every few seconds, but as I start to set up a colony they get longer and longer, at least several seconds.

And that's definitely long enough to be highly anomalous.

Ask away.

Fantastic, I'll ask a few questions about your machine.

I have this information:
- Your CPU Model - 6600K
- Quantity of RAM - 16GB
- Your GPU (Video Card) Model - GTX 970

I could use this information:
- Storage Drive (especially interested in RPM / SSD speed)
- Your Motherboard (if you know it)
- Your Sound Card (if one is present)

Pertaining to the stuttering:

1. Are you listening to in-game music? If so, does it stutter, too? And if you use the options menu to turn music off, does this solve the problem? (This pertains to a fairly wild idea I have, but I don't expect it to fix the problem.)

2. Is there water on your world? If so, is it ocean or fresh water? Does it have any flowing water, like rivers or creeks? You should be able to see this all by checking out the world map.

3. What operating system are you running?

4. If I share my email address and instructions via a PM, would you be willing to email me the world save file? I could look at it on my end, and see if it's something about the world that's causing this funkiness. Don't worry if you're not comfortable doing this, as there are other ways I can continue to work on the issue!

Thanks again, man. I'm really appreciating your patience here.
User avatar
Administrator
Site Admin
Posts: 315
Joined: Fri Oct 21, 2016 4:05 am
Location: Pennsylvania
Contact:

Re: Optimization?

Postby Administrator » Mon May 15, 2017 9:53 pm

I've been looking into this for a while, and Torvus texted me to let me know he was running into the same issue, which he hadn't seen until recently.

This makes me almost sure that it's related to the gravity / physics changes I've made recently, which are covered on the TIGSource forums post, here.

Looking at the code, I think I see what the problem is. If a situation arises where a physics processing task doesn't get completed in the allotted time, another physics processing task can be enqueued right on top of it. This backup will continue, getting worse and worse, until everything starts locking up. On my normal stress test worlds, I'm not hitting this case, because there aren't any regions around my colonies that call for much physics processing.

I'm going to keep digging into this, but I hope to have another patch out in a few hours, and it might help. This is the downside of distributing a demo for a game that's still in very active development. Sometimes a new release will include code that has unintended side effects, and sometimes these side effects and bugs get past my own personal testing.
wafehling
Posts: 9
Joined: Sat May 13, 2017 8:54 pm

Re: Optimization?

Postby wafehling » Tue May 16, 2017 12:59 am

Storage: SanDisk SSD PLUS 2.5" 240GB SATA III Internal Solid State Drive (SSD) SDSSDA-240G-G25

Motherboard: GIGABYTE GA-Z170XP-SLI (rev. 1.0) LGA 1151 Intel Z170 HDMI SATA 6Gb/s USB 3.1 USB 3.0 ATX Intel Motherboard

Sound Card: Nope.

1. The music doesn't stutter.

2. I've tried several worlds with and without water, there's no difference I can tell.

3. Windows 7 64-bit

4. If you want, sure. Don't think it'll make much difference, but can't hurt.
User avatar
Administrator
Site Admin
Posts: 315
Joined: Fri Oct 21, 2016 4:05 am
Location: Pennsylvania
Contact:

Re: Optimization?

Postby Administrator » Tue May 16, 2017 1:48 am

wafehling wrote:Storage: SanDisk SSD PLUS 2.5" 240GB SATA III Internal Solid State Drive (SSD) SDSSDA-240G-G25

Oh man, I love the SSD. And Outer Colony loves it, too! Load times can get a bit crunchy with huge worlds, but this is the kind of hardware that makes playing with them feasible.

I borderline needed to get one last summer as Eclipse, which I use as my IDE for developing Outer Colony, was locking up and freezing on me all the time when I was using auto-complete. Outer Colony uses Google Protocol buffers for serialization, and these generate some very large classes from Outer Colony's schema. I love protocol buffers - they're amazing! But there are literally hundreds of thousands of lines of auto-generated code in Outer Colony's serialization classes, and it was giving Eclipse fits, because something about that + using auto complete was furiously taxing the hard drive.

As soon as I dropped an SSD in my machine, all problems were solved. Are you a Java developer, by the way? It seems from your posts that you've got software development experience.

4. If you want, sure. Don't think it'll make much difference, but can't hurt.

You're right, since it's every world, it's likely that sending me one won't reveal any useful information.

I'm hoping that the root of these problems was the set of changes I recently made to the physics processing framework. I cut a new release tonight, version 0.5.48, which your installation of Outer Colony should automatically download and install the next time you start it up. I can't say authoritatively that this will fix the stuttering, as I failed to reproduce it on my end, but this release should drastically reduce CPU load, which I hope will make things more playable.

If you can let me know if this works, that would be a huge help! Thanks again for lending a hand here.

Return to “Issue Reporting”

Who is online

Users browsing this forum: No registered users and 1 guest