Outer Colony details?

For discussing general game development, including other projects.
Anachronic
Posts: 67
Joined: Tue Nov 08, 2016 4:23 am

Outer Colony details?

Postby Anachronic » Tue Nov 08, 2016 4:53 am

Hey Voyager, can you talk at all about what you're using to develop Outer Colony, like languages or an IDE? How long has it been in development? Thanks!
User avatar
Administrator
Site Admin
Posts: 315
Joined: Fri Oct 21, 2016 4:05 am
Location: Pennsylvania
Contact:

Re: Outer Colony details?

Postby Administrator » Tue Nov 08, 2016 10:38 pm

Anachronic wrote:Hey Voyager, can you talk at all about what you're using to develop Outer Colony, like languages or an IDE? How long has it been in development? Thanks!

Some great questions, man! First, I've got to say that I'm sorry for having been so slow to respond here. I've been derelict in my forum duties for the last few days. I've just been head-down fixing multiplayer bugs and working on dedicated server mechanics, but we're getting close to smooth, stable multiplayer.

As for the technologies being used, Outer Colony has one of the leanest, simplest tech stacks possible. It's 100% Java with virtually no external libraries at all. The only external library we use is Google's protocol buffers, for ultra-high-performance serialization.

There are a lot of reasons for choosing this minimalist approach, and some of them are described in the article about Outer Colony's 100% in-house, custom engine. I am a big fan of a bunch of different game engines, especially Unity. Unity is a really special piece of technology that opens a million doors, and under most circumstances, if I wanted to build a game, I'd build it using the Unity Engine.

Outer Colony's processing requirements are extreme and uncommon, though, and the engine needed to be built in a different, fundamentally multi-threaded way. It's architected more like some of the distributed simulation systems I've worked with than a traditional game.

My IDE of choice is Eclipse, but this is just a matter of familiarity and personal preference.

So, with that said, the actual development mechanisms are shockingly primitive! Just coding tons of Java in Eclipse, without much window dressing. The downside is that I have to implement 100% of everything myself, from the rendering framework to physics. The upside is that I get to implement 100% of everything myself, from the rendering framework to physics. I need that extreme degree of control to make it all work. For a more mechanically normal game, it would be an insane waste of time to write all of these things myself, but it's just necessary for OC.

Finally, one of the major benefits of this kind of simplistic approach is that Outer Colony should run on Windows, Linux, and Mac without too much effort. The QA requirements imposed by LInux and Mac releases may make them initially prohibitive, but I'm really, really hoping to make them realities in the near future.

As far as time in development is concerned, that's another really good question! And it's not as easy to answer as one might think.

In terms of how long I've been working full time on Outer Colony proper, that'd be since February of 2015, so we've been going at it for a few months shy of two full years. However, there are elements of Outer Colony's codebase that were lifted from its previous incarnation as TFR (The Far Reaches), which started full time development in January of 2013. Although the initial Unity prototyping phases date back to around Thanksgiving of 2011. When we started depends on the definition of started, and it's hard to clearly delineate that point. We've been at it for a long, long time, and the vision has grown and changed as we've progressed on our voyage.

In 5 years, nothing has stopped us, and nothing is going to stop us moving forward.

I hope that this answers your questions! If you'd like any more details about any of this, please feel completely free to ask. I really believe in as much transparency as possible in development, and I'm happy to share details about how we got to where we are today.
Anachronic
Posts: 67
Joined: Tue Nov 08, 2016 4:23 am

Re: Outer Colony details?

Postby Anachronic » Wed Nov 09, 2016 5:04 am

That's some awesome info, and it's crazy cool that you're building it all from a basic Java foundation. Serialization is one of those things that I've been dabbling with trying to learn for a while, but it's just not really necessary for smaller mobile games where I can get away with Unity's Player Preference system, so I just keep putting it off. Gotta keep the learning agile too right? Just researching things as I need to implement them without getting too off-track with my lists of user stories. It would be nice to learn more about multi-threaded coding too. That's something I understand the concept of but not really the implementation. I'm sure I'll come up with some more questions to ask you as OC progresses.
User avatar
Administrator
Site Admin
Posts: 315
Joined: Fri Oct 21, 2016 4:05 am
Location: Pennsylvania
Contact:

Re: Outer Colony details?

Postby Administrator » Thu Nov 10, 2016 1:54 am

Anachronic wrote:Serialization is one of those things that I've been dabbling with trying to learn for a while, but it's just not really necessary for smaller mobile games where I can get away with Unity's Player Preference system, so I just keep putting it off.

Oh man, I'm positive that serialization would be a breeze for you if you ever had to deal with it. It's really just a mechanism to convert data to some persist-able or transferable format, conceptually almost no different from Unity's preference system.

There are a bunch of different approaches I could've taken for saving Outer Colony world data, and I briefly entertained the idea of using something like embedded HSQLDB or SQLite, just because of my familiarity with RDBMS, but I decided that rolling my own file formats and using object serialization to persist a world's object tree would be faster, leaner, and simpler. It's one of those choices that was made after prototyping very early in the technical design, and I haven't looked back.

For a long time, I was actually using native Java serialization to persist everything, but I needed a finer degree of control and a higher performance option. There's still a switch you can flick internally to make the engine change serialization formats, but I haven't tested the Java serialization mode or used it in ages, since there's really no advantage it would confer once the work is done to put a protocol buffer implementation is in place.

Gotta keep the learning agile too right? Just researching things as I need to implement them without getting too off-track with my lists of user stories.

Spoken like a true professional. One of the most severe problems I've ever seen in a production system was when the project's chief engineer decided to treat it like a science experiment. Every week he was fiddling around with some new piece of technology and actually jamming it into the codebase. The system turned into a byzantine, cobbled-together nightmare that barely worked, and that no one could even begin understand or maintain.

I find it's always best to apply technologies only when they're absolutely necessary. Learn as you go! And just because I've learned about some new piece of technology doesn't necessarily mean I should use it indiscriminately.

It would be nice to learn more about multi-threaded coding too. That's something I understand the concept of but not really the implementation. I'm sure I'll come up with some more questions to ask you as OC progresses.

Anytime you want to start experimenting with multi-threaded designs, let me know! I'm certainly not the world's foremost expert or anything, but I'd be glad to answer whatever questions I can.
Anachronic
Posts: 67
Joined: Tue Nov 08, 2016 4:23 am

Re: Outer Colony details?

Postby Anachronic » Tue Nov 15, 2016 4:18 am

Thanks for the links, I'll take a look! That sounds like a nightmare with the over-engineering. Outer Colony really is on another level...

Can you tell me a bit about the process of coding for multi-threading? Like, do you need to access a Java library or something, or is it just about the way you structure your code? Is multithreading when you have to use coroutines instead of just calling functions?

Also, are you using automated testing all for Outer Colony? I keep seeing that topic creep up and I get the concept but my experience with it is limited to a bit of Java BlueJ from school.
CzarQwerty
Posts: 12
Joined: Thu Oct 27, 2016 3:37 pm

Re: Outer Colony details?

Postby CzarQwerty » Wed Nov 16, 2016 3:29 pm

Sam, I can help you answer some of these questions....

Multithreading is a complex beast, and every language handles it differently. Since we are talking about Java, I'll get you started there.
Here is a great primer on threads in java. https://www.tutorialspoint.com/java/jav ... eading.htm

The basic idea is that you can create a new class that implements Runnable or extends Thread. Calling Thread.start() will start the class as a new process and execute the run() method. When the run method ends, the process is completed. (This is an extremely basic explanation).
The handle more complex things you can make Workers and manage threads in groups. That's the low level approach that gives you the most control.

In Java 8, parallel processing was introduced in Lambda expressions so that you can process a list (or other collections, feeds, whatever) in parallel. You can find tutorials about this as well https://docs.oracle.com/javase/tutorial ... elism.html
Lambda introduces functional programming to Java.
For instance, you can sum a list like this:
for (int i :myList)
result+=i;

In java 8:
myList.stream().sum();

In parallel:
myList.parallelStream().sum()

This last option will start multiple threads for you and close them when they are done.

The other thing to mention is that if you are trying to do any kind of graphical interface in Java, you'll need to learn threads.

Hope this helps.

Tony
Anachronic
Posts: 67
Joined: Tue Nov 08, 2016 4:23 am

Re: Outer Colony details?

Postby Anachronic » Wed Nov 16, 2016 9:45 pm

Hey thanks for the info CzarQwerty! This might be a dumb question, but does the whole code need to be structured as threads or can you pick and choose what functions you want? Say, could you just choose to start an intensive thread on a separate processor when necessary?
CzarQwerty
Posts: 12
Joined: Thu Oct 27, 2016 3:37 pm

Re: Outer Colony details?

Postby CzarQwerty » Thu Nov 17, 2016 1:51 pm

Yes to the second option. Normally, you want to start extra threads only when you need them for a specific task.
So you would write your java program as normal, and then have it kick of other threads when processing needs to take place.
In Outer Colony code, I would imagine that Sam has several threads running, all with specific tasks to do or monitor, and a main thread that is managing and monitoring the status of the other threads.

Multi-threading is very useful but very challenging to work through all of the timing issues and concurrency bugs that could be introduced.
User avatar
Administrator
Site Admin
Posts: 315
Joined: Fri Oct 21, 2016 4:05 am
Location: Pennsylvania
Contact:

Re: Outer Colony details?

Postby Administrator » Fri Nov 18, 2016 1:04 am

Some great questions here, Anachronic! And some excellent answers from the Czar, which is a very fitting name for him in this thread in particular.

CzarQwerty was actually the chief software engineer at my second (and, from a learning perspective, most important) job, and I basically learned everything I know about software engineering under his tutelage, including the meaningful theory and practice of concurrency. As such, there's probably no one more qualified to answer questions about multi-threading or provide an introduction to the topic than the Czar. If there were a guru for this sort of thing sitting in the mountain temple atop the Himalayas, it'd be CzarQwerty.

The links he's provided are a great place to start learning about concurrency, if you'd want to get started in Java. If you're looking to dive into it on the Unity side, you'll find that C#'s multithreading structures mirror Java's almost perfectly, as is the case with 95% of the language. This is especially true at the more fundamental level. I find that Java provides a more robust set of concurrency-related data structures out of the box than C#, but anything that you can do in one language could be done in the other.

Multithreading can get a bit strange when it comes to game engine design, because almost all engines, Unity included, run some sort of central "game loop" that's responsible for rendering frames, (sometimes) ticking physics, and doing other fundamental, engine-y things. Outer Colony doesn't exactly have a traditional setup in this regard. Without getting too technical and (possibly) boring, Outer Colony's engine works on a fixed set of worker threads that independently process jobs. The engine doesn't really care what the jobs are, and pretty much everything is happening all at once without any engine-level synchronization. And when I say "everything" here, I really mean everything - even frame rendering and authoritative time progression happen with (almost) no regard for everything else that's happening in the system. This design is actually extremely simple, but its proper use can be where things get tricky.

There are ways that you could build a mostly similar setup from within the confines of Unity, but there were some scary things about trying that implementation at design time that made me decide to just build it all myself in Java. I could probably write a dozen pages about this specific set of technical choices, but I'll try to keep my answers at a high level, for now.

Anachronic wrote:Can you tell me a bit about the process of coding for multi-threading? Like, do you need to access a Java library or something, or is it just about the way you structure your code? Is multithreading when you have to use coroutines instead of just calling functions?


I'll echo CzarQwerty's statement that the mechanics of multithreading are different in every language, but you're right here that coroutines / code using the yield keyword are a way to implement concurrency in C#. If you check out tutorials, you'll see that the implementation of a coroutine or most any sort of concurrent mechanism is mechanically very simple! It's often just a few lines of code to make the low-level threading happen.

Where it becomes complicated, though, is in the abstract effects of now running multiple processing streams in parallel. One thing I always try to impress (and something I learned from the Czar) is that the mechanics of code aren't where elite developers make their money. Anybody can learn to write code and be a "programmer", and in this regard, anybody can learn the mechanics of writing a coroutine or creating a new thread. What's hard is the abstract problem solving - the data structures and algorithms - the design that goes into building systems that solve real problems is something that comparatively few people can do, and this is what separates a proper software engineer from a "coder" or "programmer" in my vernacular.

The power of multithreaded code is the ability to take advantage of modern hardware (like many-core processors or larger, distributed systems) to compute things faster. The difficulty comes from having no guarantees about execution order for code that runs in parallel. If you're writing single threaded code, you know that all the lines of code are processed sequentially. When you're writing multithreaded code that's manipulating shared data structures, like an Outer Colony world, every thread is doing whatever it feels like to the world, whenever it feels like doing it, excluding the flow controls that you implement yourself.

The best place to start learning about the kinds of problems that can arise is probably the Wikipedia article on race conditions:
https://en.wikipedia.org/wiki/Race_condition

This sums up the fundamental difficulty in the whole domain of concurrency. There are a lot of ways to deal with race conditions, and there are a host of synchronization mechanisms you can employ to combat these problems:
https://en.wikipedia.org/wiki/Synchronization_(computer_science)

The problem is that, without very careful design, the synchronization mechanisms can create their own host of issues:
https://en.wikipedia.org/wiki/Deadlock

So, how do you make all this work? The real answer is simply that it's hard, and any multithreaded implementation is going to be far more difficult to build than a single threaded alternative. This is why people usually don't bother with multithreaded code: it's really hard, and unless there's some extreme processing demand that makes it necessary, you'd never want to touch it.

I could go into some details about how Outer Colony fundamentally copes with race conditions, but I don't want to write way too much in one reply (although I probably already have).

Also, are you using automated testing all for Outer Colony? I keep seeing that topic creep up and I get the concept but my experience with it is limited to a bit of Java BlueJ from school.

Ohhhhhhhhhh, this is another fantastic question, and one for which I'd also like to write a thorough response. I've got pretty strong feelings about automated testing, and I think that there's a time and a place for it. Some applications lend themselves to automated regression testing, but most non-trivial, user driven applications (like games) do not.

The reason for this is simple. Users do too broad a range of things in too erratic a way to meaningfully cover in an automated fashion. Automated tests are time consuming to write, and no matter how good your QA team is in terms of personnel (and they're usually quite poor), you'll never have time to write automated tests that can actually behave like a real user. You might be able to cover some set of typical cases and boundary cases, but whatever you write is going to be very limited. In my experience, the batteries of automated tests are usually so limited that they're already covered naturally by developers in their personal, rudimentary, in-development testing.

For example, an automated test to make sure that login functionality works for a base privilege user is always going to be covered in normal, rudimentary testing. And this is usually the most advanced kind of test that I see QA teams produce for their automated battery.

So what I see in practice is automated test batteries that exercise trivial cases in an extremely on-rails fashion, the batteries take huge amounts of time and money to produce, and they rarely exercise any more code than a developer does in his own private testing. Worst of all, though, is the sense of security that the tests provide. A poor QA team will say, "Well, the code passed all of our automated tests! Regression is green! Move it to production!" without sufficient user testing. I've seen QA teams that would do this with zero user testing, and the result was that every production push was a total disaster that could've been prevented by ten minutes of actual testing with a real user.

How does a QA system based on automated testing, that's delivering these very poor results, even come to be? Organizational deficiencies at the management level are the root of all evil. If you've got senior management that doesn't have a solid understanding of software engineering, they can go off the rails really fast and really hard with this stuff.

Management wants to be metrics-driven, they want to be numbers-driven, because measuring performance is a large part of what management does. This measurement and judgment is a core aspect of management's role, and automated QA systems can yield all sorts of juicy, management-friendly numbers. "What percentage of our codebase is covered by automated tests?" This is a fundamental question I see people at the VP level ask, and the real answer is, "It ultimately doesn't matter. Are the production releases coming out full of bugs, or are they clean and stable?" This is the question that matters first.

I've seen teams get consumed by this obsessive desire to reach "100% coverage", as reported by automated testing tools, as though this number is going to mean that the system is free of bugs. But this number doesn't necessarily mean anything. If the tests are all contrived and trivial, then they're meaningless. You can build an automated system that gives "100% coverage" that doesn't do any good in stopping real bugs from reaching production, and this is almost always what I see automated testing systems devolve into. The desire for metrics and misleading / bad seminars push senior management to mandate automated systems, then they're terribly implemented by second-line personnel that make up QA teams, and resources are actually drawn away from useful endeavors in the SDLC, like conducting proper code reviews.

It's not that automated testing is fundamentally bad as a concept, and it's not that automated tests are fully without merit. The problem is that, in practice, the ROI on automated tests is, in my experience, very poor, the obsession with "code coverage" and automated testing saps resources from real user testing (which I almost always see as having a far better ROI), and it creates this awful sense of security by generating green lights and putting meaningless checks into checkboxes, which results in terrible code reaching production.

I do believe that there are cases where automated testing can provide value, but these cases are few, and they should always be used as a supplemental tool, never supplanting user / acceptance testing.

I'll be the first to acknowledge that what I've just said will put me in conflict with widely held views in the field, with the multi-multi-multi-million dollar QA / automated testing industry, and with most "hip" and "modern" senior management. If I have to work in a realm with automated testing mandates, I absolutely can do it, and I can and will write the best unit tests and GUI / macro tests that I can. My experience, though, tells me that these things will yield little benefit in an extremely user-driven system like Outer Colony, and as the authoritative manager of the project, it's my call (yay!!!!), so I don't employ them.

I could elaborate a lot further on how I'm handling user testing and how one could try to apply automated testing to a project like OC, but Thursday Night Football will be kicking off soon, so I'll end this reply for now.
Anachronic
Posts: 67
Joined: Tue Nov 08, 2016 4:23 am

Re: Outer Colony details?

Postby Anachronic » Fri Nov 18, 2016 5:24 am

My mind is blown :shock: That's some awesome information folks, thanks a ton! I will retreat into my cave for a while to meditate on this newfound knowledge

Return to “Game Development”

Who is online

Users browsing this forum: No registered users and 1 guest