Category Archives: Game dev

6 November, 2023

TeamCity vs Jenkins

We’ve been using Jenkins at Unknown Worlds for over 10 years now. It has been a solid CI/CD system allowing us to handle various tasks around our Unity Engine projects. We started using Perforce again for the new project we’re working on and Jenkins caused multiple problems with it. We had to try a different automation server. We decided to try out TeamCity. I personally love Jetbrains products, so it was quite exciting.

It turned out the systems were quite different. Let’s compare them.

Build nodes management

Jenkins is very simple – you can install the agent (jar file) manually and handle launching it by yourself. That’s it.

TeamCity is much more sophisticated. It offers an agent installer which takes care of heavy lifting. From within TeamCity UI you can clean the workspaces and restart the machine, also access the console.

Scripting

Jenkins uses Groovy for scripting. It’s a very capable system you can program exactly to your needs. It offers a set of pre-defined steps, but also full logic, exceptions, file operations etc.

TeamCity has a different idea on how to handle the pipelines. It offers a list of pre-defined steps, and you manage the flow in a visual way. The scripting itself is made outside of it, but you have way better options – shell, Power Shell, python etc.

Backups

Jenkins is very traditional about it. Since it keeps all the data in flat files, you just back those up. Specifics are listed properly in the documentation.

Backup and config storage are one of the biggest selling points for TeamCity. JetBrains had a great idea – just keep all of it in git! This way we can always roll back or restore the entire TC instance. Brilliant.

Plugins

Jenkins offers hundreds of plugins for almost every problem you might need to solve. They are entirely community driven, so the quality and development speed varies from plugin to plugin.

TeamCity on the other hand has a pretty narrow catalogue. I couldn’t find anything useful in it, but we didn’t really need anything outside of the standard setup.

UI

UI is not the strong point of Jenkins. It does offer two versions of the UI – legacy and Blue Ocean. Both of them are pretty slow, outdated and chaotic. It takes a while to get used to them.

TC on the other hand offers a very modern, responsive interface. It’s almost perfect, but non-technical people require some guidance at first.

Plastic SCM support

Jenkins works great with Plastic. Occasional cleanup is required, but overall experience is great.

I haven’t tested Plastic inside of TC.

Perforce support

That’s why we switched from Jenkins to TC. Workspace management in Perforce is a terrible thing, and Jenkins doesn’t help with it at all. On the contrary – it adds another layer of misery on top of it.

TeamCity on the other hand is OK with it. Not perfect – we still have weird inconsistencies with speed, but it’s OK. Shelves and moving files is still miserable, but manageable.

Upgrades

Jenkins require you to do it from the console.

For Perforce it’s one click from the UI.

Pricing

Jenkins is open source, so you can’t beat that.

TeamCity is pretty expensive. It’s free if you’re using 3 build agents. Anything on top of that is going to cost you. Having 10 agents is 3100 EUR including VAT for the first year, and then the price drops by 50% for the consequent years.

Summary

For me it’s a simple choice. If you can afford it – go for TeamCity. It’s the backbone for your entire company.

Got any questions? Leave them in the comments, I’d be happy to answer.

1 August, 2023

Zabbix

Zabbix is a very flexible infrastructure monitoring tool. Full manual is available here.

The central component of Zabbix is the server. You can find the installation instructions for Zabbix 6 on Ubuntu 22.04 LTS is available here, alternatively you can grab a pre-set up image from the download page.

The second crucial component of the Zabbix system is the Agent that provides the easiest and most flexible way for monitoring the servers. When downloading it, always use the latest version – agent2 in this case.

The configuration for Zabbix Agent is stored at /etc/zabbix/zabbix_agent2.conf

The minimal configuration for an Agent is as follows:

Server=127.0.0.1
ListenPort=10050
Hostname=apache

As you can probably guess, we configure the Zabbix server address, port, and client hostname. After the basics are configured, let’s enable the agent auto-start and make sure it picks up the config changes:

systemctl enable zabbix-agent2
systemctl restart zabbix-agent2

When that’s done, the agent is ready to be added via Zabbix GUI or auto-discovery (we’ll talk about it later).

The Hosts screen in Zabbix GUI allows you to:

  • Monitor dozens of items at the same time – server load, RAM usage, service health (reference)
  • Set up triggers for certain events (eg. /etc/paswd being modified), which can be used for notifications
  • Set up graphs for visualising your data
  • Manage discovery features (hardware – disks, partitions, network interfaces, software)

Templates in Zabbix vocabulary are pre-configured sets of features that are great for specific needs. Out of the box you can get fully-featured monitoring package for things like web servers, SSL certificates or other common use cases. Bear in mind that templates can contain conflicting item names, which could prevent you from using specific templates in some cases.

Monitoring custom scripts is relatively easy. Note that scripts are executed by the same user that handles Zabbix agent – usually that user is simply zabbix. After you create the script you want to monitor, edit the Agent config (/etc/zabbix/zabbix_agent2.conf) and define it at the bottom, following the pattern below:

UserParameter=apache_config_test, apachectl configtest
UserParameter=apache_status, systemctl status apache2.service | grep active

Remember to restart the Aagent with systemctl restart zabbix-agent2. Rest of the work is done via GUI in Hosts -> Items -> Create item. Just input the self-defined item name in the Key field, so Zabbix can pick it up.

No monitoring is serving it’s purpose without proper alerts in place. This is where Triggers come in. They usually come bundled with Templates, which makes them globally accessible. You can configure your own triggers easily, all the expected features are in place.

Notifications require some setup to work effectively. First step would be configuring the notification media. You can do that at Administration -> Media Types, where you can configure pre-defined media like Slack, Jira, Discord, SMS and many others. You can also define your own media types.

Second step is to visit Administration -> Users and configure user’s mediums – you need to set the email address, phone number and such.

Last step is customizing the trigger parameters at Configuration -> Actions -> Trigger.

Agent-less monitoring is exactly what the name says. You can monitor servers without agent installed. This limits the usefullnes of this feature, since we are limited mostly to monitoring services on open ports. We can use agent-less monitoring to check ping, SSL certificates, HTTP, TCP availability, HTTP APIs etc.

Automating host discovery is something very useful in production environment, especially where you auto-scale your infrastructure. Check it out at Configuration -> Actions -> Autoregistration actions. An example would be to accept all hosts with “Host meta data” value to set something specific (Like “webserver”), where you define two Operations:

  • Add host – to simply link it to Zabbix
  • Link to template – to attach specific monitoring package to given machine

Make sure you install and configure the Agent when you provision the new machine. The minimal per-server changes to zabbix_agent2.conf would be:

ServerActive=5.5.5.5 # Server IP
HostMetadata=webserver

As always, make sure to restart the Agent after making any config changes.

Discovering network services is something not very useful in my use cases. It can be used to scan network IP ranges for available services, like FTP, SSH, HTTP etc.

11 March, 2023

Nonviolent communication

Another read recommended in Unknown Worlds Slack. You’ll probably hear that “everyone should read it”. I’m sure that magical solutions do not exist, but it turned out that Marshall Rosenberg’s book is very close to that.

NVC is a process that enables empathic collaboration. It can be used to handle very complex issues, like mediation. I’d call it a special language that requires a lot of practice, time and consciousness. Sadly, I’m not very good at keeping language skills when not using them on a daily basis (sobs in German). I’ll probably have to re-read the book to make some good notes about the most important stuff.

The most important takeaway for me was not the communication with others, but… internal dialogue. I think of myself as a kind person, so I didn’t have to work on the basics. Thanks to NVC I now remember about one thing when talking to others – their point of view. That changes a lot. It’s easier to understand the reasoning and wording of other people.

Back to the internal dialogue thing! The book showed me something that I never considered – I’m too hard on myself. I was raised in a way that makes my life hard sometimes, and the book helped me realize that. I’ve read it a few months back, and now I’m sure I have to revisit it soon…

5 March, 2023

Think like a game designer

Charlie Cleveland recommended this read to the UW team. I wasn’t ever interested in game design, so I decided to finally change that and check out the book. Justin Gary’s work has a subtitle “the step-by-step guide to unlocking your creative potential”. I never had any great game ideas, so I doubted that.

After reading it I had to admit that I was surprised. Gary’s systematic apporach to design seems like a no-nonsense approach that could produce game ideas that could be easily verified. I wish I had time to verify that, haha.

Contents, for future reference:

Part I Understanding Design 19
Chapter 1 Learning Fundamentals 21
Chapter 2 Getting Started 25
Chapter 3 Overcoming Obstacles 35

Part II Learning the Core Design Loop 39
Chapter 4 The Steps of the Core Design Loop 41
Chapter 5 Inspiring 43
Chapter 6 Framing 53
Chapter 7 Brainstorming 59
Chapter 8 Prototyping 69
Chapter 9 Testing 77
Chapter 10 Iterating 85

Part III Refining Your Designs 91
Chapter 11 The Phases of Design 93
Chapter 12 Engine Design 101
Chapter 13 Engine Development 113
Chapter 14 Component Design 119
Chapter 15 Component Development 125
Chapter 16 Polish 131

Part IV Building Great Games 143
Chapter 17 What Makes Games Great? 145
Chapter 18 Elegance 147
Chapter 19 Excitement 155
Chapter 20 Depth 161
Chapter 21 Motivation 169
Chapter 22 Engagement 177

Part V Making Money 185
Chapter 23 Monetizing Games 187
Chapter 24 How to Be a Professional Game Designer 193
Chapter 25 How Can I Get My Game Published? 197
Chapter 26 Game Business Models 203
Chapter 27 How to Make Games That Last 211
A Final Note: Living the Lessons 217

13 February, 2023

Ansible in an hour

I stumbled upon a nice server automation course made by an expert I follow. I don’t do many tedious, repeatable tasks in my daily work, but I wanted to prepare for future.

Ansible is useful for bulk server configuration, application deployment, and other automation tasks. The course I finished is very compact, but it explains the most important topics:

  • Prepping Ansible for use (installation, management node, inventories)
  • Ad-hoc modules (running commands on all servers)
  • YAML configs (playbooks)
  • Facts, variables
  • Playbook creation (generating SSH keys, using variables, loops, groups, creating users, conditionals, file operations, tags, templates, firewall config)
  • External roles (using playbooks from Ansible Galaxy, Docker containers)
  • Creating own roles (complete web server setup example)
  • Ansible Lint (config validation)
  • Ansible Dynamic Inventory (useful for large server farms)
  • Ansible Vault (credentials storage)
  • Ansible AWX (free counterpart of Ansible Tower; a web interface for playbook management)

Looks like a quite useful, pretty complex tool. Sadly, most of the Linux servers I use are handled by Laravel ecosystem tools, so I might need to wait a while before putting Ansible to use.

6 January, 2017

Jenkins, SSL, and cPanel

Straight to the point. The basic idea is to hide Jenkins behind an Apache reverse proxy. I’m using cPanel on CentOS – and cPanel doesn’t like fiddling with httpd.conf. You’ll find lines like this in it:

# To customize this VirtualHost use an include file at the following location
# Include "/usr/local/apache/conf/userdata/std/2_4/user/domain.com/*.conf"

I created two config files – I want to set up proxy and redirect non-HTTPS requests to HTTPS:

"/usr/local/apache/conf/userdata/ssl/2_4/user/domain.com/ssl.conf":
ProxyRequests     Off
ProxyPreserveHost On
AllowEncodedSlashes NoDecode

<Proxy *>
 Order deny,allow
 Allow from all
</Proxy>

ProxyPass         /  http://localhost:8080/ nocanon
ProxyPassReverse  /  http://localhost:8080/

RequestHeader set X-Forwarded-Proto "https"
RequestHeader set X-Forwarded-Port "443"
"/usr/local/apache/conf/userdata/std/2_4/user/domain.com/redirect.conf"
RewriteCond %{REQUEST_URI} !^/.well-known
RewriteRule (.*) https://jenkins.domain.com/$1 [R=301,L]

After that, run commands to refresh the configs and restart Apache:

$ /usr/local/cpanel/bin/apache_conf_distiller --update
$ /usr/local/cpanel/bin/build_apache_conf
$ service httpd restart

And we’re done!

UPDATE – iptables rule to block port 8080 traffic outside of localhost:

iptables -A INPUT -p tcp --dport 8080 -s localhost -j ACCEPT
iptables -A INPUT -p tcp --dport 8080 -j DROP