Author Archives: ace

10xDevs: What Five Weeks of Agent-Driven Development Actually Taught Me

I recently completed the third edition of 10xDevs, a five-week Polish course by Przemek Smyrdek and Marcin Czarkowski on integrating AI agents into the full software development lifecycle. It wasn’t about “how to prompt ChatGPT” – instead it was about how to plan, build, test, deploy, and maintain real software with an agent doing most of the typing while you stay accountable for the result.

Here’s my honest review.

What the course actually is

The premise is simple: casual AI coding (“vibe coding”) doesn’t survive contact with real projects. What does survive is a disciplined workflow, and the course teaches one end to end:

  1. Idea → Product Requirements Document. Before any code, the agent runs a Socratic session that interrogates your idea – who’s the user, what’s the first valuable flow, what’s explicitly out of scope. The output becomes a PRD that acts as a contract for everything downstream.
  2. Product Requirements Document → code. Stack selection based on the problem (not trends), project bootstrap, then a roadmap broken into plan → review → implement cycles – with you reviewing the plan before the agent writes anything.
  3. Quality gates. Test plans with priorities, unit and E2E tests (Playwright driven by the agent), hooks that make the agent react to errors on its own, and techniques for reviewing AI-generated code quickly without rubber-stamping it.
  4. Legacy code. The part I found most valuable – generating a project map of an unfamiliar codebase, analyzing features, refactoring safely behind tests, and using DDD to carve out domains the agent can work on independently.
  5. Team scale. Agent SDKs, cost and metrics tracking, AI reviewers in the CI pipeline, and shared skill registries so a team’s agent knowledge doesn’t live in one person’s head.

Certification required shipping a real project: working auth, actual data management, non-trivial business logic, at least one user-perspective test, and a CI/CD pipeline. I completed mine – the requirements were sane enough to force real engineering rather than a demo page.

What I took away

The specific tools will age – that’s the nature of this space, and the course’s very opinionated workflow is a snapshot of how agent tooling works right now. But the habits underneath it transfer to whatever comes next, and those are what I’d actually put on a CV:

  • Requirements as contracts. An agent will confidently implement the wrong assumption at scale. Forcing decisions into written artifacts before implementation eliminated a whole class of rework – and it made me realize how often I’d been skipping this with human teams too.
  • Short verification loops. The skill isn’t trusting the agent; it’s structuring work so it can be checked cheaply – small increments, tests as gates, review of plans instead of just diffs.
  • Context engineering. Deliberately managing what the agent knows – project maps, rules files, thread lifecycle – is the difference between an agent that’s useful on a 500-file legacy codebase and one that hallucinates architecture.
  • Reviewing at speed. AI generates code faster than anyone can traditionally review it. Learning to verify effectively, not just read line by line, may be the most durable skill in the whole course.

Who it’s for

Working developers who already write production code and want a system for AI-assisted development rather than a bag of prompting tricks. If you’ve never coded, this isn’t your entry point. If you’re already deep into agentic workflows, you’ll know parts of it – but the legacy-code and team-scale modules will likely still earn their keep.

Verdict

Recommended. The workflow is prescriptive – deliberately so – and I’d advise treating it as scaffolding rather than doctrine: the exact commands and artifacts will evolve, quickly. But the engineering discipline it drills – contracts, verification, context management – is exactly what separates developers who ship with AI from developers who generate code with AI. Five weeks well spent.

But it’s not flawless. I can’t agree with giving an AI agent a full access to git, project management software and some other parts of the ecosystem – even in a tight harness. It makes me really uncomfortable. There’s no way I’d want to be held accountable for such code.

The course was originally written in Polish, and it’s available in English. I went for the Polish version… and it was really, really hard to process. Terrible mixes of Polish and English, made-up words etc. On top of that you could clearly see parts of the text were AI-generated. It was also full of acronyms. Tough read!

Was it worth the original price? Well, no. Definitely not. It’s priced the same way as the top-of-the line knowledge you can get elsewhere, yet you can see significantly less effort put into making the course.

I thought soft skills training was snake oil. Then I took one.

For most of my career I was convinced that soft skills trainings sell false solutions — vague frameworks, feel-good slogans, nothing you can actually use on Monday morning. So when I signed up for Difficult Conversations and Emotional Dysregulation at Kozminski University, my expectations were low.

Two intense days later, led by Magda Żmijewska — an executive coach with 25+ years in business and a former HR manager — I walked out with a changed perspective and, more importantly, a set of tools I’ve been using ever since.

What actually stuck

Difficult conversations can be iterative, not always final. This was the single biggest shift for me. A hard conversation doesn’t have to resolve everything in one sitting — you can approach a problem piece by piece, clarify, improve, and come back to it. The training frames this with the Deming PDCA cycle applied to communication: plan, try, check, improve. Once you stop treating a difficult conversation as a one-shot, all-or-nothing event, most of the fear around it disappears.

Low tension is the foundation of good communication. Stress pushes people out of their creative, productive mode and into survival mode. A surprising amount of the toolkit is simply about reducing tension: no surprises (send an agenda, especially for a hard meeting), take a 5–10 minute break when a discussion gets emotional instead of postponing it “to next week,” and control your own voice — calm, deliberate speech de-escalates more than any clever phrasing.

Open questions beat ready answers. Instead of jumping in with judgments or solutions, ask: “How did it go?” “What was the biggest challenge?” “What surprised you?” Only after that: “What would you do differently next time?” It sounds simple, but structuring feedback conversations this way — positives first, difficulties later, improvement last — completely changes their dynamics. It also taught me to stop “rescuing” people by finishing their sentences.

Communication styles are a practical lens, even if you doubt the theory. The training uses a DISC-style model of communication types and their stress triggers. I went in thinking this was pseudoscience — and I’m still not going to defend it as hard science. But as a working model for adapting how you talk to different people, it delivers real results. I also learned something about myself: the conversations I find hardest are the ones centered on emotions, which told me exactly where I need to grow.

Kindness is a strategy, not a weakness. Genuine goodwill, paired with being “soft on people, hard on problems,” gets you remarkably far — even with people you’ve just met.

Why this matters beyond one training

This was my first extensive soft skills training, and it recalibrated how I think about the whole category. The difference wasn’t the topic — it was the trainer. Magda combined solid theory (psychology of communication, basics of the neurobiology of stress) with exercises that made the tools stick: group work on communication styles, Gestalt chair technique for repairing strained relationships, facilitation frameworks like the “12 powerful questions” for teams in trouble.

The practical result: difficult conversations stopped being something I avoid. I now have a repeatable way to prepare for them, run them without escalation, and — when they go sideways anyway — return to balance and try again. That’s a skill that pays off in every code review, every planning meeting, and every disagreement with a colleague or client.

If you’re an engineer or specialist who rolls their eyes at “communication workshops” — I was you. This one is worth your skepticism being proven wrong.

Deep dive into Production AI Systems

What Five Weeks of AI_devs 4 Taught Me About Building With LLMs

AI slop sucks. And “AI” is not even AI. Large language models are glorified word generators. Yeah, AI is everywhere and it’s easy to hate it. To say the least: I was hesitant.

I saw how quickly trends come and go in the “AI” world. Getting specialized in something that’ll be obsolete within months is laughable. But hey, I quickly learned that there are concepts that will stay with us. Prompting is one of them, and I quickly learned that every letter of the prompt can matter.

And prompt is only the visible edge of a larger system. Behind it are decisions about data, tools, memory, permissions, observability, product design, cost, failure modes, and the boundary between deterministic software and probabilistic reasoning.

All of that was quite… exposed when tons of data was leaked from Anthropic recently.

Anyway, the course lasted five weeks and covered a wide range of topics: language model APIs, structured outputs, function calling, MCP, multimodality, context engineering, RAG, agent design, evaluation, internal company workflows, production architecture, and AI security.

But the deeper lesson was simpler:

Building useful AI systems is not only about asking models better questions. It is about designing environments in which models can perform useful work reliably.

That distinction changed how I think about AI engineering.

First Principles

Before summarizing the course, it helps to reduce the topic to first principles.

An AI system has five fundamental parts:

  1. A model that can reason, generate, classify, transform, and decide.
  2. Context that shapes what the model can understand in the current moment.
  3. Tools that let the model affect the outside world.
  4. State and memory that preserve information beyond a single interaction.
  5. Control systems that make the whole process observable, secure, measurable, and economically viable.

If any of these parts is weak, the system becomes unreliable.

A strong model with poor context gives confident nonsense. A strong model with dangerous tools becomes a security risk. A strong model without evaluation becomes impossible to improve. A useful prototype without cost controls may ruin your budget. A clever agent without permissions may do the wrong thing at scale.

This is why AI engineering needs to be treated as engineering, not prompt improvisation.

The core questions are not only:

  • What should I ask the model?
  • Which model is best?
  • How do I make the answer sound right?

The better questions are:

  • What does the model need to know?
  • What should the model never be allowed to decide?
  • Which parts should be deterministic software?
  • Which parts benefit from probabilistic reasoning?
  • How do I validate the output?
  • How do I observe failures?
  • How do I keep data, tools, and users safe?
  • How do I measure whether the system is improving or deteriorating?

AI_devs 4 was valuable because it kept returning to those questions from different angles.

The Shift: From Chat to Systems

The first stage of the course focused on programming interactions with language models. That included model APIs, structured outputs, JSON Schema, prompt organization, provider capabilities, smaller versus larger models, and the practical limits of working with token-based systems.

The important idea was that a language model should not be treated as a text box. It should be treated as a component inside an application.

That means the application needs contracts.

If the model returns data, the data should be structured. If the system expects JSON, the schema should be explicit. If the model can fail, the software should handle that failure. If the result triggers a business action, it should be validated before execution.

This is one of the clearest boundaries between experimentation and engineering. In a demo, a natural-language answer may be enough. In a product, the model has to cooperate with software.

The same principle applied to multimodality. Images, PDFs, audio, and video are not just “attachments”. They require ingestion, transformation, preprocessing, interpretation, and often multiple steps before the model can use them well.

A production AI application is rarely one model call. It often is a pipeline.

Tools Are Interfaces for Models

The next major concept was tool use: function calling, custom tools, native tools, APIs, CLI tools, and MCP.

This was one of the most practical parts of the course because it reframed APIs from a model’s perspective.

Developers usually design APIs for other developers. AI systems force a different question: can a model understand this interface well enough to use it correctly?

That changes the design criteria.

Names matter. Descriptions matter. Parameter schemas matter. Defaults matter. Error messages matter. Permission boundaries matter. The response after a successful tool call matters because it becomes part of the next step of reasoning.

A tool is not just an implementation detail. It is part of the model’s working environment.

A poorly described tool can make a capable model look weak. A focused tool with clear schema and useful feedback can make a smaller model effective. Too many tools can overwhelm the context window and make selection unreliable. That leads to patterns like dynamic tool lists and tool search, where the model first discovers the right capability before using it.

The lesson for me was direct: tool design is AI system design.

Context Is Architecture

One of the strongest themes in the course was context engineering.

At first, it is tempting to think that larger context windows solve the problem. They do not. More context can mean more noise, more cost, more latency, and more opportunities for the model to lose track of what matters.

Context is not a storage bucket. It is an active design decision.

The course covered dynamic system instructions, conversation state, external documents, masking, shared context between threads, global context, local task context, and strategies for deciding what should be placed in the prompt versus stored elsewhere.

This is where the distinction between context, memory, and knowledge became important.

Context is what the model sees now.

Knowledge is information the system can retrieve when needed.

Memory is information the system preserves across interactions and uses to adapt future behavior.

Conflating these creates weak systems. Dumping everything into the prompt is expensive and fragile. Hiding everything behind retrieval can fail when search quality is poor. Letting memory grow without structure creates noise and risk.

Good AI systems decide what belongs where.

That principle also shaped the course’s treatment of RAG. Retrieval-augmented generation is not solved by adding a vector database. It depends on source quality, document structure, metadata, chunking, query transformation, ranking, context presentation, and evaluation.

The first-order principle is simple: the model can only reason over the information it receives, and retrieval decides what information arrives.

Agents Need Boundaries

The course moved from workflows to agents, but it did not present agents as magic.

A workflow follows known steps. An agent receives a goal, context, and tools, then decides some of the steps itself. That autonomy is useful when the path is uncertain, but it also increases risk.

The mature question is not “Can I make this an agent?”

The mature question is “What level of autonomy does this task deserve?”

Some tasks should remain deterministic. Some should use a model for classification or extraction. Some should use an LLM inside a controlled workflow. Some justify an agent with tools, memory, and planning. Some require multiple specialized agents with separate responsibilities.

The course emphasized agent roles, instructions, tool assignment, context sharing, manager-worker patterns, multi-agent systems, file-system-based memory, sandboxing, and human-in-the-loop approval.

The most useful design principle was specialization.

One broad agent with many responsibilities is harder to evaluate, harder to secure, and easier to confuse. Smaller agents with narrower responsibilities can be tested, observed, and improved more effectively.

This maps well to traditional software design. Clear boundaries reduce complexity. AI does not remove that principle. It makes it more important.

Evaluation Turns Guesswork Into Engineering

A recurring production theme was evaluation.

Manual testing is not enough. A prompt that works three times in a chat window may fail with a different user, a different model version, a longer context, a malformed input, or a changed tool response.

The course covered observability, logging, replaying interaction state, versioning system instructions, building test datasets, defining success metrics, and using model-based judging carefully.

This matters because AI behavior changes.

Providers release new models. APIs evolve. Output formats shift. Smaller models may appear to work until edge cases expose them. Fallback models may respond differently to the same prompt. Long-running agents may accumulate errors across steps.

Without evaluation, teams are left with vibes.

With evaluation, AI systems become improvable.

That is one of the most important professional habits I took from the course: treat prompts, tools, model choices, and context strategies as versioned artifacts that can be tested.

Production Changes the Problem

The final stage of the course focused heavily on production realities.

This included architecture, provider abstraction, model migration, rate limits, cost monitoring, latency, fallback models, moderation, privacy, tenant isolation, logging, prompt caching, and security.

These topics are easy to ignore during a prototype. They become central as soon as users, data, and business processes are involved.

A production AI system has to answer practical questions:

  • What happens when the provider is down?
  • What happens when the model changes behavior?
  • What happens when one user generates most of the cost?
  • What happens when a prompt injection tries to manipulate a tool?
  • What happens when sensitive data appears in input or output?
  • What happens when logs contain private information?
  • What happens when an agent has access to files, APIs, or a browser?

The security principle was especially clear:

The model is never the security boundary.

Authorization belongs in backend code. Tool permissions should be explicit. Model output should be treated as untrusted input. Prompt injection is not solved by telling the model to ignore malicious instructions. Classic application security still applies, but AI introduces new paths for old risks.

That is the difference between building something impressive and building something responsible.

AI Inside Organizations

The course also looked beyond individual applications and into company workflows.

This was important because many AI projects fail by starting with the technology instead of the process. A chatbot is not always the right interface. Sometimes the better solution is a background workflow, a document pipeline, a review assistant, a search layer, or a tool embedded into existing operations.

The course covered process mapping, internal knowledge bases, company tools, privacy constraints, human review, asynchronous AI work, and the difference between visible chat interfaces and operational automation.

The practical lesson was that AI should be introduced where it changes the work, not where it looks impressive.

That requires understanding the business process, the users, the risk of errors, and the cost of human review.

What This Says About How I Build

The biggest change in my thinking is that I now see AI features as systems with moving parts, not isolated model calls.

When I look at an AI use case, I think about:

  • the task boundary
  • the data sources
  • the context strategy
  • the model’s role
  • the tools and permissions
  • the deterministic parts of the workflow
  • the evaluation method
  • the failure modes
  • the cost profile
  • the security model
  • the user’s actual workflow

I also think more carefully about whether an agent is necessary. Autonomy is not automatically better. The right design may be a simple structured extraction, a deterministic workflow with one model step, a retrieval system, a specialized agent, or a multi-agent process.

The goal is not to use the most advanced pattern. The goal is to build the most reliable useful system for the problem. And sometimes less is more – forcing people into using AI won’t work. On the other hand improving existing workflows might get real traction in a team.

Skills I Can Bring Into a Team

AI_devs 4 gave me a practical foundation across the full lifecycle of AI application development.

I can reason about model interaction, structured outputs, tool design, context management, RAG, memory, agents, evaluation, and production constraints as connected parts of one architecture.

The skills I strengthened include:

  • designing LLM API interactions with structured outputs and validation
  • building model-facing tools with clear schemas and safe execution paths
  • understanding MCP as a protocol for connecting models to external capabilities
  • working with multimodal inputs such as images, PDFs, audio, and video
  • designing context strategies instead of relying on oversized prompts
  • building retrieval and knowledge-base workflows
  • distinguishing workflows, agents, and multi-agent systems
  • planning observability and evaluation for AI behavior
  • thinking through model migration and provider fallback
  • identifying prompt injection and unsafe tool-use risks
  • considering cost, latency, privacy, and rate limits before production
  • mapping business processes into practical AI-assisted workflows

This is the kind of knowledge I’d love to keep exploring: building AI systems that are useful, observable, secure, and grounded in real user needs.

Closing

AI_devs 4 started with model interaction, but the real value was the architectural perspective it built over time.

The course made one thing clear: production AI is not a prompt and not a chatbot. It is a system composed of models, tools, context, memory, data, workflows, people, and safeguards.

That is where the interesting engineering happens.

And most of all: completing the course gave me a great overview of the risks and temptations. Sometimes not doing something is the best answer, right? I’m happy that it will be way easier for me to decide the right path to take with the things ahead of me.

Refactoring: Improving the Design of Existing Code

Well, it took me 10 years to read this book. Seriously. Fowler wrote a second edition during that time, making the book way more relatable for a modern programmer.

I took many courses and matured as a programmer a lot during this time, yet I had so much trouble with this one. The reason is pretty simple: Polish translation of the original book is famously atrocious…

But hey, I’m getting way ahead of myself. Let’s start from the beginning!

At some point of your career you get annoyed by the code you produce. You might figure out what’s wrong by yourself, but when you’re neck deep in it, it’s hard to get the perspective. You can feel the smell, but you’re not always sure where it’s coming from. And Fowler’s book is going to give you that perspective.

I started reading Refactoring in 2025, while it was initially released in 1999. Sounds like madness? Well, some of the parts of the book definiitely feel dated, but the general principles are still valid. That’s because you don’t need to reinvent the wheel (which is exactly why I hate “modern” web frontend tech). The book teaches you that the code needs to be readable – it’s written for humans, not for machines. It also gives you a big chunk of solutions to frequent problems that we all have to conquer.

I’m really happy I finally found the time to grab it from the top of my Pile of Shame. It was a nice refresher and a good exercise. I just wish I had more opportunities to focus on code… It’s funny I recently did some contract work on a hellish landscape that was WordPress with hardcoded WooCommerce stuff. HTML entangled within business logic inside a theme. Like in “good” old times. It would be such an excellent case for implementing the solutions from the book!

Upgrading TeamCity’s Java

Recent versions of TeamCity started to warn about Java 17 support being deprecated. Well, got to update it, right? The trouble is I forgot how to properly override the environment it runs on. It’s super easy when you know where to look…

But first, let’s start with upgrading Java to version 21. Using Ubuntu Server here.

sudo apt install wget gnupg -y
wget -qO - https://repos.azul.com/azul-repo.key | sudo gpg --dearmor -o /usr/share/keyrings/azul.gpg
echo "deb [signed-by=/usr/share/keyrings/azul.gpg] https://repos.azul.com/zulu/deb stable main" | sudo tee /etc/apt/sources.list.d/zulu.list
sudo apt update
sudo apt install zulu21-jdk -y

This should get you a fresh install of Java 21. Now, let’s make it the default – make sure to pick Java 21 from the list when prompted.


sudo update-alternatives --install /usr/bin/java java /usr/lib/jvm/zulu21/bin/java 1
sudo update-alternatives --install /usr/bin/javac javac /usr/lib/jvm/zulu21/bin/javac 1

sudo update-alternatives --config java
sudo update-alternatives --config javac

And that’s it! Verify it’s available by running:

java -version

Next, let’s edit TeamCity’s environment. This is the crucial part.

sudo systemctl edit teamcity-server.service

### Editing /etc/systemd/system/teamcity-server.service.d/override.conf
### Anything between here and the comment below will become the new contents of the file

[Service]
Environment="TEAMCITY_SERVER_MEM_OPTS=-Xmx4096m"
Environment="JAVA_HOME=/usr/lib/jvm/zulu21-ca-amd64"

### Lines below this comment will be discarded

The crucial line is bold, note I used the same method to override memory allocation for Java so it’s better suited for our needs. That’s the only way I found to be actually working for TeamCity. Setting global environment variables, editing internal properties won’t work.

After editing the file make sure to reload and restart:

sudo systemctl daemon-reexec
sudo systemctl edit teamcity-server.service

That’s it! After restarting TeamCity will pick up new Java version.

TeamCity – updating Perforce SSL certificate

I recently updated SSL certificate for a Perforce server. Turns out TeamCity had issues with p4 trust not accepting the new server identity.

At this moment TeamCity’s documentation says to set the teamcity.internal.perforce.forceTrust=trueconfiguration parameter to the related project or build configuration” See for yourself at the documentation. This is simply wrong!

Thankfully TeamCity’s bug tracker has the proper solution. Turns out you have to navigate to the diagnostics tab in admin panel, go for internal properties and set it there. For reference, the link will look like this:

https://teamcity.example.com/admin/admin.html?item=diagnostics&tab=dataDir&file=config/internal.properties#!:config

That’s it! Don’t forget to disable the parameter once the new identity is picked up by TeamCity.

Masters of Doom

Wolfenstein 3D was one of the first games I have ever played. Another one that was installed on the PC I had access to was Doom II. Both of them shareware – I had no idea what that meant back then, and I didn’t care. I just had tons of fun, and I wanted more. Well, I did!

It’s now almost 30 years later and I’m still waiting for id software games. I still hope Carmack will do something cool to surprise us. I still don’t care that much for Romero. I haven’t read any biographies of either of them, until recently. Friend of mine recommended me a book: Masters of Doom: How Two Guys Created an Empire and Transformed Pop Culture by David Kushner, originally released in 2003.

The book itself is a must-read for any gaming enthusiast. It’s well written, clear and fun to read. I just wish it had more technical details, but that’s the price for making the book easy to approach for anyone. Anyway, I don’t want to write about the book, just give it a read, it’s worth it.

I wanted to focus on the big picture of Two John’s career. Both of them came from different backgrounds, and both of them loved games and making them. It’s incredible how different their approaches were. And it’s stunning how their careers developed.

Now, was their story unique? No, not at all. I was mind boggled how similar it was to what I observed for the past decade. There’s no use getting into details, but there’s one observation I want to share and remember: never stop pursuing your goals, but stay sane. Things do change and you have to keep up with them. No one is going to care about you if you don’t.

Laravel Homestead – MongoDB not starting up

I recently had to work on an old Laravel project using MongoDB. Laravel Homestead is a great development environment solution that supports it out of the box. I didn’t expect any issues – it worked just fine previous time I worked on that project. I got the latest version of Homestead, upgraded vagrant box and to my surprise the provisioning failed. On my second attempt I disabled Mongo and it worked just fine. Uh oh!

I spent quite some time looking for a solution. To my surprise there was nothing recent or solid. I started debugging it and I found out the server was unable to start and produced no logs!

sudo service mongod status
● mongod.service - MongoDB Database Server
     Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
     Active: failed (Result: signal) since Wed 2024-02-21 17:18:06 UTC; 13min ago
       Docs: https://docs.mongodb.org/manual
   Main PID: 729 (code=killed, signal=ILL)

Feb 21 17:18:04 homestead systemd[1]: Started MongoDB Database Server.
Feb 21 17:18:06 homestead systemd[1]: mongod.service: Main process exited, code=killed, status=4/ILL
Feb 21 17:18:06 homestead systemd[1]: mongod.service: Failed with result 'signal'.

Interesting, right? The signal=ILL part suggests that the CPU tried to process an invalid instruction. This made my search way easier. Turns out that recent Mongo versions require very specific set of CPU instructions, which collide with Windows virtualization.

AVX – the set of instructions required by MongoDB – is conflicting with Hyper-V, Windows virtualization solution. It might be enabled on your system, especially if you’re using WSL.

Run command prompt / Power Shell as an administrator and issue the commands below to disable Hyper-V:

bcdedit /set hypervisorlaunchtype off
DISM /Online /Disable-Feature:Microsoft-Hyper-V

Restart your system to apply the changes.

If you have Mongo installed on your Homestead and the provisioning (or manual install via /vagrant/scripts/features/mongodb.sh) failed – you might be missing the homestead user. You can add it by issuing the command below:

mongosh admin --eval "db.createUser({user:'homestead',pwd:'secret',roles:['root']})"

That’s it! Have fun.

TeamCity vs Jenkins

We’ve been using Jenkins at Unknown Worlds for over 10 years now. It has been a solid CI/CD system allowing us to handle various tasks around our Unity Engine projects. We started using Perforce again for the new project we’re working on and Jenkins caused multiple problems with it. We had to try a different automation server. We decided to try out TeamCity. I personally love Jetbrains products, so it was quite exciting.

It turned out the systems were quite different. Let’s compare them.

Build nodes management

Jenkins is very simple – you can install the agent (jar file) manually and handle launching it by yourself. That’s it.

TeamCity is much more sophisticated. It offers an agent installer which takes care of heavy lifting. From within TeamCity UI you can clean the workspaces and restart the machine, also access the console.

Scripting

Jenkins uses Groovy for scripting. It’s a very capable system you can program exactly to your needs. It offers a set of pre-defined steps, but also full logic, exceptions, file operations etc.

TeamCity has a different idea on how to handle the pipelines. It offers a list of pre-defined steps, and you manage the flow in a visual way. The scripting itself is made outside of it, but you have way better options – shell, Power Shell, python etc.

Backups

Jenkins is very traditional about it. Since it keeps all the data in flat files, you just back those up. Specifics are listed properly in the documentation.

Backup and config storage are one of the biggest selling points for TeamCity. JetBrains had a great idea – just keep all of it in git! This way we can always roll back or restore the entire TC instance. Brilliant.

Plugins

Jenkins offers hundreds of plugins for almost every problem you might need to solve. They are entirely community driven, so the quality and development speed varies from plugin to plugin.

TeamCity on the other hand has a pretty narrow catalogue. I couldn’t find anything useful in it, but we didn’t really need anything outside of the standard setup.

UI

UI is not the strong point of Jenkins. It does offer two versions of the UI – legacy and Blue Ocean. Both of them are pretty slow, outdated and chaotic. It takes a while to get used to them.

TC on the other hand offers a very modern, responsive interface. It’s almost perfect, but non-technical people require some guidance at first.

Plastic SCM support

Jenkins works great with Plastic. Occasional cleanup is required, but overall experience is great.

I haven’t tested Plastic inside of TC.

Perforce support

That’s why we switched from Jenkins to TC. Workspace management in Perforce is a terrible thing, and Jenkins doesn’t help with it at all. On the contrary – it adds another layer of misery on top of it.

TeamCity on the other hand is OK with it. Not perfect – we still have weird inconsistencies with speed, but it’s OK. Shelves and moving files is still miserable, but manageable.

Upgrades

Jenkins require you to do it from the console.

For Perforce it’s one click from the UI.

Pricing

Jenkins is open source, so you can’t beat that.

TeamCity is pretty expensive. It’s free if you’re using 3 build agents. Anything on top of that is going to cost you. Having 10 agents is 3100 EUR including VAT for the first year, and then the price drops by 50% for the consequent years.

Summary

For me it’s a simple choice. If you can afford it – go for TeamCity. It’s the backbone for your entire company.

Got any questions? Leave them in the comments, I’d be happy to answer.

PowerShell – modifying Task Scheduler trigger

Here’s a small code snippet I wrote for modifying triggers of a specific task – I needed to add a delay for all of them.

$task = Get-ScheduledTask -TaskName "Start TeamCity"
	
foreach($trigger in $task.Triggers) {
	$trigger.Delay = "PT5M"
}
	
Set-ScheduledTask -InputObject $task

That’s it!