Reality-checked Tips for Working in a Distributed Agile Team

Source:, CC0 Creative Commons

Tip 1: Don’t distribute your team members.

Instead, co-locate everyone in a place of their own.


Tip 2: Fight hard for tip 1.

Establish a relocation service if needed, find temporary apartments, … be creative.


OK… now the problem is that, in 2018, people don’t want to work onsite every day anymore. If you, as a company, don’t offer home office, flexible work hours and support for work-life balance you’re out of the hiring game.

And it keeps getting harder to find good people, too. I don’t know about your company but we’ve been looking for talent from all over the globe, and some of them actually want to stay where they feel at home.


Tip 3: Personal contact is key

Distributed agile teams CAN be successful – but you need to meet in person when you start working and repeat that move every once in a while. I cannot imagine doing a retrospective – in a psychologically safe and effective way – with people that I’ve never met in person before. Lunch or drinks work even better.

Here’s a model that I know to be effective:
Each developer starts onsite, stays for a couple of weeks (3-4), returns back home and works there. Repeat this so people work onsite at least twice per year.

As a result…

Tip 4: Professionalize travel and accommodation

Visas need to be renewed, flights need to be booked. You might want to rent company-owned apartments because hotels are expensive. People need to be picked up from the airport, apartment keys need to be handed over, cleaning service must be managed.

Get good at this.

BTW: keep the additional cost in mind. A distributed team is not cheaper per se.


Tip 5: Technology is key, too

Personal contact is the foundation – but you also need decent technology for your in-team communication:


  • Good internet connectivity at each end-point
  • Low latency. It’s super-annoying when someone tries to reply but lags 5 seconds.



  • Good microphones and speakers!!!
    When these are bad you just can’t understand each other which is the end of all communication. And they must be REALLY good because reality makes people with Indian English, German, Russian, Polish, Serbian, Macedonian, Ukrainian, Cockney etc. accent work together.
    For example I learned that “what about the books?” is British “English”(?) for “what about the bugs?”.
  • Good: Headsets
  • Bad: Conference Phones
    If you see any of these (regardless of make and model) throw them away.
  • (There are OK-ish alternatives for meeting rooms.)



  • Every remote participant should have their cam switched on to promote focus and a more personal experience.
  • For the onsite people:
    • You should have at least 1 high resolution camera so you can film a whiteboard when someone writes stuff on it.
    • You should also have at least 1 more cam to capture the room and people. This makes the experience more natural.
  • Alternatively, you might also choose to have everyone use remote access, even those who could potentially meet onsite. See also tip 6.



  • I love physical boards and stickies. However, with a distributed team, you will have to take pictures of them and send them around. As this can get tedious fast you might want to switch to JIRA, Trello, Targetprocess or similar to do this electronically.
  • For some events you might want to try out some specialized tools. Funretro for example helps you do your retros with distributed people. There are tools for doing planning poker online as well like Estimation Poker and many more.
  • Evaluate the right conferencing tool. Google Hangout is reliable and super-easy to use, WebEx also supports telephone dial-in, Zoom allows for handing over your mouse to a remote user, etc. Try some.


Tip 6: Integrate them all

Make sure working onsite and offsite are truly equal – and be consistent about it.

Do pair programming, feedback talks, trainings, etc. with everyone, regardless of their current location. Do career planning with them and promote people based on their merits, not on their location or onsite-time.

Also, you might abolish the concepts of “onsite” and “remote” altogether – make everyone use the same remote technology even if they could meet in person. (As a positive side effect each participant will use headphones so good audio is most certainly a given.)


Tip 7: Ensure “mental health”

A CTO Think podcast reminded me of this: Remote work is different whether you work in a remote office, co-working space, a shop of some sort – or if you work mostly isolated in your home office.

Getting a regular, minimum dosage of human interaction has proven to be important, even for nerdy introverts.


Which tools do you recommend?
Leave your comment, like and share.

The managing n3rd, part I

Source:, CC0 Creative Commons

When you write software, let’s say with a code base of 100,000 lines of code, and you screw up only 1 thing then the consequence might be that the program won’t work. You forget to close 1 parenthesis or 1 line is missing the semicolon at the end – as a result it won’t compile.

When you do sales, let’s say you contact 10 potential clients, and 1 of them eventually signs a 6-figure contract then you’re actually pretty successful.

When the software you wrote does not work properly then the reason MIGHT be because of a flaw in the operating system, some bug in a library you use or even cosmic rays – but in 99% of cases it’s just your own fault. The good news, though, is that it’s also you yourself who can fix it.

When you deal with people, e.g. in sales, line management or simply finding a mate, then your strategies need adjustment.
As a coder you’ve been successful by being critical and especially self-critical. If you apply the same attitude to people related situations you’ll fail.

In management, success is not having it right 100,000 out of 100,000 times. Not even close.

In people situations, failure does not necessarily demand diligent analysis. This can lead to over-thinking and making you feel bad about yourself. Failure does not necessarily demand ignoring it either.

Sometimes it’s best NOT to assume it’s your duty to fix something you messed up. Being smart but still just listening to others, showing empathy for the other instead of mainly trying to fix things – that’s the challenge.

Just as you keep learning new programming paradigms, patterns and languages you can also learn new and different ways to deal with people situations and, as a result, advance in your career to a management position.

I know it’s hard for nerds but we’re smart, creative and friggin’ hardcore. You can do it!


Agree? Leave your comment, like and share.

XLS Management vs. People Management

Source:, CC0 Creative Commons

As a manager, how do you make decisions?

Do you mainly base them on spreadsheets and then just pick the lowest or highest value, e.g. the cheapest product or the highest profit opportunity?

Well, tough news for you: that’s what a machine can do and, boy, can it do it better than you!

If you have no empathy, no understanding of values like trust, courage and integrity or any other skill that a machine does not have (yet) then get ready to be replaced soon.

How to write clean & testable code without losing your mind

At 2017’s edition of the adaptTo() conference I talked about the relationship between testability and Clean Code, Design and Architecture – and how to achieve it in AEM (Adobe Experience Manager) applications.

“If you create software that is to be developed continuously over several years you’ll need a sustainable approach to code quality.

In our early days of AEM development, however, we used to struggle with code that is rigid, hard to test and full of LOG.debug calls.

In this talk I will share some development best practices we have found that really work in actual AEM based software, e.g. to achieve 100% code coverage and provide high confidence in the code base.

Spoiler alert: no new libraries, frameworks or tools are required – once you know the ideas, plain old TDD and the S.O.L.I.D. principles of Clean Code will do the trick.”

See the full presentation at adaptTo 2017

You screwed up. Now what? 8 things professionals should do next.

It wasn't her.

We’ve all been there:
We missed a deadline.
We promised something but didn’t keep the promise.
We let a bug go to production.

Sometimes the effects are just annoying, sometimes they may have a big impact for other people:

  • If your deadline is “before that super important event” and you miss it then, well, an event like the FIFA World Cup won’t be postponed just because you‘re late.
  • If that bug you deployed to production messes up data and you need to restore from yesterday’s backup then you’ll lose today’s data. At least this way you can undo the damage…
  • …but if that bug causes a zillion mails to be sent out containing coupons for a web store then you won’t be able to undo it.
    (Yes, this is based on a true story. Interestingly, although it had caused some financial damage on the first days, eventually, it turned out to be the most successful marketing measure ever for that store. Still embarrassing for us developers, though.)

Sometimes we find out by ourselves that we screwed up.
But sometimes it’s other people who break the news – clients, bosses, co-workers, friends.

So, what should you do?
Here are 8 recommendations on how a professional should handle it:

1. Listen

First, listen to the person who is complaining to you.
Don’t defend yourself, deny it, wipe it off, ignore it or pull an excuse.

Be open.
Mind your body language:  avoid crossed arms, aggressive, exasperated or amused facial expressions.

Listen calmly, even if the other one speaks emotionally.
Showing this kind of respect and interest is the first step to restore your integrity.

You needn’t be mute all the time, though.
Ask questions: “what happened?”, “when did it happen?”, “who was involved?”.
Ask for facts – but don’t ask “why did it happen?”. It will only blur the facts and might lead to premature bias. Plus, it’s actually your job to find out the reasons.

2. Show you understand

Listening is good but you should also assert the other person that you’re not just play “the nodding game”.
Paraphrase what you have just learned.
Show you understand the impact, the problem that the other one is facing now.

3. Be grateful and show it

Always – really: always – say “thank you for telling me that”.
You may not always fully share the other one’s opinion but he/she talked to you – and that’s great.


Criticizing someone is tough. It makes people feel uncomfortable when they do it. Some folks go to lengths in order to avoid criticizing someone else to the face.
Anonymously? “Oh sure, no problem. Let’s post something on the internet…”
By email? “Yeah, and I’ll send it out late so the addressee can’t call me back today.”

But those people who criticize you do give you direct feedback although it may make them just as uncomfortable as you if you were in their position.
Ultimately, this means they care about you, especially if the criticism was face to face.

Be grateful for it and say so.
It will be the first step to restore the relationship between you and the other one.

4. Apologize

Say you’re sorry.
What exactly you are sorry about depends on the situation:

If you know you screwed up personally then say “sorry I screwed up”.

If you’ve just learned about the issue for the first time or you’d like to investigate some more before you “confess” then there’s still something you should always show empathy for:
Someone (who obviously cares about you) is having a problem, so say
“I’m sorry that you’re in this mess now”.

This way you avoid a premature confession but you still continue restoring the relationship between you and the other one.

(If you later find out you or someone else you are accountable for really did cause the problem then don’t hesitate to apologize for screwing up.)

5. Take responsibility

If you are personally accountable then say so.
Continue by promising you will take measures to prevent the problem in the future.

If you are not personally accountable or not sure if you are or who is to blame in the first place then, at least, say:
“I’ll take care of it”.

It’s important to show the other person that the time, emotional strain and courage talking to you were not in vain:
You’ll take care of it, albeit just delivering the information to the people you know are really responsible and will take it from here.

Even more importantly, this step should be the turning point.
So far you have been the “receiver” of information, the more passive partner of the conversation.
Now it’s time to become active.

You can start by saying that what happened is clearly not how you / your team / your company usually work. Continue by explaining what should have happened instead.
Say “I’ll take care of it” and start being the active partner. Assert the other person of your skills and your drive to fix and improve things in his/her interest.

6. Fix it, help fixing it

The most important task should be to fix the problem.

If you can’t fix it yourself at least ask the other one if there’s anything you can help with.

Restoring your integrity and the relationship is selfish if you don’t focus on the problem resolution first.

7. Learn your lesson

It feels strange saying this, and it might upset people who had to criticize me but screwing up is the best driver for learning.

Embrace it.

A lot of the things I know today, maybe even most of them, I know because I learned from having screwed up or being criticized for.

Analyze the problem, drill down to the root cause, research potential solutions, evaluate, implement… this is how I learned about estimations, deployments, hiring, quality assurance, project management, Oracle’s bizarre licensing terms and many more.

Also, use this step to make sure you don’t run into the same mistake or a similar problem for the same reason again. The first time it’s a tragedy, the second time it’s a farce. If you keep screwing up eventually you will lose credibility.

8. Follow up

You said you’d take care of it, so get back to the person who criticized you and show what you’ve taken care of so far, e.g.

  1. The fix. And ask if it worked.
  2. Show you care: ask if the other person’s problem has been resolved or at least mitigated.
  3. You will most likely have new, updated or revised information. Share it.
    Show you’ve worked hard.
  4. Show your lessons learned and the measures you have taken to prevent the same thing from happening again.

What do you think?
How do you handle negative feedback?
Please leave a comment and let me know.

If you like this post then please share it.

Happy hacking!

Image: my wild princess at the age of 2.
(c) Andreas Czakaj, all rights reserved.

100% Code Coverage via automated tests in Symfony applications

On 2017-04-26 the Symfony User Group Cologne gave me the opportunity to speak about Code Coverage.

If you create software that is planned for continuous enhancements and maintenance over several years then you’ll need a sustainable strategy for code quality.
Code coverage by automated tests is one important metric, and its value should equal 100% at all times.
My talk shows why this is so and how it can be achieved in PHP / Symfony based applications.

See the full presentation at Speaker Deck

Natural Enemies of “Moving Fast” in Software Development Projects, Part 1

“Writing tests is the natural enemy of moving fast.”

Some of our dev teams create software using Adobe Experience Manager (AEM, formerly known as CQ).
When you do AEM or Apache Sling then the annual adaptTo tech meetup in Berlin is the place to go.

This year, i.e. September 2016, I heard an interesting phrase.
“Writing tests is the natural enemy of moving fast.”

I was surprised.
Sure, if you’ve written only production code in a project and only later on start writing the tests then that will take a lot of time, it will likely be painful and it will slow down progress.

But the 2nd time I heard the phrase was in a talk about TDD, and that’s when I was really surprised.
I’ve been doing XP and TDD since 2005 and – as far as my experience goes – writing tests (and preferably early) is key if you want to move fast in any project of considerable size (… and AEM projects usually are of considerable size).

But I might be wrong, so let’s do an example coding task.

The irritat0r coding task


Let’s create a functionality that displays personalized texts to web users
in order to irritate the hell out of them
(which is my understanding of what advertising is basically all about).

The text shall look like this:
Hey ${salutation}, did you know that ${message}?“.

    • If the user is known, e.g. a logged in principal, then salute her by the first name.
    • If the user is anonymous then salute her by just “you”.
    • As for the message it shall be possible to send a String message.


  • Hey Andreas, did you know that the longest recorded flight of a chicken was 13 seconds?
  • Hey you, did you know that it is 8 times more likely to get killed by a pig than by a shark?

In version 1 we will implement the following message types:

      • text message, i.e. a fixed text, like
      • message that displays how many kilometers earth has moved around the sun since the user’s birthday, which of course only applies to logged-in users who provided their birthday.

To keep things proper irritat1ng the application shall choose the message randomly from a pool of all appropriate messages.

In version 1 the functionality shall be available as

  • a HTTP response of type text/plain.
  • a JSON representation
  • an AEM component

As for non-functional requirements:

  • Production code shall be 100% covered by tests
  • The code shall be part of a larger system
  • The code shall allow for being changed and extended for new functionalities


Solution Outline

First, let’s do a short design session:

  • We’ll create a Java / maven OSGi bundle called “irritat0r“.
  • We’ll create an AEM servlet called “Irritat0rAemServlet” which allows access to the HttpSession and will send the irritat0r text via the HttpServletResponse
  • We’ll get the user’s id from the HttpSession if HttpSession contains such an id
  • If it does we’ll use the (existing) UserService and get the User object by that id.
  • We’ll create a function that creates the salutation (for known user or anonymous)
  • We’ll create a function that gets 1 appropriate message from the pool of messages.
  • We’ll create a function that assembles the text out of the salutation and the message.
  • (intentionally left blank… will follow in part 2)

So, where should we start?

We could start with the OSGi part

To do so we would need some OSGi container which needs to be set up and started.
Luckily, OSGi requires us to start it only once – but we’d need some way to deploy the bundle into the container.

We could deploy manually

  • Compile the code
  • Package it as a jar including the meta data
  • Open Felix (or Equinox etc)
  • Choose Bundle
  • Upload
  • Restart Bundle

That’s what needs to be done each time we’ve changed the code and we want to see the effect.

I guess we agree that this is a waste of time so let’s state that
“Manual work is a natural enemy of moving fast.”


We could use Maven

(or Gradle etc)

So the manual part is transformed into an automated process. Good.

But we’d still need to run Maven each time we’ve changed the code and we want to see the effect.

You may have become accustomed to this procedure but, believe me, somewhere out there PHP, Ruby, JavaScript Developers and even HTML coders are laughing at a random Java hacker right now.
How long do your Maven scripts take? 30 seconds? 2 minutes? Now multiply that times the number of changes you do to your code.

I believe that
“Deploying to verify code changes is a natural enemy of moving fast.”


We could fake it

There’s a reason why ZERO TURNAROUND has been successful with JRebel, and the reason is Maven. And ant. And, you name it.

Using JRebel would indeed remove the need to run Maven for verifying code changes – at least for code without OSGi annotations. We’ve been using JRebel in AEM projects. It works for “normal” code but it cannot (reliably) redeploy OSGi services and Servlets, so it has been Maven time for us again.

Also, JRebel is not cheap – it will cost you ~ $475 (as of 2016-10-10).
Per year.
Per seat.


But there’s another reason

and it’s less obvious: It’s what happens to the way you think.

You probably know that writing tests for an OSGi service is not the easiest task you can think of.
You might need to install OSGi Mocks (which are in fact stubs, not mocks), Google for code examples and fiddle around with them.
You’ll probably not make the 100% coverage but you’ll have an excuse because, hey, “that’s hard to test” or “that part needn’t be tested”.

Eventually, you’ll get used to a code coverage below 100% before the project even started.

And you’ll be so happy you made it run after all the hard work that you might feel inclined to add more code – i.e. domain logic –  to the OSGi service.
As if it was to make it pay off the pain.

Chances are that this additional code will not be fully covered with tests either, partly because it will be inside of a class that is hard to test, partly because you have already put up with less than 100% coverage.
Test coverage will be the first “victim” of that approach, and the Broken Window Theory will take its toll on you.

Second, putting domain logic into the OSGi service would break Clean Code practices like the Single Responsibility Principle.
Also, as OSGi is the container and hence by definition located on the outside layer of the application, it would violate the Clean Design principle that “All dependencies must only point inwards“.
See also “Ports & Adapters“.

Those violations will eventually lead to messed up dependencies, and, boy, if you really want to slow down development then screwing up your dependencies is a pretty efficient way to do so.

That’s why
“Losing control of dependencies is a natural enemy of moving fast.”

And thus
“Binding domain logic to external devices, ports etc is a natural enemy of moving fast.”

Third, if you need to deploy to verify your code you’ll probably verify your code less often.
If you know that running Maven takes 2 minutes you might not want to run it after each code change.
Instead you’ll keep piling up code changes and deploy them all at once, every 15 minutes or so, and go get a cup coffee while Maven is running.
Sounds familiar?
(Maybe Maven is secretly sponsored by the coffee machine industry, but that’s just a theory.)

Chances are that parts of the code you created within those 15 minutes turn out to be flawed. Sometimes it will be “just a bug”.
But sometimes you will realize that you’ve chosen the wrong approach and now have to delete, change and redo a major part of the code. And as you piled up code changes this might lead to a significantly large piece of rework. That’s of course a waste of time and it will slow down progress.

And that’s why
“Verifying code changes in chunks of more than just a few lines is a natural enemy of moving fast.”.


…and there’s nothing to learn here

Lastly, why would we want to start by testing OSGi?

It’s safe to assume that OSGi works, that’s why we use it in the first place.
There’s nothing to learn about the domain here.

We could start with the Servlet/JSP part

Alternatively we could start by coding the Servlet or maybe some JSP.

However we’d be facing the same problems:

  • We’ll need to deploy each time to see the effect.
  • Testing web output will require additional effort like setting up Selenium, HtmlUnit etc
  • Getting 100% coverage will be hard to do and we might put up with less than 100%.
  • Again, you might want to make the pain pay off and add more code, i.e. domain logic, to the Servlet or – even worse – the JSP.

There may be something to learn here, though, i.e. the look and feel of the front end.


…but you might be F5’ing a lot

Making sure the front end looks the way it should is an important task, and I admit it’s still pretty hard to do so without manual work. (BTW: has anyone evaluated Galen yet? I’m curious…)

However, if that requires running mvn install each time you want to preview you UI then that’s a waste of time. (HTML coders like immediate feedback, e.g. by using Firebug, etc.)

Also, once you have set up this process you might feel inclined to use the UI to also test functionality, not just look and feel.
As a result you’ll be pressing F5 a lot (-> waste of time), comparing the browser’s content to some expectation about the outcome.
In order to do so you’ll be using – your eyes.

First, it’s a manual process and we already know that’s a natural enemy.

Second, it’s exhausting. You might feel as if you were moving fast.
You’re not.
In fact you’re just keeping yourself busy and the way you experience time will change.

Third, your eyes will only see what they are looking for.
If your code change has broken stuff that your eyes are not looking for then you’ll miss that bug.
Let alone if stuff has got broken in other areas of the site, i.e. pages you did not F5. You’ll miss those bugs as well.

And delivering a bug, even more so an avoidable bug, thus making testers report it back and then fixing it, is a HUGE waste of time (cf. Cost to Fix), so
“Bugs are natural enemies of moving fast.”

And that’s why
“Testing domain functionality via the UI is a natural enemy of moving fast.”

First pay-off

We haven’t written a single line of test or production code yet.
However, the Test Driven Development approach has already paid off I think:
TDD kept us from starting with the wrong tasks.
It prevented us from using “Natural Enemies of Moving Fast”.

For some real coding stay tuned for Part 2 which I’ll publish soon and which will include a lot of code examples, I promise.


Spoiler alert

Where should you start?
Start at the core, i.e. the domain logic.

You’ll easily be at 100% coverage right from the beginning and you’ll stay at 100% from that point of time on.

What do you think?
Please leave a comment and let me know.

If you like this post then please share it.

Happy hacking!

The REAL reason why developers need Soft Skills

Developers like to code.
All day, sometimes all night, too.

Developers like playing around with things.
New frameworks, yeah. New tools, whoopie.
We like to see a one-liner have big effects, or a small piece of configuration dramatically change the behavior of a program.
We look for that Arthur C. Clarke moment when “Any sufficiently advanced technology is indistinguishable from magic”.

When it comes to Soft Skills, well, that can wait for later when none of the IMPORTANT stuff is left.

Picture a new development project.

You do some analysis, some solution design and you get a rough idea about how much effort it will take.

Let’s say the effort needed is 2,100 person days.
So if 1 person does all the work, and we assume 21 work days per month then this 1 person will need 100 months to finish the project.

That’s 8 years.

The processes will be super-efficient, though, – no overhead whatsoever!
But you’ll need to wait 8 years before anyone can use the software.
Just think of the major versions of your favorite programming languages, application servers, operating systems, tools, libraries etc 8 years ago and you know how outdated the software will have become by the time it’s done.

And what about that poor guy or girl hacking code on the same project for 8 years using stuff that has become outdated years ago?
Unless you find a way to legalize slavery, odds are that this developer will quit at some point during the project.

OK, new idea.
Let’s have 100 developers work on the project.
This should make the project get done in only 1 month!
I guess you already know that this is a fallacy. Planning, coordination, integration, QA, communication paths etc, well, overhead will explode instead.
And how many of those 100 people did you have in mind when you estimated the 2,100 days?
Probably you were thinking about the effort it takes to design, implement and test the software but chances are you weren’t thinking about all the additional overhead needed to make 100 people work together.
I’m, of course, referring to “The mythical man-month” by Fred Brooks.

So what is the ideal number of people to work on such a project?
Well, it’s more than 1 and less than 100, this much we know by now.
As a matter of fact, our industry is too young to know that number for sure, based on hardcore scientific research at least.

There’s one rule-of-thumb, though, that I stumbled upon some years ago and I like to use in such situations. (I think it originates from the works of Brooks, Rausch-Schott and Retschitzegger but I can’t find the reference, sorry).

The math goes like this:
${ideal number of people on the team} := square root of ${number of person months}
In our scenario the effort was 100 months, i.e. √100 yields 10 people.*
If that was my project I might suggest setting up a team of 10 and let them work for something like 1 year. (Actually, I prefer slightly smaller teams.)

When 10 people work closely together for 1 year straight then they’d better know how to get along with each other.
They’ll need commitment, empathy, respect, reliability, team spirit, collective ownership and focus on the domain. They’ll need the ability to help, criticize and take criticism, make rules and follow them and most importantly:
They need the willingness and the skills to communicate about the project every day. Well. A lot. Even face-to-face.
That’s the Soft Skills required for such projects.

What if you don’t have these skills?
Well, according to the rule of thumb you will be able to work only on projects that can be done by 1 person.
Do the the math, allow some rounding and you’ll get 2 months max. (√2 months => ~ 1.4 people)

So if you don’t take Soft Skills seriously you will be limited to projects that require up to 2 months effort. Maybe 3 months; it’s a rule-of-thumb after all.
But still, if you like to evolve into more complex challenges, well, there aren’t that many projects that are complex, challenging and take just 2, maybe 3 months to get done.

Frameworks, tools, libraries come and go but good communication and team skills will permanently add to your profile. They are the really important stuff you need for complex tasks.


What do you think?
Please leave a comment and let me know.

If you like this post then please share it.

Happy hacking!

* And minus 10 people, of course.
But, personally, I don’t think that firing 10 people will get the project done in 10 months…

Functional DAOs: Amazingly easy way to make Java faster and less hungry for RAM

Some weeks ago I did a code review and I saw something like this in the source:

class UserExporter {
    private final User2CsvConverter user2CsvConverter = new User2CsvConverter();

    private final UserDao userDao;

    public UserExporter(final UserDao userDao) {
        this.userDao = userDao;

    public List<String> getCsvLines() {
        final List<User> userList = userDao.loadAllUsers();
        final List<String> lines = new ArrayList<>();
        for (User userItem : userList) {

        return lines;

So there’s some data storage with “Users”. The userDao (DAO = Data Access Object) is responsible for getting the data and then returning it as a list of Users.
The next step loops thru all those User items in the list we just got from the DAO and executes some magic on it, i.e. in this case it converts each User object into a String, obviously in order to export all those this line into a CSV file.
Each CSV line will, again, be added to a List of Strings.
Later on the actual CSV export happens:

    int process() {
        final List<String> csvLines = userExporter.getCsvLines();

        int linesWritten = 0;
        csvWriter.writeLine("First Name;Last Name;Email");
        for (String line : csvLines) {

        return linesWritten;

So the List that the UserExporter.getCsvLines() method returned will, again, be looped in order to write each line to some external output device, e.g. a file or System.out.
The code is straightforward, it has a clean layering (DAO abstracting the storage, CsvWrite abstracting the actual output device) and it even has 100% test coverage.
So what the heck could I possibly be complaining about, right?

Well, first of all not only have I seen a lot of similar code like this but I actually used to code like this myself.
Also, this is obviously a simplification of the real code (which I cannot show you, of course).

There are 2 important things to know about the real code:
1) The real data storage is pretty big so in some cases the DAO may return a lot of items.
2) The real code does much more magic on the lists.
As a result I know that we shot the server a bunch of times when we started big exports. It struggled for hours and then failed with OutOfMemory Errors. (And as web client users got impatient they repeated the export some more times only to find out that it – surprisingly – didn’t make the server work faster.)

Someone suggested the usual “we need more RAM” thing but there’s an easier and better solution at hand.
The biggest problem are all those ArrayLists that are being created all over the place.
The DAO creates one, passes it to the CsvConverter which “only” loops it but creates its own ArrayList and so on.
Everytime you use up a lot of memory for those Arrays and a lot of CPU for looping.

For the complete original, imperatively coded example, cf.

So how can we fix it?
Let’s get rid of the lists!
How do we do it?
We use functional style programming.

Here’s how you can use Java 8 to get an equivalent solution with NO (in numbers: 0) Lists at all.
Instead of a returning a List, let’s make the DAO take a Java 8 Consumer that will be called back for each “User” item:

interface UserDao {
    void loadAllUsers(Consumer<User> userConsumer);

Now the Exporter can do this:

    public void getCsvLines(final Consumer<String> lineConsumer) {
                (user) -> {
                    final String line = user2CsvConverter.convert(user);

It calls the DAO and for each User item the dao “returns” (line 3), i.e. has to “offer”, the converter can do it’s magic (line 4). No List, no loop needed.
The CSV line that is being created for each User item doesn’t get added to a list either. Instead it will be “returned” or “offered”, well, handed over to a Consumer that is called back for each CSV line (line 5).

This String Consumer is passed to this method by the outer layer of this application:

    void process() {
        csvWriter.writeLine("First Name;Last Name;Email");

Again, no Lists, no loops.

When you use this approach you kind of execute the “front end”/output code when you access your data while still keeping a clean architecture.

Or, as another way to look at it: In imperative programming, when your method returns a value, you force the method’s clients to “live” with this value no matter what. If that return value happens to be a big Collection then there’s no other way for the client but to use this Collection with high memory consumption and all.
But if your method takes a Consumer instead then the clients can choose how to handle the data. If all they need is just a list after all then they can pass a plain and simple consumer that will do the trick. Actually that’s what I do in my UnitTests (line 4+5):

    public void testLoadAllUsers_ok() throws Exception {
        UserDaoJsonImpl dao = new UserDaoJsonImpl("/users-integration-test.json");
        List<User> data = new ArrayList<>();
        assertEquals(3, data.size());

But if they want some special handling, like in this example application, then the clients will have the freedom to choose a more clever way.

This gets even more exciting when you think of how to handle special cases:

    public List<User> loadAllUsers() {
        try {
            final URL resource = this.getClass().getResource(path);
            if (resource == null) {
                throw new FileNotFoundException(path);
            } else {
                final InputStream is = resource.openStream();
                final ObjectMapper mapper = new ObjectMapper();
                return mapper.readValue(is, new TypeReference<List<User>>() {});
        } catch (IOException e) {
            throw new RuntimeException("loadAllUsers: IOException e", e);

In this solution, when an IOException occurs, the loadAllUsers method throws a RuntimeException hence forcing each client code to handle it.
Alternatively the method could have caught the IOException and logged a messages to a logging framework.
Or it could have swallowed the IOException exception hence leaving no clue for client code about what happened.
Whatever the solution we choose – the client code will just have to accept it.

Now think of giving the client code freedom to choose how to handle special cases:

    public void loadAllUsers(final Consumer<User> userConsumer) {
                e -> {throw new RuntimeException("loadAllUsers: Exception e: " + e.getMessage(), e);}

    public void loadAllUsers(final Consumer<User> userConsumer, final Consumer<Exception> errorConsumer) {
        try {
            final URL resource = this.getClass().getResource(path);
            if (resource == null) {
                throw new FileNotFoundException(path);
            } else {
                final InputStream is = resource.openStream();
                final ObjectMapper mapper = new ObjectMapper();
                final TypeReference<User> valueTypeRef = new TypeReference<User>() {};
                final JsonNode jn = mapper.readValue(is, JsonNode.class);
                final Spliterator<JsonNode> jsonNodeSpliterator = jn.spliterator();

      , true)
                        .forEach(jsonNode -> {
                            try {
                                final JsonParser jsonParser = jsonNode.traverse(mapper);
                                final User user = jsonParser.readValueAs(valueTypeRef);
                            } catch (IOException e) {
        } catch (IOException e) {

In line 28 an IOException is caught. The client of this method can now choose what to do in such an event, cf. line 5:
It can choose to throw re-throw the Exception, aborting the whole export if only 1 User item is broken, as it does in the example above.
Or it could choose to ignore broken items, just log them and continue:

    public void loadAllUsers(final Consumer<User> userConsumer) {
                e -> {LOGGER.error("An oopsie occurred: " + e.getMessage(), e);}

Or, how about a progress indicator that should be triggered on each item in order to output some info?
Just add some code for that to your Consumer.

    public void getCsvLines(final Consumer<String> lineConsumer, final Consumer<User> progressConsumer) {
                (user) -> {
                    final String line = user2CsvConverter.convert(user);

For the full code please cf.

OK, I can hear you: “We’d like to use Java 8 but they don’t let us”.
Fair enough, but actually I have been lying to you – there’s hardly any “real” Java 8 in the solution I’m proposing.
You can use the same approach and architecture in Java as early as probably 1.0.
Sure, there’s no Consumer interface in Java 7 but so what: just create it yourself.
Actually I should have done that in my Java 8 code, too, because now my public methods have a dependency to java.util.function.
This is generally not the cleanest way to do it, just as you should not use Maps, Lists or Sets in your public method signatures.

interface UserConsumer {
    void doOnUser(User user);


interface LineConsumer {
    void doOnLine(String line);


class UserExporter {
    private final User2CsvConverter user2CsvConverter = new User2CsvConverter();

    private final UserDao userDao;

    public UserExporter(final UserDao userDao) {
        this.userDao = userDao;

    public int getCsvLines(final LineConsumer lineConsumer) {
        final int out[] = new int[]{0};

                new UserConsumer() {
                    public void doOnUser(final User user) {
                        final String line = user2CsvConverter.convert(user);
                        out[0] = out[0] + 1;

        return out[0];

So you can use the functional way in Java 7, too. You just cannot use lambdas.
And, the code looks somewhat awkward, right?
(For more details cf. the complete Java 7 solution)

That raises an interesting question:
If high memory consumption and expensive loops in Java have been bothering us for years while there has been a way to solve it using Callback interfaces – why aren’t we using them in Java 7 all the time?
Is it because developers are too lazy to write those extra lines of code? Hmm.

Maybe it’s just because it feels wrong.
The Java 7 solution just looks unnatural, unfamiliar, it feels like you’re coding wrongly.
So is using your intuition the problem?

Of course you should listen to your intuition.
When you start coding you need to realize what you don’t know in order to find out what to learn.
You will travel from this “unconscious incompetence” to “conscious incompetence”, then “conscious competence” until you eventually know all those things while you’re not even aware of them (“unconscious competence”). (See Four stages of competence for more about this)

It’s this unconscious competence that’s yelling at you from the inside when you’re about to do something awkward, weird, and most of the time this intuition is right: you’re about to do something wrong, so stop it.
But sometimes it’s just another unconscious incompetence.
I learned imperative style programming some centuries ago so it feels natural and familiar. Functional is new so of course it feels strange at first.

And, programming is using a language to utter ideas. So just like human languages, programming languages influence the way you think. If your programming language has no “natural” support for functional programming it’s no wonder you don’t think of it all the time.

So I think we should go easy on ourselves. It’s OK to do things like imperative programming for a while and then improve your style later.
If it wasn’t OK how would we ever start? There will always be a next big thing we cannot afford to wait for.

In fact, I try utilizing this effect: I try to make the good things easy, familiar and natural to use while making the bad things hard.
For example, decoupling by using interfaces is a good thing.
I encourage my developers to create dependencies only toward interfaces. That’s why I want interfaces to be named in a pretty way, like “Exporter”, so it “feels right” to use them. If they were called “IExporter” or “ExporterInterface” then developers might avoid using them.
Implementations, however, should be called in a way that makes developers know immediately that they should not bind their code directly to something like an “ExporterJava8FooBarUsingJsonCsvImpl”.

What do you think?
Please leave a comment and let me know.

If you like this post then please share it.

Happy hacking!