Anton Osipov
•
Growth...
Senior Product Designer
CV
Email
•
Growth...
Senior Product Designer
CV
Email

Growth through tryouts

Specifics of testing hypotheses in a growing startup

Growth through tryouts

Specifics of testing hypotheses in a growing startup

Unreliable data

Metrics are crucial for every company. Nevertheless, their importance changes correspondingly with the amount of users you have.

When there aren't enough users, data is almost random. Out of curiosity, I monitored a conversion funnel and, for the first half-year, mostly saw noise.

That's why the primary strategy at this stage was testing through implementation.

My primary focus at this point was speed. Every week, ~5 features and updates were implemented. 

To keep up such speed, I used a few techniques:

  •  Quick design iterations with feedback. I sought feedback after every design iteration and did so as often as it made sense. Feedback could be collected from the team or random people

  •  Switch to a parallel task when stuck on something. This allowed me to break a loop of thoughts that are formed in such cases. Finding a solution was much easier when I returned to the initial problem

  •  Simplify until making it simpler is impossible. The same feature can be designed in different ways. To move fast, I had to choose the simplest possible UI, layout, and logic (even when it slightly reduced UX)

Thanks to the iterative approach, the product drastically changed in a year. It started generating revenue, and I've got enough users to use A/B tests.

Unreliable data

Metrics are crucial for every company. Nevertheless, their importance changes correspondingly with the amount of users you have.

When there aren't enough users, data is almost random. Out of curiosity, I monitored a conversion funnel and, for the first half-year, mostly saw noise.

That's why the primary strategy at this stage was testing through implementation.

My primary focus at this point was speed. Every week, ~5 features and updates were implemented. 

To keep up such speed, I used a few techniques:

  •  Quick design iterations with feedback. I sought feedback after every design iteration and did so as often as it made sense. Feedback could be collected from the team or random people

  •  Switch to a parallel task when stuck on something. This allowed me to break a loop of thoughts that are formed in such cases. Finding a solution was much easier when I returned to the initial problem

  •  Simplify until making it simpler is impossible. The same feature can be designed in different ways. To move fast, I had to choose the simplest possible UI, layout, and logic (even when it slightly reduced UX)

Thanks to the iterative approach, the product drastically changed in a year. It started generating revenue, and I've got enough users to use A/B tests.

The age of A/B tests

The hypotheses for the tests came from various sources: my own ideas, other team members' ideas, user interviews, Hotjar recordings, and Userbob tests.

The payment modal was an interesting example of A/B test usage.

It consisted of mandatory elements that couldn't be removed, so I could just tweak them a little. Experiments started from changing texts (the title and the description) and visual appearance later. None of them led to a significant improvement.

Meanwhile, moving a few words to a more visible location made a huge difference.

Adding a link to a pricing page was even too successful.

Half of the system functionality was blocked for free users. These pages contained explanations and a link to a payment modal.

I mentioned that there was no easy way for a logged-in user to check prices. One way was opening settings (in 3 unobvious clicks), and another was through the blocked functionality (also in 3 unobvious clicks).

I added the corresponding item to a user menu. Out of curiosity, I added a highlighted variant, which won with an unexpected advantage.

I had a few ideas of making it less extravagant but hadn't the opportunity to test them.

Results

Tests became an essential part of the product evolution. When I left, there were ~30 simultaneously ongoing A/B tests in production.

Thanks to endless experiments, the product changed drastically. From a confusing, sloppy design at the start, it became a solid product with quickly growing revenue.

All NUMI cases

Mar 2022 — Present