Personal experience: test task to search for a junior product manager vacancy

My name is Herman, I'm Head of Product at CSI (Crystal Service Integration) - we are developing a customer data platform (CDP) for retail marketing called Set Loyalty. Recently we decided to expand our product team and I needed to develop a test task to select the right candidates for a junior product manager position. It was the first time I’d done this, and I decided I should share the principles that guided me in its preparation. I also show the test itself, it has already helped us find a new colleague. 

Our team has members with deep expertise in retail. However, we also lack colleagues with real experience in using tools for marketers, such as advertising, communications, analytics, CRM, or CDP platforms. There are several questions in the test task, aimed at making a suitable candidate demonstrate our required qualities, namely: expertise in tools for marketers and willingness to apply their knowledge in designing solutions for the product being developed. (This is question 4 in task 1, and questions 2 and 3 in task 2 - the entire test is below, without context the questions will be unclear). 

I selected tasks that I think will occupy up to 70-90% of the candidate's time after hiring. This approach is fair, as it is immediately clear to the candidate what they will have to do in the team. 

The test questions are quite difficult to answer without knowing anything about the product. We provided information about the product - the target market, the user, the top-level structure of the menu sections and the key features. We didn't mention one part of the product, because it could immediately lead the user to the right answer. 

Our test consists of two tasks. Both tasks are real issues that had to be solved at a previous stage of product development. My team and I spent a lot of time discussing possible options. I was well aware of the pros and cons of all possible solutions. This helped me to evaluate the candidates' answers less subjectively. 

Our test didn't seem very time-consuming to me. Candidates told us they took between 3 to 8 hours to complete it.

We were looking for a junior product manager. We had 39 responses to the vacancy and sent a test job to four candidates. 

We received responses from all four candidates. The answers were informative, with reasoning and constructive argumentation. Three out of the four answered the key question of the task description in the same way as the team answered it in real working circumstances (I should note this was not a selection criterion). Two of the candidates demonstrated the qualities we were looking for in their answers: familiarity with tools for marketers and the ability to use this knowledge in designing solutions.

We made an offer. Interestingly, I was initially quite skeptical about this candidate: their resume did not contain any mention of the tools I needed them to be proficient in, but in the test they demonstrated this proficiency and offered a solution based on specific tools. 

It's good to note that candidates in interviews also responded positively to the test task: 

"Thank you for the test. It was very interesting and consistent. Everything was laid out in detail. It was a pleasure to do it. And it was even more pleasant that it really correlates with the activity to be engaged in. I haven't come across anything like that before." 

"It was nice that the introductory information about the product, the target audience, the structure was quite detailed. It was interesting to do such a test task" 

You will see the task itself below. Let me just say that this experience will definitely come in handy for our future hiring of product team members - and maybe for someone else who is currently thinking about their test task? 

Let us know what tasks you've gone through when hiring for positions in product teams in the comments. Or what tests you give to your candidates :) 

Message from Head of Product Set Loyalty 

"Hi! 

Earlier I told you about our cashbox mechanics editor. In the cashbox mechanics editor, the Set Loyalty user customizes the rules of how a purchase promotion works. For example: 

1. Buy 2 packs of Country House milk for $1.

2. Buy "Dutch" Cheese 200 gr. and "Healthy" Sausage 300 gr. in a package and get twice as much bonuses as usual by the card - and so on. 

So. 

We have just been approached by a retailer called First Store. They are using our product and they have a promotion, similar to the promotion number 2 above, set up through this editor. But now they need to send a message to the customer's phone right after he took part in the promotion - no matter in which channel the customer made the purchase - online or offline. 

The approximate text of the message is "Way to go, Steven. Now you have 234 bonus points on your account. We are waiting for you for new purchases in the First Store chain". This message would be assigned to each user. 

The scenario, I think, is valuable - can be useful not only "First Store". But from my experience I can say that the task is not as simple as it seems at first glance. I suggest that tomorrow the product team meet and discuss the approach to designing settings for the realization of such a scenario. 

Please prepare your position by answering the following questions in detail: 

1. What is the best way for us to go - to add a new stage with mailing list settings to the stock editor (cashbox mechanics) or still make a separate editor with mailing list settings? Why do you think so? 

2. Is there enough information above to design mailing list customizations? Do you need to ask any clarifying questions about First Store? What questions do you want to ask? Which employees on their side should be asked additional questions (in which positions)? 

3. Is the information from "First Store" alone enough to design a quality solution that adds value to our product for all users, not specifically for "First Store"? Why? 

4. What stages of mailing list customizations should be emphasized? What kind of customizations do you think the user should set at each stage?

But remember, we are not a design company and are not obliged to solve the user's task the way they tell us to. We have the right to develop an optimal solution, which at least - will not harm our other users, and at most - will be very useful to them" 

Message from Head of Product Set Loyalty 

"Hi. 

We've been approached by marketers from a retail chain called Second Store. And they need a new type of segment creation in our system. The situation is as follows. 

You're already aware that this is a pretty big client. And large retailers tend to use BI systems for advanced analytics of buying behavior. Such users pour into the BI system a very large amount of information from a variety of sources: CDP platforms, communication services on the site, website, mobile application, services for collecting feedback and many others. Then marketers build reports, graphs and tables in the BI system. A large amount of data allows marketers to see a bigger picture than the analytical tools of each of the above services can provide separately. 

So - Second Store now wants to generate a list of customers in this BI system and upload a list of phone numbers from it. And then add them to a file, upload it to our system, form a segment and use it when setting up conditions for promotions, communications and for other tasks in our system. 

This is the first such request to us. And I personally find it ridiculous. I have not seen such functionality anywhere and I am not sure that it can be of value for marketers. 

But nevertheless we should discuss it within the team. If we recognize it as valuable for all our users-marketers, we will make it free of charge in our product. And if we don't recognize it - we will ask "Second Store" to compensate us for the costs of implementing this functionality. In general - we should discuss it at tomorrow's meeting. 

Please prepare your position on this issue by answering the following questions:

1. Do you think there's value in this feature for marketers (all of our users, not just Second Store)? Why? 

2. What customizations do you think a user should make from the moment they initiate a script to create a segment this way to the moment they hit the "Save" button? 

3. Draw draft mockups of a window with the settings you wrote about in the previous paragraph. Don't worry about the drawing quality - just draw with pen on paper and without details and send a picture here. 

4. Do you think there should be any requirements to the uploaded file? Why?