We sat down with Elad Yaniv, Business Analyst Team Lead at Natural Intelligence, to find out how his team has been using geo-optimization to learn more about users—and boost earnings per visit.
Every time a person visits your website, they leave a footprint that reveals information such as their IP address and physical location. Geo-location tools have been around for a long time, but being able to use a visitor’s location to deduce user behavior is now gaining steam.
Listen to the interview here:
Interview Transcript
NS: Welcome, Elad. Before we begin, could you please introduce yourself and tell us briefly about your role at Natural Intelligence?
EY: I’ve worked at Natural Intelligence for the past 3 years. I started in the strategy department and then moved into analytics, where I serve as a team leader. Our primary focus is to help other departments, such as product, content, and biz dev. The different focus is to have some innovation of our own and to look for trends and user behavior. This way, you can give them a better experience and hopefully generate more profits for the company.
NS: I understand you’ve been working on a project where you optimize sites based on the geographic location of the user, known as geo-optimization. Could you define geo-optimization?
EY: Geo-optimization is related to a broader thread around personalization for our users. We want to give our users the most personalized experience we can provide them. Still, of course, there is a tradeoff between the personalization you can give the user and the number of experiences you want to give in total because you have to be able to maintain those experiences.
We decided to try and optimize our websites for the users based on their geo. We started with the meal delivery website. Unlike other (Natural Intelligence) products, the meal delivery website actually delivers real, physical stuff to the user’s address, and is not just an online service. So we thought it made sense to start with this website.
NS: When the user arrives at the main landing page, they see a chart with 5 or 10 top meal delivery companies. When you practice geo-optimization, does this mean the landing page is determined by the user’s location?
EY: Yes, that’s the case. We noticed that some of the brands we show on our charts are more or less popular in certain regions. For example, we saw that some brands are very popular in California. We can tell the popularity of the brands per region by the number of users that click on these brands on our charts and by the number of users that actually buy (conversion) on the website of the brand we send them to. We noticed that the popularity of the brands differs between states in the United States. That led us to the conclusion that we can give a better experience to our users—which is a win-win for ourselves and the users.
We started by mapping the brands and making a list of their popularity by state. We pulled the data using a relatively large SQL (language used in programming for managing data) query. Once we had the data, we worked on it in Python. The first stage was to create a table with the states represented in rows and the columns holding data about those states.
The final part was to group states into clusters [based on how users behaved in different states] and then give each cluster a unique experience. We could give every user on our website a very personalized experience. But the tradeoff is we would lose resources that are needed to maintain the experience. So, we decided to create clusters of states, which would avoid the need to maintain 50 experiences, but instead have 5 experiences, for example.
There are various statistical methods you can use to create clusters. The most common is the K-means statistical model. After reviewing the options, we decided to create the clusters by ourselves, and the first step was to choose big states to lead those clusters. For example, if Texas and California are different in terms of user behavior, the number of clicks received by every partner on our chart, or conversion rate, then we would assign them to lead different clusters and place smaller states below the lead states according to the similarity model we implemented.
NS: When you analyze each state, is it based solely on the user behavior you observe or are you also able to predict behavior based on theories. For example, people in California being geared toward organic or vegan meal plans?
EY: We had some theories, as you said, about people in California. And we all know that people that live in California are probably more aware of fitness or vegan food. But we based our analysis on the actual results, the actual behavior of users coming from those states. Sometimes, not always, those behaviors did match our theory. I don’t remember the exact names. Still, we saw popularity for certain brands that we expected would be popular in California. So, sometimes, the user behavior matched our theories in those states. But the main thing was to build the model around the real results of our users.
NS: Was this the first time you had worked on geo-optimization? Will you run it on other sites?
EY: We tried to run the model on a few sites. We chose sites where we expected the biggest impact on profit. The main site was meal delivery, which is one of the biggest sites here at Natural Intelligence.
NS: Did the experiment meet your expectations? And what KPIs did you use to measure results?
EY: We run a vast amount of A/B tests here at Natural Intelligence. In this case, it was no different. We test every change we make to our sites by splitting the traffic 50%-50% between the 2 experiences we provide. For example, if we want to change the chart content, we direct 50% of users to the original experience and the other 50% to the new experience with the new chart.
We tested a few KPIs in order to determine if the test was a success or failure. The main KPI is the average amount of revenue we generate per user, called EPV (Earnings Per Visit). We look at a few other KPIs, such as the click share of the partners on the chart, i.e., each partner’s share in the total amount of clicks on the site. And we look at the conversion rate of the partner. If, for example, we wanted to switch partners on the chart, we would test if this rotation changed the conversion rate positively or negatively.
NS: With the meal delivery experiment, did you see a significant improvement in the KPIs?
EY: Yes. Meal delivery was a success. We did see an improvement when we changed the charts according to the results of our model. We tested it in the states where the model recommended making changes. The test was a success, fortunately. We increased the EPV of the site, and this is the main KPI, so we were pretty happy about it.
NS: What sort of team do you need in order to do this kind of work? How many people do you need, and what sort of roles and expertise?
EY: Firstly, you need to be very familiar with the data you’re analyzing. It’s easy to be deceived by other factors when you’re checking the behavior of users in different states. Many things can influence [your data], so you want to isolate only the effect of the state on the users’ behavior. In addition, I think technology is a must. We could have tried doing this with tools other than Python, but we saw that Python gave us a few options in terms of analyzing and playing with the data. For example, when we wanted to clean the data, we were able to filter out some outlying information easily. When we tried to create the groups [clusters] and add more states to those groups, we used a model that finds out the similarities between one state and another. This helped us to check if they should or shouldn’t be in the same cluster. All those things were done pretty easily with Python, and I don’t think they could have been done with other tools in the same easy way.
Analyst-wise, it was only me with the guidance of my manager. Before we launched this product, we had some meetings with the Biz Dev guys in charge of the meal delivery site, and the campaign manager of these sites. We were looking to better understand the users’ behavior and what we should expect to see in the data—for example, the behavior of users from California.
NS: For anybody out there who might be thinking of geo-optimization: Any tips? Or, conversely, anything to avoid?
EY: I think before even trying to think about this kind of project, you have to check if you’re able to maintain multiple experiences for your audience. If the maintenance is going to cost more than the benefit you expect to receive, then this is useless. Secondly, you need to make sure you ignore the noise that you can see in the data from other factors. You have to make sure—if you’re saying there’s a difference between California and New York—that the difference is only a result of users being in different states and not because of their age, gender, or any other factors that may influence the data.
Other than that, you should make sure you have the technical skills in order to build this kind of model. I can tell you that during the process, there were some ups and downs. We changed the methodology a few times when we saw that changes should be made. It’s not a very easy or short process. Still, in the end, I think it can be a win-win situation for the company—for Natural Intelligence and the users.
Want to keep the conversation going? Click here and fill out the short form to connect with our marketing specialists and learn more about optimizing your product or service to user location.
Contact our marketing specialists
