It’s a familiar sight to any train traveller in the Netherlands - the dreaded delays that always seem to inch upwards. First, a five minute extra wait time and then the next thing you know you’re looking at 10 or 15 minutes longer before your train arrives.
The duration of the extra wait might appear arbitrary to most travellers, but behind the scenes ProRail is assigning delay times based on their model for prognosis. However, the national railway operator knew their response system wasn’t optimal for the company’s business model or their customers.
ProRail reached out to Xomnia to help them more accurately predict how long it takes to resolve an incident. They wanted to provide better information on the most logical steps to take leading up to the incident resolution.
Together we tested out different machine learning models. That’s when the team discovered a major flaw that derailed the entire approach.
The problem with the problem
ProRail was relying on a decision tree type model to make an initial prognosis following an incident, but users were already skeptical about the accuracy of the model.
Pieter Kouyzer, lead Data Scientist at ProRail, explains that the team first decided to try providing a better prediction of potential incidents or at least reveal the inaccuracies of the current model.
“We initially focused heavily on including as many features as we could, so we could see what could improve the prediction. However, as we started out testing different machine learning models, we could not help but conclude that the accuracy of our model was really bad. This was due to the large span of time predicted for the resolution.”
The team found the same inaccuracies in the initial prognosis model and their added features had little impact on the accuracy. That’s when Pieter says they decided to switch tracks.
“We didn't focus on making the best possible machine learning model for the problem we were given. Instead, we reformulated the problem so we could have more impact for the business.”
How they solved it
The Xomnia and ProRail team tackled the challenge of the broad time range for a repair in two ways. First, because very little information is available at the onset of the incident, predictions are difficult to make at that moment. However, when the mechanic arrives or the cause of the incident is determined, new information becomes available. So, the team created a model that can be updated during the resolution process.
Next, they accepted the wide ranges of the repair times, but also realised that different categories of incidents came with different ranges. Pieter explains that the team decided it would be interesting to give the time span as part of the prediction. For example, instead of predicting 100 minutes, they predict a repair time of between 70 and 120 minutes with a 50 percent chance of falling within that range.
“Communicating our uncertainty is both more honest and allows users to make decisions more in line with the unpredictability of reality, instead of assuming the single prediction must be accurate in order to execute their job.”
By combining the two methods, the team is able to predict a time range for the repair and update it throughout the process.
The key to better machine learning solutions
The model is still being expanded and improved, but ProRail is happy with the initial results. Pieter says tailoring the solution to the problem at hand enabled the team to provide a more useful machine learning solution.
“It is easy to get caught up in optimizing the problem in front of you. By talking to our product owner and stakeholders extensively, we managed to reframe the goal of our project in such a way that we were able to add value. Not by slightly improving the answer to the question asked, but by altering the question so the answer could improve the incident process.”
Xomnia junior Data Scientist Roshell Haaker says the project has also helped the company gain a clearer vision of what can be achieved with data science.
“ProRail has a lot of data on incidents available, but it isn’t being communicated in the same way throughout the company. Now they can see how creating a cohesive process for gathering data will make it possible to utilise more advanced methods in the future.”
Pieter’s advice to Data Scientists is to always keep an eye on what the business needs.
“Quite often it’s not the flashiest coolest new neural network and doesn’t appeal to our optimisation and coding instincts. But, by providing these simple solutions and making sure they are implemented, we set the stage for delivering more effective and technically challenging models further down the road.”