Hello from the other side! A small insight in casino operations

[Melvin - Panda]

I was recently thinking about giving you guys some little insights into casino operations. If you guys like these kind of posts, I might do some more in the future.

We’ve had some minor issues regarding our website two weeks ago. In the end this resulted in 2x 2 hours of downtime/big issues and a couple of times for a few minutes of lag/crashing game rounds. Things seem to be solved and stable again, but we will continue to monitor the website. These things always suck for you as a player, and you have all the right to be angry as well. I thought you might be interested in how we as an operator experience such issues.

From a player point of view:
The site is not working/lagging/slow/bet is stuck. I feel cheated, I want my money back, I want to play somewhere else. *&#^* this casino!

From an operator point of view:
Players are having a bad experience, while we always want to offer them the best experience possible. Players might not be able to play at the moment = loss of revenue, and there is also the added problem that the players who have a bad experience will go away and stay away. This is very bad for the reputation of the casino and the eventual loss is hard to measure but probably substantial.

An example of what can happen in practice:
If it’s a small problem, you might get players in live chat initially reporting the problem. The support tries to help out. If it’s the first time they hear about such a problem for the day, they might ask you to refresh a browser, clear cache or something similar. (a lot of times, the problem is not the website but on the user end). They will also try to replicate the issue themselves.

On top of that, they will make a note of it and discuss it internally. If more players report the same issue, they will start a further investigation and escalate it to other departments. If the issue is related to one game provider, those games might be made temporarily unavailable, to prevent other players from experiencing the same issue.

They will also contact the responsible IT people/person on duty to help investigate the issue.

In case of very serious issues concerning the entire website, alarm bells will go off. No matter what time of the day, people across the globe will be pulled out of meetings, cinemas, pubs, bed and make it their #1 priority to resolve the problem. Sometimes a site reboot does the trick and players can continue playing again. Other times it’s a lot harder. In our case, we had over 10 people in five different countries accros europe, north america and asia working on the problems. One of them was actually on holiday, but ended up spending a lot of time on helping to solve the issues.

Even if the problem is at some point solved, the issue doesn’t stop here for us! There was a problem, and even though it’s solved for now, we still might not have a cause. Since we made the problem go away to make the website function again, it might also be hard(er) to find the cause of the problem. At this point, it is still a top priority for everyone to find out what caused the problem and help fix it. People are still working around the clock for this. Our support will also go back to players who had issues and find solutions and in some cases a compensation.

At this point, the website might be up and running again, and anything they do can also affect the live playing environment. It’s a very tricky job that I don’t even know all the details about, but the people working on it are experts and do anything they can. The problem with IT issues is that you only know there is a problem. It’s up to you to find out what it actually is and what causes it, and with lots and lots of code in many systems and servers, it’s sometimes hard to find out exactly where it’s gone wrong.
Meanwhile, players might be playing again but feel mad/cheated, or they still have issues which were caused by the problem. Maybe a bet round got stuck, they didn’t get a win added or anything else. It’s now the support’s job to help out the players as well as they can.

Note: Game results are never influenced by tech issues. Either the bet gets placed and registered and the round will be played (this happens on the game provider end, not on our servers), or the bet will fail to register and you will get back the bet amount.

Then once the developers have indeed found the problem and fixed it, everything returns to normal.

…But was it really the correct problem they fixed? Did the fix possibly affect something else? Were there other issues which they haven’t found yet? Only time will tell

Please let me know if you have any further questions, and if you like such posts and insights, I might do some relating other casino operation issues.

 

[Jono777]

Very good indeed!

I'm sure many others (as I do) will appreciate these sort of posts and threads (where the information can be safely shared)

The problem you describe is indeed in my top two 'hates' more so if its a bad session (although I am not a sore loser, honest)

The player reaction(s) are also well described lol and if others are honest I'm sure we've all been there at some point during our play!

The Problematical part is what to do. Its fantastic that casinos pull out 'all the stops' to rectify the problem for all reasons (player satisfaction, business perspective etc) but I truly don't feel that there will ever be a fix for the player(s) who have just deposited the last £25 of their weeks gaming allowance, only to be met with constant lag and server disconnections.

Casinos cannot continually hand out comps for every gremlin that enters the works and although appreciated when I am 'in this boat' it does nothing to fix the 'here and now'

Great thread, look forward to more

 

[Steviedoo]

Thank you for sharing...I for one am fascinated by the behind the scenes...


I was wondering...when you say it could be your servers...would that be the casinos or the game provider servers....how does this work....I thought games are hosted on game providers and not your own servers...

 

[Tirilej]

More of this kind please. You're also very good in explaining so it's a pleasure to read

 

[Melvin - Panda]

Thank you for sharing...I for one am fascinated by the behind the scenes...


I was wondering...when you say it could be your servers...would that be the casinos or the game provider servers....how does this work....I thought games are hosted on game providers and not your own servers...

The actual game is indeed calculated by the game provider, so on NetEnt/Microgaming/NYX/Evolution servers in our case.

Still, we have a database as well where the game results are logged.

Basically it goes like this, on our website you click "Spin". We register this, take the money from your balance and send the bet to the game server which is hosted by the game provider. The game server does the RNG magic and comes back with a game result, which then goes into our database and the game win will be added to your account balance again.

 

[Jasminebed]

Thanks for taking the time to share.

Make sure you "comp" your frontline staff when they have to spend a shift dealing with us during such issues. A slice of pizza and a cold beer do a lot say thanks for your hard work.

And please please if you do know there are issues at the time, don't tell us to go clear our cache or cookies, or pretend we are the only ones having an issue.

Because we come over to CM and see that's not true, it's not our end.

And go tape a picture of Connection to the Server has failed and another of Casino error 202 please contact support to a dart board on your office wall so we don't need to send you any more screenshots of it.

 

[EbeeDog]

What a novel idea. Nicely done.

I especially like this part:
Even if the problem is at some point solved, the issue doesn’t stop here for us! There was a problem, and even though it’s solved for now, we still might not have a cause. Since we made the problem go away to make the website function again, it might also be hard(er) to find the cause of the problem. At this point, it is still a top priority for everyone to find out what caused the problem and help fix it. People are still working around the clock for this. Our support will also go back to players who had issues and find solutions and in some cases a compensation.


It's like the difference between a " auto parts changer" and a mechanic. The changer will replace your cam shaft dampener, but a mechanic looks for why it went bad in the first place.

 

[dunover]

Excellent! It's always good to hear the casinos' side of things and thanks for taking the time to explain. Like I said in Jason's Raging Bull thread it's educational to learn of the challenges the 'other side' faces as well as our own.

Common-sense tells us all that no casino wants disconnects/downtime and especially not be beholden to crappy or unreliable servers. There are no winners in that scenario - literally!

 

[Harry_BKK]

Melvin, respect and thanks for your informative post. :notworthy

However, straight away a few questions pop into my mind:

1. according to your experience, where do the most problems happen:

-> communication player PC - RP server
-> Royal Panda servers (a BET/SPIN request goes through different servers as you are using a whole bunch of them, front stuff, financial, game logs etc)
-> RP network (as you have to relay the bet request to the respective game server)
-> communication RP servers - game provider servers
-> on the return journey through the web of servers
-> data carriers/ISP's​

It is just mind blowing to see that an industry, which is not just shifting around billions/year but also is very profitable, can't find solutions for these problems.

A investment company in NYC spent over 150Mio USD to have a new data cable installed to receive the results from the Chicago exchange 0.01seconds faster than the others. Now i don't expect you to do this but surely you have had just as enough frustration with these connection errors as we players.

Let it happen once, twice a day but if i take just the 780odd UK registered casinos it must be 10.000's or even 100.000's of these errors on a daily basis. For a programmer / software whiz kid it should be "fairly easy" to find the root cause of it and give us players some relief by implementing a permanent fix, not just a temporary patch, which causes then trouble in a different area.

I as a player have sometimes the impression that especially game providers don't really care about these issues and it seems that it does not cost them yet enough in revenue to search for a proper solution.

Please don't take this personal or harsh, i think the above just reflects the sentiment at present in the player community.

 

[Melvin - Panda]

Thanks for taking the time to share.

Make sure you "comp" your frontline staff when they have to spend a shift dealing with us during such issues. A slice of pizza and a cold beer do a lot say thanks for your hard work.

And please please if you do know there are issues at the time, don't tell us to go clear our cache or cookies, or pretend we are the only ones having an issue.

Because we come over to CM and see that's not true, it's not our end.

And go tape a picture of Connection to the Server has failed and another of Casino error 202 please contact support to a dart board on your office wall so we don't need to send you any more screenshots of it.

Thanks Jasminebed,

Regarding the cache/cookies part. If they have seen the issue several times that day, those will probably not be suggested anymore. If you're among one of the first few to report an issue, that will probably always remain a standard reply. Really most of the time a problem is reported to our support, there is no large scale issue on our end and the issue has to do with a player's connection/cache/computer or something else entirely.

While the community here is probably not a good representation, and you might already try all of the suggested things before even contacting support, it does resolve most of the issues. But as mentioned, once several players report the same problem it will be forwarded to other departments who will have a further look at the situation. You might be asked for (too many?) details, since it could be that the issue is only happening for players from your country/currency/game/computer/browser/operating system/internet provider/timezone and it could help to isolate the problem.

 

[Melvin - Panda]

1. according to your experience, where do the most problems happen:

-> communication player PC - RP server
-> Royal Panda servers (a BET/SPIN request goes through different servers as you are using a whole bunch of them, front stuff, financial, game logs etc)
-> RP network (as you have to relay the bet request to the respective game server)
-> communication RP servers - game provider servers
-> on the return journey through the web of servers
-> data carriers/ISP's​

Really hard to tell from my point of view. I'm also not the technical expert myself, so I don't have all the inside details and knowledge I'm afraid. I'm also not sure if the IT guys would be too happy with sharing the most vulnerable parts of our operation.

A investment company in NYC spent over 150Mio USD to have a new data cable installed to receive the results from the Chicago exchange 0.01seconds faster than the others. Now i don't expect you to do this but surely you have had just as enough frustration with these connection errors as we players.

I as a player have sometimes the impression that especially game providers don't really care about these issues and it seems that it does not cost them yet enough in revenue to search for a proper solution.

I do know that we invest in better hardware when we believe that it is a limiting factor.

Basically when we find out where a problem is caused, we will do everything we can to eliminate that cause. This could be hardware/software/code or database related. Servers and hardware also have backups and backup internet connections, so in case any one thing fails there should still not be a problem for you as a player.

Please don't take this personal or harsh, i think the above just reflects the sentiment at present in the player community.

Not at all, I started the thread for a reason and I will share any info that I'm able (and allowed) to give.

 

[Valhalla]

Thanks for posting Melvin! Really interesting read. It's good to have a bit of insight into things from your end when it all goes tits-up (pardon the expression)

The one that really annoys me is when NetEnt games instantly say "This game is currently not available. Please try again later..." whereas Microgaming games will spin for a while (sometimes a long time) to try to sort themselves out - which often happens.

 

[lotusch]

Very pleasant to read thread..
Thanks Melvin..

Much appreciated..

Had a question myself but that has already been answered by yourself as another member already asked the same question..

 

[Tirilej]

I just wish you hadn't choosen that song as a title because I can't read it without getting the melody in my head

 

[lotusch]

I just wish you hadn't choosen that song as a title because I can't read it without getting the melody in my head

Hahaha I have exactly the same. No Lie!
Trying to get rid of it as we speak.

 

[colinsunderland]

Excellent thread, thank you for taking the time to post

And please please if you do know there are issues at the time, don't tell us to go clear our cache or cookies, or pretend we are the only ones having an issue.

Because we come over to CM and see that's not true, it's not our end.

This x 100. I've never used royal panda support so this isn't aimed at them, but there have been a few times with other casino's (hello Mr Green in particular) when support have insisted I was the first one to report the problem, had me spend 20 minutes clearing cache etc and jumping through other hoops to no end, to come on here and see a thread full of complaints, even on one occasion, including chat logs from an hour previously with the same chat agent.
If theres a problem admit it and I'll try later. If you lie I probably won't come back.
As an example, one casino did this, I had deposited £3000 in the previous 5 weeks with them, in the 2 months since, I have deposited £35.

 

[Harry_BKK]

Really hard to tell from my point of view. I'm also not the technical expert myself, so I don't have all the inside details and knowledge I'm afraid. I'm also not sure if the IT guys would be too happy with sharing the most vulnerable parts of our operation.


I do know that we invest in better hardware when we believe that it is a limiting factor.

Basically when we find out where a problem is caused, we will do everything we can to eliminate that cause. This could be hardware/software/code or database related. Servers and hardware also have backups and backup internet connections, so in case any one thing fails there should still not be a problem for you as a player.


Not at all, I started the thread for a reason and I will share any info that I'm able (and allowed) to give.

Melvin, no offense but you answered as i expected Of course, i don't want you to expose your vulnerabilities but i was hoping that finally somebody would step-up and tell us honestly what is going on. After all we are the customers who spend a ton day in day out to enjoy a punt, yet have to deal with errors out of our control and never get an explanation for that. They are the same errors time and time again.

Compare this to your car, let's say you just bought a new one and the engine keeps cutting out. You take it back to the garage countless times and on every occasion they repair it but never tell you what the reason is and yet a day later the same issue starts again. Would you not feel entitled to a honest explanation or even would consider returning the car and asking for a refund???

The player experience is really suffering badly from those game errors/spin lags and they affect Microgaming and Netent alike, the other game providers to a lesser degree since they are probably less played.

It is actually laughable, i started playing MG download casinos in 1998 with a 56kB/second phone line connection. Took ages to download a new game but never ever ever ever did i have disconnections/game errors/spin freezes as i do experience them now with the browser based casinos and that with internet connections a gazillion times faster than back in 1998.

I said this already in a different thread: the industry is just lucky that a majority of its customers are "addicted" gamblers who one can send through hoops and circles as often as they like as most gamblers are just happy they can have a SPIN and will return even when knowing that the session won't be error free.

 

[geordiecolin]

I just wish you hadn't choosen that song as a title because I can't read it without getting the melody in my head

Oh god. I escape from the likes of Adele by relaxing playing slots.
In fact I will murder Adele's husband myself if he ever breaks her heart because it will mean another 5 years of new songs of heartbreaky whining on!

 

[spintee]

Great stuff, Its good to see a casino at work, Some sites could not give a sh*t, They take out a sub license from a network & do nothing and rely on them doing the work and keep there fingers crossed they get punters, If I ever took on a casino with every thing took care off including chat / payments etc than I would get my own staff & make sure players are happy, Thats not going to happen any time soon I will leave that there,

Back to to dreaded errors, When reps say delete your entire computer its wrong, Ok fair enough on some rare occasions it maybe just that but if CS will only say this if they have not had any more complaints than I must be the only idiot reporting errors, Because I will bet what ever you want to put down that 99.9% of the time its the first they have heard of any error

I bet if there was a thread about errors it would take off, Some sites I can count on one hand how many errors I have come across, But not every body is computer wise, Being told to delete cookies and cache could serious damage ones computer if they do the wrong things,

Also do you know you missed some out of your thread? The part about free spins for us members

Keep up the good work and I am sure it will be rewarded

 

[Melvin - Panda]

Hey Spintee, thanks for the comment.

Also do you know you missed some out of your thread? The part about free spins for us members

What do you mean by this part exactly? Did I forget something?

 

[spintee]

Hey Spintee, thanks for the comment.



What do you mean by this part exactly? Did I forget something?

I was messing about, You forgot to put the part in your post about free spins for us

 

[mcgameboy]

An excellent OP indeed Melvin. Anything that helps to further educate the CM membership (whether they are "seasoned veterans/old hands", "newbies/rookies" or "somewhere inbetween") is to be commended. :notworthy

As someone who is fascinated/intrigued by this kind of thing, I sincerely look forward to reading your next "Casino Ops" post.