![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
trysdynAugust was a bad month. Getting into the complete picture of why is far beyond the scope of one post, so I'm going to focus on the issue most visible to people following me in the internet spaces: my issues with Comcast and my internet service. This is long so I've chunked it into chapters to not flood timelines.
It started in the middle of my FFIV:TAY play-through actually; 10Aug-- almost a month ago now. One evening I was streaming and several of my viewers began complaining of buffering at once. I initially just assumed it was Twitch because my ISP has been rock solid for years but, once I switched to OBS and looked at its network status I realized I was dropping about 50% of my video frames. I was forced to stop streaming for the first time... ever I think.
I ran some network tests and found I was experiencing about 30% packet loss. In layman's terms this is unusable levels of service degradation. But well, this was the first incident with Comcast in years, so I assumed they were having some random issues and didn't worry about it. The next stream it happened again, forcing me again to stop early. That's when it became a problem. I left a test running overnight and found that in addition to my packet loss spiking up to 30% for about 20 minutes every few hours, I was now dropping 2% of my outgoing packets 24/7.
Unfortunately at the time, my roommate who was the account holder was out of town. I tried to call Comcast anyway to see if they could at least check my signal levels. They hung up on me. Very politely, but they hung up on me.
So after dealing with that for 4 days, we were finally able to engage Comcast. Our first attempt to contact them was via web chat. The tier 1 looked at the initial report of our issue given prior to initiating the chat and... rebooted our modem and terminated the chat. Attempt #2! The tech looked our report and... rebooted our modem and terminated the chat. Attempt #3 we started the chat with "DO NOT REBOOT OUR MODEM IT DOES NOT HELP" and actually got to talk to a tech. Phew! They did their basic tier one troubleshooting and said they would look on their end and to come back in 24 hours if nothing changed.
The next day, no improvement. We reached out to them again and... they rebooted our modem. The tech had to go all the way through tier 1 troubleshooting again. Whatever. They scheduled a tech for Wednesday; two days out. So I had to sit and deal with the unworkable packet loss for two more days. When Wednesday rolled around we got the SMS that the tech was on his way, waited for them to arrive and... nothing. 15 minutes, 20, 30, now the window for their arrival had closed. We contacted support and learned they flagged us at not home and moved on to the next appointment; we would have to reschedule. Livid... but ultimately powerless, we rescheduled for Friday. Two more days.
At this point I hadn't been able to stream for over a week. The issue remained persistently bad throughout. A tech finally arrived that Friday and looked at our setup. He verified the signal levels are fine, then walked outside and looked at our drop to the post. He claimed he saw a rotted cable and replaced it, and said "that should help" and left. I did a network test immediately and found it'd done nothing-- upon inspecting the cable he cut and left lying on the ground I found no evidence of rot or breakage in the insulation. So he did nothing.
Wanting to drive this myself, since I'm capable of being more annoying than my good-natured roommate, I had my roommate put me on the account as a second user so I could take over calling them myself. So I have a better account of the hell that started from here.
At this point it's 18 Aug. I contacted Comcast via chat (since we seemed to have better luck with chat than phone) and got a very nice and informative CSR who listened to my issue, skipped the tier 1 troubleshooting, and offered to escalate my issue to local engineers. I accepted this, they wrote up the ticket, and told the local team's SLA is 72 hours and I should either see improvement by then or contact them again. They gave me a ticket number and I gave them a 5/5 in their support survey.
The next day I had a thought occur to me (I actually forget what at this point) and re-contacted Comcast to ask them to update the ticket with more information. This time I got a CSR who barely spoke English, was trying very hard to get back into Tier 1 troubleshooting workflow despite my objections, and it took me 15 minutes to just get him to check the status of the prior ticket. He informed me the ticket had been closed, but would not tell me if any work was done or why it was closed.
I was pissed. I tried to get him to re-escalate my issue, to which he requested I re-describe the problem I'm having. I did, and he said he would be resetting my modem. I immediately replied with an all-caps instruction to not, but it was too late. Plunk went my connection, but this time it did not return. I don't know what he did, but we were offline for 45 minutes before we finally called in and had to re-activate our modem to get a connection again.
Attempt #2 to contact them that night, I immediately instructed the CSR to not change anything with my modem or account as we were far, far beyond tier 1 help. I asked, again, for escalation and we went back and forth for 30 minutes with the CSR trying to tell me he had to follow workflow. I requested a supervisor, he softly refused, requesting very strongly that I allow him to try to fix the problem. Ultimately he could not, nor could he escalate, so I got my supervisor. Said supervisor very curtly told me there was no escalation, no tier 2, and the only option I have is to accept another tech visit and if THEY saw fit to escalate, THEY would. Fine... They scheduled a tech visit for next Monday. Three more days without working internet.
This tech showed up and had an entirely different MO than the last. He didn't check my signal, didn't look at my wiring. He immediately pulled out his tablet and ran a network and signal diagnostic from the other end of the connection. This test came back perfect, even while I was in the middle of a packet loss burst. I thought maybe, finally, this would prove it's not a problem with us, but he wasn't convinced. Looking around the living room he said "there must be a splitter degrading the signal". I showed him the splitter we'd already pulled out of the pipeline as part of our own troubleshooting and he shook his head. "No I mean one splitting the signal coming into the house"
There wasn't one. We spent 15 minutes checking every wall in the house for the sign of a split cable or multiple drops, and found nothing. He shrugged and said "I'll escalate you, but I have to tell them we didn't eliminate every possibility of a splitter; they may reject the ticket". I was someone between seeing red and bursting into tears-- he was going to leave and again nothing would be done. He insisted there was no more we could do unless we could find this mysterious splitter that either didn't exist, or was embedded INSIDE a wall. So he filed the ticket and left.
That night I finally exercised an option I'd been hesitant to exercise: Twitter. You know the ComcastCares account? The one that swoops in and offers to help any time someone grumbles on Twitter about Comcast? I finally reached out to them. They immediately demanded a ton of info tying my real info to my online handle (which I've since purged everywhere I can), listened to my issue and directed me to contact tier 1. Great, thanks, jerks. Basically they're useless and only serve to make it look like they have a support network on Twitter.
Out of options, infuriated, I turned to the Xfinity support forums and unleashed a blast of rant, technical data, POV storytelling, and cursing. It was here that, for the first time, I got someone that I felt like was actually capable of listening to my problem and providing meaningful information and escalation. An individual who goes only by "Che" on the forum, an official Comcast employee, PMed me and offered to look into the issue.
There was some disjoint between Che's working hours and my waking hours, but through that Tuesday they were able to confirm the local escalation team had picked up my issue and was investigating, and that they would convey all my technical data into the ticket and upgrade its priority. Also Che reported to me that multiple customers in my area were describing similar problems and this was now a regional problem. How validating!
That night something miraculous happened: I saw improvement. It wasn't perfect, but I could stream. The persistent 2% packet loss vanished and the 30% bursts had turned to bursts of 3-5%. Bad, but not enough to completely knock my stream offline every time it happened.
Over the next few days I streamed several times. Each stream I saw small bursts of dropped frames, but not enough to stop streaming or immediately yell to Che. This improved state lasted maybe 4 or 5 days, then the problems returned. I think it was Saturday I was, for the third time ever, forced to end a stream early because of extreme levels of packet loss. I PMed Che once again to report the problems had returned and... nothing.
I continued to stream despite the loss issues, the dropped frames getting more severe with each stream until I was forced offline again around Thursday. At this point my contact at Comcast hadn't replied to any of my updates and request for update from them for 8 days. I sent my third message since the ghosting started, practically begging for an update so I could know if I need to start over with tier 1. I also unloaded another rant on the forum, this time threatening to cancel my service if they can't tell me SOMETHING.
This got some leeway. Not only did Che reply soon after, but a second Comcast employee who only goes by 'Zach' on the forum also chimed into the discussion. They both told me the same thing: that my issue was now being escalated to my "Local market" as the escalation ticket the tech from 10 days ago opened had been closed and obviously my issue had not been fixed.
That's where we are now. I got that last bit of news early today, so since this is a long weekend I do not expect any kind of change to the situation until Tuesday at the earliest. At the moment I'm sitting at a fluctuating 1-5% packet loss 24/7, with spikes back up to 30%. Exactly how it was before. On the plus side this time around the 30% spikes seem to be once or twice a day rather than every couple of hours-- but it's still bad enough that I feel a need to push for a resolution.
If this new escalation falls short, I have no idea what I'm going to do. My only real other option here is UVerse and I know from experience we live in a place barely outside their support range, so we'd probably get awful speeds and reliability with them anyway.
Looks like the loss has vanished. It's been solid all afternoon, but I'm definitely not calling this fixed until it's clear for 24-48 hours. I haven't heard from the super secret local escalation team at all, so I don't know what's up or if I can expect this to persist.
This is different, at least, from the improvement I saw for 4 days in the middle of this. That was a drop in loss to a reasonable level; this is complete elimination of loss.
If/when they call I'm definitely not going to let them off the hook. I plan to tell them I don't know if the problem is fixed and that I will reach out to them later. I definitely want at least 2-3 days of solid testing before I let go of the escalation ticket.
Fingers crossed.
Comcast reached out to me today to let me know work was finished. Everything's been perfect since the 5th, so I guess things are fixed. Exactly one month from initial issue to resolution... phew.
For two weeks, things were solid, then I began noticing light fluctuations in my SSH sessions and small instances of dropped frames on stream. I dusted off my network testing scripts and found I was experiencing 0.2-1.0% packet loss, but only in the evenings. 7pm to midnight or so. The rest of the time it's perfectly clear.
It looks like the root cause has been Comcast oversold our area and we have issues when everyone is using their 'net in the evening, but I don't know for sure.
I reached out to my contacts at Comcast early, before this could escalate. Unfortunately they were disinterested in replying, or even reading my messages, as we'll see...
I noticed my first major stream blip since the fix around 5 Sept. I had about a 15 second period of hard frame drops enough to cause Twitch to warn me my stream was unstable. Only one incident, but I knew the returning problem was getting worse. Packet loss at this point was around 3%, just enough to cause buffering issues over time.
No reply from Comcast on my message the prior week yet. In fact, I could tell via the forum software that my message hadn't even been read. I sent a second message to another engineer that was involved in fixing the initial issue.
Here we are again. I just had to close Crea after being completely unable to play it with my friends due to the packet loss and lag. Crea doesn't have the best netcode, so that 2-3% packet loss is exacerbated into a constant melange of position corrections and jittering.
My messages to Comcast STILL have not even been read, let alone responded to. I bumped the forum thread with a (mostly) polite yet clearly angry request to get some attention on my issue. Again, this seems to only occur in the evenings now. It's almost surely an oversell problem, not that I can prove that.
Switching to UVerse is looking really good right about now.
10 Aug: The Beginning
It started in the middle of my FFIV:TAY play-through actually; 10Aug-- almost a month ago now. One evening I was streaming and several of my viewers began complaining of buffering at once. I initially just assumed it was Twitch because my ISP has been rock solid for years but, once I switched to OBS and looked at its network status I realized I was dropping about 50% of my video frames. I was forced to stop streaming for the first time... ever I think.
I ran some network tests and found I was experiencing about 30% packet loss. In layman's terms this is unusable levels of service degradation. But well, this was the first incident with Comcast in years, so I assumed they were having some random issues and didn't worry about it. The next stream it happened again, forcing me again to stop early. That's when it became a problem. I left a test running overnight and found that in addition to my packet loss spiking up to 30% for about 20 minutes every few hours, I was now dropping 2% of my outgoing packets 24/7.
Unfortunately at the time, my roommate who was the account holder was out of town. I tried to call Comcast anyway to see if they could at least check my signal levels. They hung up on me. Very politely, but they hung up on me.
14 Aug: The Initial Support Request
So after dealing with that for 4 days, we were finally able to engage Comcast. Our first attempt to contact them was via web chat. The tier 1 looked at the initial report of our issue given prior to initiating the chat and... rebooted our modem and terminated the chat. Attempt #2! The tech looked our report and... rebooted our modem and terminated the chat. Attempt #3 we started the chat with "DO NOT REBOOT OUR MODEM IT DOES NOT HELP" and actually got to talk to a tech. Phew! They did their basic tier one troubleshooting and said they would look on their end and to come back in 24 hours if nothing changed.
The next day, no improvement. We reached out to them again and... they rebooted our modem. The tech had to go all the way through tier 1 troubleshooting again. Whatever. They scheduled a tech for Wednesday; two days out. So I had to sit and deal with the unworkable packet loss for two more days. When Wednesday rolled around we got the SMS that the tech was on his way, waited for them to arrive and... nothing. 15 minutes, 20, 30, now the window for their arrival had closed. We contacted support and learned they flagged us at not home and moved on to the next appointment; we would have to reschedule. Livid... but ultimately powerless, we rescheduled for Friday. Two more days.
At this point I hadn't been able to stream for over a week. The issue remained persistently bad throughout. A tech finally arrived that Friday and looked at our setup. He verified the signal levels are fine, then walked outside and looked at our drop to the post. He claimed he saw a rotted cable and replaced it, and said "that should help" and left. I did a network test immediately and found it'd done nothing-- upon inspecting the cable he cut and left lying on the ground I found no evidence of rot or breakage in the insulation. So he did nothing.
18 Aug: Tier 1 Hell
Wanting to drive this myself, since I'm capable of being more annoying than my good-natured roommate, I had my roommate put me on the account as a second user so I could take over calling them myself. So I have a better account of the hell that started from here.
At this point it's 18 Aug. I contacted Comcast via chat (since we seemed to have better luck with chat than phone) and got a very nice and informative CSR who listened to my issue, skipped the tier 1 troubleshooting, and offered to escalate my issue to local engineers. I accepted this, they wrote up the ticket, and told the local team's SLA is 72 hours and I should either see improvement by then or contact them again. They gave me a ticket number and I gave them a 5/5 in their support survey.
The next day I had a thought occur to me (I actually forget what at this point) and re-contacted Comcast to ask them to update the ticket with more information. This time I got a CSR who barely spoke English, was trying very hard to get back into Tier 1 troubleshooting workflow despite my objections, and it took me 15 minutes to just get him to check the status of the prior ticket. He informed me the ticket had been closed, but would not tell me if any work was done or why it was closed.
I was pissed. I tried to get him to re-escalate my issue, to which he requested I re-describe the problem I'm having. I did, and he said he would be resetting my modem. I immediately replied with an all-caps instruction to not, but it was too late. Plunk went my connection, but this time it did not return. I don't know what he did, but we were offline for 45 minutes before we finally called in and had to re-activate our modem to get a connection again.
Attempt #2 to contact them that night, I immediately instructed the CSR to not change anything with my modem or account as we were far, far beyond tier 1 help. I asked, again, for escalation and we went back and forth for 30 minutes with the CSR trying to tell me he had to follow workflow. I requested a supervisor, he softly refused, requesting very strongly that I allow him to try to fix the problem. Ultimately he could not, nor could he escalate, so I got my supervisor. Said supervisor very curtly told me there was no escalation, no tier 2, and the only option I have is to accept another tech visit and if THEY saw fit to escalate, THEY would. Fine... They scheduled a tech visit for next Monday. Three more days without working internet.
21 Aug: Tech Visit #2
This tech showed up and had an entirely different MO than the last. He didn't check my signal, didn't look at my wiring. He immediately pulled out his tablet and ran a network and signal diagnostic from the other end of the connection. This test came back perfect, even while I was in the middle of a packet loss burst. I thought maybe, finally, this would prove it's not a problem with us, but he wasn't convinced. Looking around the living room he said "there must be a splitter degrading the signal". I showed him the splitter we'd already pulled out of the pipeline as part of our own troubleshooting and he shook his head. "No I mean one splitting the signal coming into the house"
There wasn't one. We spent 15 minutes checking every wall in the house for the sign of a split cable or multiple drops, and found nothing. He shrugged and said "I'll escalate you, but I have to tell them we didn't eliminate every possibility of a splitter; they may reject the ticket". I was someone between seeing red and bursting into tears-- he was going to leave and again nothing would be done. He insisted there was no more we could do unless we could find this mysterious splitter that either didn't exist, or was embedded INSIDE a wall. So he filed the ticket and left.
That night I finally exercised an option I'd been hesitant to exercise: Twitter. You know the ComcastCares account? The one that swoops in and offers to help any time someone grumbles on Twitter about Comcast? I finally reached out to them. They immediately demanded a ton of info tying my real info to my online handle (which I've since purged everywhere I can), listened to my issue and directed me to contact tier 1. Great, thanks, jerks. Basically they're useless and only serve to make it look like they have a support network on Twitter.
22 Aug: A Glimmer of Hope?
Out of options, infuriated, I turned to the Xfinity support forums and unleashed a blast of rant, technical data, POV storytelling, and cursing. It was here that, for the first time, I got someone that I felt like was actually capable of listening to my problem and providing meaningful information and escalation. An individual who goes only by "Che" on the forum, an official Comcast employee, PMed me and offered to look into the issue.
There was some disjoint between Che's working hours and my waking hours, but through that Tuesday they were able to confirm the local escalation team had picked up my issue and was investigating, and that they would convey all my technical data into the ticket and upgrade its priority. Also Che reported to me that multiple customers in my area were describing similar problems and this was now a regional problem. How validating!
That night something miraculous happened: I saw improvement. It wasn't perfect, but I could stream. The persistent 2% packet loss vanished and the 30% bursts had turned to bursts of 3-5%. Bad, but not enough to completely knock my stream offline every time it happened.
Over the next few days I streamed several times. Each stream I saw small bursts of dropped frames, but not enough to stop streaming or immediately yell to Che. This improved state lasted maybe 4 or 5 days, then the problems returned. I think it was Saturday I was, for the third time ever, forced to end a stream early because of extreme levels of packet loss. I PMed Che once again to report the problems had returned and... nothing.
27 Aug: Ghosted by Comcast Support
I continued to stream despite the loss issues, the dropped frames getting more severe with each stream until I was forced offline again around Thursday. At this point my contact at Comcast hadn't replied to any of my updates and request for update from them for 8 days. I sent my third message since the ghosting started, practically begging for an update so I could know if I need to start over with tier 1. I also unloaded another rant on the forum, this time threatening to cancel my service if they can't tell me SOMETHING.
This got some leeway. Not only did Che reply soon after, but a second Comcast employee who only goes by 'Zach' on the forum also chimed into the discussion. They both told me the same thing: that my issue was now being escalated to my "Local market" as the escalation ticket the tech from 10 days ago opened had been closed and obviously my issue had not been fixed.
2 Sep: Escalation #4
That's where we are now. I got that last bit of news early today, so since this is a long weekend I do not expect any kind of change to the situation until Tuesday at the earliest. At the moment I'm sitting at a fluctuating 1-5% packet loss 24/7, with spikes back up to 30%. Exactly how it was before. On the plus side this time around the 30% spikes seem to be once or twice a day rather than every couple of hours-- but it's still bad enough that I feel a need to push for a resolution.
If this new escalation falls short, I have no idea what I'm going to do. My only real other option here is UVerse and I know from experience we live in a place barely outside their support range, so we'd probably get awful speeds and reliability with them anyway.
5 Sep: Improvement!
Looks like the loss has vanished. It's been solid all afternoon, but I'm definitely not calling this fixed until it's clear for 24-48 hours. I haven't heard from the super secret local escalation team at all, so I don't know what's up or if I can expect this to persist.
This is different, at least, from the improvement I saw for 4 days in the middle of this. That was a drop in loss to a reasonable level; this is complete elimination of loss.
If/when they call I'm definitely not going to let them off the hook. I plan to tell them I don't know if the problem is fixed and that I will reach out to them later. I definitely want at least 2-3 days of solid testing before I let go of the escalation ticket.
Fingers crossed.
10 Sep: Resolution
Comcast reached out to me today to let me know work was finished. Everything's been perfect since the 5th, so I guess things are fixed. Exactly one month from initial issue to resolution... phew.
23 Sep: Recurrence
For two weeks, things were solid, then I began noticing light fluctuations in my SSH sessions and small instances of dropped frames on stream. I dusted off my network testing scripts and found I was experiencing 0.2-1.0% packet loss, but only in the evenings. 7pm to midnight or so. The rest of the time it's perfectly clear.
It looks like the root cause has been Comcast oversold our area and we have issues when everyone is using their 'net in the evening, but I don't know for sure.
I reached out to my contacts at Comcast early, before this could escalate. Unfortunately they were disinterested in replying, or even reading my messages, as we'll see...
29 Sep: First Major Issues
I noticed my first major stream blip since the fix around 5 Sept. I had about a 15 second period of hard frame drops enough to cause Twitch to warn me my stream was unstable. Only one incident, but I knew the returning problem was getting worse. Packet loss at this point was around 3%, just enough to cause buffering issues over time.
No reply from Comcast on my message the prior week yet. In fact, I could tell via the forum software that my message hadn't even been read. I sent a second message to another engineer that was involved in fixing the initial issue.
3 Oct: Service Unusable
Here we are again. I just had to close Crea after being completely unable to play it with my friends due to the packet loss and lag. Crea doesn't have the best netcode, so that 2-3% packet loss is exacerbated into a constant melange of position corrections and jittering.
My messages to Comcast STILL have not even been read, let alone responded to. I bumped the forum thread with a (mostly) polite yet clearly angry request to get some attention on my issue. Again, this seems to only occur in the evenings now. It's almost surely an oversell problem, not that I can prove that.
Switching to UVerse is looking really good right about now.
