[Solved] [Known Issue] We are seeing some errors with some email campaigns and private domain/branded links showing errors and non SurveyGizmo pages.

0

Our System Administrators are working hard on this issue as we speak, we hope to have it 100% corrected soon!

Taom answered

    Best answer

    0

    Hey Everyone,

    This is all fixed up!  Our System Administrators worked hard to fix the domain issue as soon as we could.  However, you may still be seeing some issues for a brief time until everything clears up on the interwebs.

    DNS changes can take a few minutes to a few hours to propogate, so it’s mostly resolved now, but some users may still get a bad link.

    We are so sorry that this happened, we of course are taking steps to make sure this NEVER happens again.

    -SG Engineers

    tvadney commented
      • Hi Jon,

        Will users with bad links need new ones? Or once DNS finishes propogating, it will work?

        Thanks,
        Matt

      • Hi Matt,

        As far as we know at this point, once DNS finishes propogating, these links will work. It’s a little tricky for us at this point, since we cannot replicate this issue, as our Time Zone wasn’t yet affected by the change that caused these errors.

        If a few hours go by, and the links still aren’t working, please let our Support Heroes know you are still running into issues.

        Jon

      • Thanks Jon,

        Can you please provide an a little more in depth explanation of the issue/what caused this issue? And provide the downtime experienced (from notice to resolution)? I have several internal and external stakeholders I need to inform and they will want this information.

        Thanks,
        Matt

      • Hey Matt,

        No problem at all, we are so sorry that this happened. Basically, one of our domains, sgizmo.com, was briefly expired this morning due to a communication issue between our domain host and our IT team, and we worked to renew this domain as quickly as possible, and have fixed things so this will hopefully never happen again going forward. The downtime was between approximately 7:00 am to 8:33 am MT and was very sporadic. The impact was sporadic as it depended on very specific behavior/features/domains and propogation of DNS across the internet. Email Campaign links (more-so in Europe than in the US) and some Private Domain links were possibly impacted. I hope this helps!

        Jon

      • Thanks Jon,

        That is helpful. I know this will take some time to resolve itself, and in the meantime some links will work, some won’t. However, is there a time where your team feels confident in saying that all links will be working properly again?

        Thanks,
        Matt

      • This is happening now to a live survey that was released at 12:48 PDT. Select recipients are being directed to:

        http://s-e194c5-i.sgizmo.com/s3/i-bYobnXh6pzBspw9gB-1337382/?sguid=bYobnXh6pzBspw9gB

        when clicking through on the survey link.

        Using branded Pro service with whitelisted SPF in place

        PLEASE FIX ASAP. THIS IS 100% UNACCEPTABLE!

      0

      Hello all,

      I’m sorry links continue to not be working in places.  I have been doing further research into ways we can help you speed up or get around the issues blocking the links from working properly on your side and/or for certain respondents.

      Here’s what I have found that you can try:

      The core of the problem working against us here is that to solve the problem we had, we renewed the domain, sgizmo.com, that we use to serve any private domains used through SurveyGizmo and for email campaign links.  When this is done, the update to tell end users clicking these links where to find the web page is cached in cases and beholden to what is called DNS Propagation.

      DNS Propagation essentially is the time it takes for the changes to a domain on the internet to take effect.  The DNS, Domain Name System, settings cache the information about the domain for a certain amount of time and then refresh to look for any changes to their settings.  Our renewal of the domain is part of this refresh.  While many DNS caches, whether they are on your private domain, our domain, a local network at an office, company, school, or elsewhere, can update within a few hours (and that’s what we have been seeing for the majority of links that have been having issue) our research shows that the common fringe of the DNS cache settings before they refresh is up to 24 hours.

      The amount of time over which a DNS cache will take to be refreshed is called TTL, or “Time to Live”.  Once that time passes, the changes we have made to fix the problem will propagate for that end user.  So the question to ask is, can you get around this time it takes to propagate the changes we have in place to get your links back up and running.  The answer is “possibly” but not totally likely depending on the situation.

      You can try speeding it up by lowering the TTL number set on your private domain or on any settings on the network in your office or at your respondent’s company.  Do remember that if you update your domain itself, the network you or your respondents are using to access the internet on might refresh at a slower rate.  This can be a solution, but might be detrimental overall for accessing the internet from the location.

      The other thing to explore is to ask your IT team or the IT team on your respondent’s end if they can clear the DNS cache.  This can work, but isn’t a comprehensive solution for getting around the time to wait for propagation to occur.

      A final good step is to clear the cache and cookies on your web browser and ask your respondents to do so, if you feel its appropriate to reach out to them to ask.  Here’s how in case you want to provide instructions:

      https://help.surveygizmo.com/help/article/link/clear-cache-and-cookies

      While that too isn’t a comprehensive solution, it has the potential to help and is a quick step to see if it will get things rolling.

      So the long story as short as possible here, is this might take up to 24 hours for some respondents and locations to get back into the links correctly.  Getting around that delay is difficult and not always comprehensive for all respondents depending on what DNS settings their computer or device is beholden to.

      We do want to keep helping and are looking for any other avenues to keep working towards a speedier solution on.

      The final key note I can make on this end is that while you might still be seeing the incorrect page from your links because the DNS cache has yet to clear on your end, a good amount of respondents around the world are not having issue now because their DNS cache has cleared on their end.  Obviously if you have a survey you’re sending to a single company or single office all sharing the same DNS cache that hasn’t cleared, this is still effecting a good amount of your respondent base.  That is very damaging for you and your reputation.  I’m happy to work to do anything I can to work with you there on damage control.

      tvadney commented
        • That first to the last sentence in your post sums it up nicely. Please have someone from your account management or accounting dept. contact me about a fee credit.

          Also, it would be great to understand what business process controls SG is putting in place to ensure that the “uh, we forgot to renew the domain” issue doesn’t occur again.

        0
        Taom 10 Rep.

        This problem has not yet been resolved for us here in India. It’s been 2 days now. Could you give us an alternative. We have surveys running and the respondents are not able to get into the survey yet. 

        Taom answered
          0

          This is continuing to be a problem for my clients. We have more than 200 links affected by this outage.  As of 7:30 in AZ (PST) the links were working for me and my clients in VA (EST). But a main end client with 27 links is still redirecting to the register page — using a new browser where cookie cleaning would not matter. That client is in CA (PST).  

          What are others seeing? 

          It has now been 24 hours since this was supposed to have been solved. What gives?

          Please provide us communication or set up some sort of outage report map so we know where issues still remain. 

           

           

          Topper Shull commented
            • Thank you for letting us know this is continuing to happen for your specific clients. I am reaching out by email to you now to keep working on solutions.

            1

            Morning, 

            Our clients are still seeing an issue this morning (UK time). This is embarrassing when we told them yesterday that it was fixed.

            Please provide an update.

            Thanks,

            Kirsty

            Topper Shull commented
              • Hello Kirsty,
                As an update, we have seen widespread resolution of the problem around the world, but are also fielding final cases where the issue is still being cashed in locations and for companies/respondents that have not yet had their system refresh to have an update.

                Thank you for reaching out to us by email to discuss with my colleague Josh. We will reach out shortly.

              0
              Erik 13 Rep.

              Guys,

               We received ZERO notification that there was an issue with this today and it resulted in a significant decrease in revenue.

               

              Please be sure to let your customers know that things are broken, especially when they are spending money to drive traffic to your service.

              Topper Shull commented
                • Hello Erik,

                  Thank you for following up with us on the damage this has caused your team. I thought I would reach out to you via email to discuss. Let me know if you have any questions.

                ×

                Login

                Question stats

                • Active
                • Views18633 times
                • Answers11 answers
                • Followers7 followers