[Solved] [Known Issue] We are seeing some errors with some email campaigns and private domain/branded links showing errors and non SurveyGizmo pages.

0

Our System Administrators are working hard on this issue as we speak, we hope to have it 100% corrected soon!

Taom answered

    Best answer

    0

    Hey Everyone,

    This is all fixed up!  Our System Administrators worked hard to fix the domain issue as soon as we could.  However, you may still be seeing some issues for a brief time until everything clears up on the interwebs.

    DNS changes can take a few minutes to a few hours to propogate, so it’s mostly resolved now, but some users may still get a bad link.

    We are so sorry that this happened, we of course are taking steps to make sure this NEVER happens again.

    -SG Engineers

    tvadney commented
      • Hi Jon,

        Will users with bad links need new ones? Or once DNS finishes propogating, it will work?

        Thanks,
        Matt

      • Hi Matt,

        As far as we know at this point, once DNS finishes propogating, these links will work. It’s a little tricky for us at this point, since we cannot replicate this issue, as our Time Zone wasn’t yet affected by the change that caused these errors.

        If a few hours go by, and the links still aren’t working, please let our Support Heroes know you are still running into issues.

        Jon

      • Thanks Jon,

        Can you please provide an a little more in depth explanation of the issue/what caused this issue? And provide the downtime experienced (from notice to resolution)? I have several internal and external stakeholders I need to inform and they will want this information.

        Thanks,
        Matt

      • Hey Matt,

        No problem at all, we are so sorry that this happened. Basically, one of our domains, sgizmo.com, was briefly expired this morning due to a communication issue between our domain host and our IT team, and we worked to renew this domain as quickly as possible, and have fixed things so this will hopefully never happen again going forward. The downtime was between approximately 7:00 am to 8:33 am MT and was very sporadic. The impact was sporadic as it depended on very specific behavior/features/domains and propogation of DNS across the internet. Email Campaign links (more-so in Europe than in the US) and some Private Domain links were possibly impacted. I hope this helps!

        Jon

      • Thanks Jon,

        That is helpful. I know this will take some time to resolve itself, and in the meantime some links will work, some won’t. However, is there a time where your team feels confident in saying that all links will be working properly again?

        Thanks,
        Matt

      • This is happening now to a live survey that was released at 12:48 PDT. Select recipients are being directed to:

        http://s-e194c5-i.sgizmo.com/s3/i-bYobnXh6pzBspw9gB-1337382/?sguid=bYobnXh6pzBspw9gB

        when clicking through on the survey link.

        Using branded Pro service with whitelisted SPF in place

        PLEASE FIX ASAP. THIS IS 100% UNACCEPTABLE!

      0
      Taom 10 Rep.

      This problem has not yet been resolved for us here in India. It’s been 2 days now. Could you give us an alternative. We have surveys running and the respondents are not able to get into the survey yet. 

      Taom answered
        0

        This is continuing to be a problem for my clients. We have more than 200 links affected by this outage.  As of 7:30 in AZ (PST) the links were working for me and my clients in VA (EST). But a main end client with 27 links is still redirecting to the register page — using a new browser where cookie cleaning would not matter. That client is in CA (PST).  

        What are others seeing? 

        It has now been 24 hours since this was supposed to have been solved. What gives?

        Please provide us communication or set up some sort of outage report map so we know where issues still remain. 

         

         

        Topper Shull commented
          • Thank you for letting us know this is continuing to happen for your specific clients. I am reaching out by email to you now to keep working on solutions.

          1

          Morning, 

          Our clients are still seeing an issue this morning (UK time). This is embarrassing when we told them yesterday that it was fixed.

          Please provide an update.

          Thanks,

          Kirsty

          Topper Shull commented
            • Hello Kirsty,
              As an update, we have seen widespread resolution of the problem around the world, but are also fielding final cases where the issue is still being cashed in locations and for companies/respondents that have not yet had their system refresh to have an update.

              Thank you for reaching out to us by email to discuss with my colleague Josh. We will reach out shortly.

            0
            Erik 13 Rep.

            Guys,

             We received ZERO notification that there was an issue with this today and it resulted in a significant decrease in revenue.

             

            Please be sure to let your customers know that things are broken, especially when they are spending money to drive traffic to your service.

            Topper Shull commented
              • Hello Erik,

                Thank you for following up with us on the damage this has caused your team. I thought I would reach out to you via email to discuss. Let me know if you have any questions.

              0

              It would have been nice to know about this BEFORE sending out the survey link this evening.  I would have waited.  Perhaps in the future you can add a warning message to the “share” page so people getting ready to send out the survey can opt to WAIT until it is FIXED.

               

              Topper Shull commented
                • Hi Sharon,

                  I believe this is a great practice we can take going forward. Our current methods of alerting about problems in this vein, should they occur, is through this Known Issues area that you can click “Follow” on the following page to receive alerts when we have issues:

                  https://community.surveygizmo.com/questions/category/known-issues/

                  We also update our Status Page whenever we have such an incident. You can also subscribe for emails for the future here:

                  http://surveygizmo.statuspage.io/

                  I truly am sorry for the damage this has caused. We have fielded a good amount of feedback on the ripple effect this has had on our customers and it has helped us put new processes in place to avoid this problem going forward. We are also working with anybody effected or continuing to have concerns going forward by email at support@surveygizmo.com.

                  Please reach out if you have any further issues.

                ×

                Login

                Question stats

                • Active
                • Views19488 times
                • Answers11 answers
                • Followers7 followers