Scary Tickets😨

Before you start reading the blog, I highly recommend reading Inti's Ticket Trick blog to get an idea of what this overall means. Here is the link: https://medium.com/intigriti/how-i-hacked-hundreds-of-companies-through-their-helpdesk-b7680ddc2d4c

When the blog was originally published, a lot of hackers including me were super excited on how it worked and wanted to test it in a live target. Multiple hackers found and reported it in bug bounty targets that used it. As companies started to fix the vulnerability, I started to wonder how Zendesk and other support systems worked and if there was a bigger issue that was missed or overlooked. This year, while working at my school's security club, we decided to do a workshop titled "Threats with third-party services". This gave me the perfect opportunity to understand how these support system work and understand different ways to exploit them. In this blog, I will break down how I used Zendesk and Help Scout to exploit the ticket trick yet once again in a more dangerous way with an interesting attack vector. Throughout this research, I worked with two really good friends of mine: @cmdrsnuggle & @rawalshree

Researching Zendesk

Before sending spam emails to companies' support system, I used Zendesk and researched on how some features worked. This required endless hours of works such as researching Zendesk features, reading community support pages and understanding documentation.

When starting to understand Zendesk, something that I was super curious was about email and ticket threading. If you have used Zendesk before, you will know that when you reply to a ticket through an email, the content of the email gets added to the ticket. Inti originally did this by emailing support+id<ticketid>@domain.com, but I was interested to see if there were other ways to get that email added to the ticket. During this, I found this Zendesk documentation: https://support.zendesk.com/hc/en-us/articles/360000334947-Getting-started-with-email-Part-2-Incoming-email-requests-and-notifications#topic_r1f_4gd_v3

In this article, Zendesk outlines three ways on how it matches an incoming email to a ticket:

  1. Emails are checked for In-Reply-To and References headers. If those exist, they are used to add ticket content to the email
  2. support+id<ticketid>@company.zendesk.com is present in the receiver's email. If that is the case, email content is added to the ticket.
  3. There is a hidden reference to the ticket in the email body.

The process outlined in step number 2 is something super close to Inti's so I decided to ignore that and focus on step 1 and step 3. Step 1 is not feasible in an attack because when a third party company sends an email to my "victim" I cannot control the references and in-reply-to headers. Now, because I wanted to bring Ticket Trick back, I decided to check step 3.

To understand step 3, I went ahead and sent my research system an email:

Once the support system received my email, I got an automated reply from it. This is often how tickets in Zendesk are created. The automated reply is an acknowledgment from the support saying it received the ticket.

Cool, so now I had the automated response. The goal was to find the hidden reference to the ticket in the email body. To do so, I used the show original feature on Gmail to see what was in the email body.

The first few lines of the email were the email headers until ----==_mimepart_5c7df374e772c_27fa3f8b3e4bcf18131485 . After this, the email body started.


There are two ----==-mimepart on the above picture. The second part is the HTML body of the email and first is the plain text. This is designed so that email clients that only accept plain text email will also be able to accept the email and display it. Everything looks normal until we go pass the This email is a service from Zendesk. After that, we see an interesting alphanumeric string: [M7VRMO-LV6L]. This looked super interesting to me so I decided to copy that and save it on my notes to begin the next phase of the testing.

So far, I had only sent the request as a customer. Now I wanted to see how it will look like on the Zendesk agent side. This is what I had so far on my agent side:

Everything looked good so far, so I asked my research partner Brandon Nguyen to send an email to support@redacted.zendesk.com with the subject set to [M7VRMO-LV6L] (we will call this hash from here). I wanted to see what it looked like on the Zendesk side when an email with the hash is not sent by the original sender. Something was super weird on the agent side then:

Soon I found out that, the email he had sent went as a private note in the ticket instead of a public comment that I as the original sender could see. I wondered how that was working so I decided to check the ! sign it had: Brandon Nguyen was not a part of this conversation. Right next to it, there was a link explaining why this occurred: https://support.zendesk.com/hc/en-us/articles/203661606#topic_d32_mzc_3r

Something in that article stood out to me: If you want to allow the new user to comment publicly on the ticket, an agent, the requester, or a CC on the ticket needs to add them as a CC. Now that I knew that the current sender also had to be CCed in, I went ahead and added my friend's email as a CC to the ticket. After this, I asked him to send another email with the hash as the subject. As suspected, the email had gone through and was a public message:

Zendesk Agent Portal
Customer side email thread

This concludes the Zendesk analysis. Let's analyze one more support system before diving into exploitation.

Analyzing Help Scout

When looking for Zendesk alternatives, I ended up finding another support platform known as Help Scout. This system runs in the same way Zendesk minus some features that Zendesk has. To analyze this system, I decided to signup for the trial and understand the features.

Understanding Help Scout

To better understand how Help Scout works, I decided to follow the same process that I had with Zendesk. Except, in this case, Help Scout did not have much documentation focusing on how their system worked so I decided to investigate this myself. Like always the first process started with me sending an email to my test instance. After that, I received an automated response from my instance because I had enabled Auto Reply feature on Help Scout.

Once I had the reply, I decided to see if this system also had a similar pattern like Zendesk. I decided to check the source of the email and quickly found that there was a hash included in this email as well:

Once I had the hash, I decided to check if this will also add my email content to the ticket. To do so, similar to Zendesk, I sent an email from my second account to the support email. In this email, the hash was added on the body.

Once the email was sent, I was able to confirm that Help Scout, similar to Zendesk will also append my email to the ticket. However, there was a small problem. When checking on the agent side, I realized that my second email was now the original sender. This meant that any future connection from the support agent will go the second email and not the original email (attacker).

This made me realize that it might be hard to exploit this because my attacker email was no longer getting any response. However, when checking back I noticed that the only reason my second email was now the "original sender" because it sent the most recent email.

So, all I had to do now was send a new email from my attacker email to the ticket with the hash on the body. This will put my attacker email back to the "original sender" list allowing me to receive all the future email correspondence.

Now that we have gone over the analysis of these two support systems, lets see how this can be exploited.

Exploiting Zendesk Configurations

The whole part of analyzing Zendesk was to see how this could be used to exploit it against a customer. To make sure I was not rebuilding the wheel in a way, I decided to give a second read to Inti's report and see if he had missed any attack vector. In his report, he talks about exploiting similar systems to get access to Slack and Yammer. One thing I realized could have a more drastic effect was getting a valid @domain.com email in Google and exploiting access from there. In this part, I will give examples from the report I submitted.

Starting the attack - Zendesk

Getting the Google Account

Each test began with finding the companies' support emails. To do I used hunter.io to get a list of all potential support emails. Once I had the email, I would send out a simple email asking for help on my account. As previously explained, if the service was using Zendesk, there will most likely be an automated reply confirming the email. Once I had this, I grabbed the hash just like from the original analysis.

Then, I went to accounts.google.com and started the signup process. Google has a signup feature that allows you to signup for your domain. This is a useful process if you want to register on Google with your company's domain to use things like calendars, groups etc without having to create a GSuite account or signup to GSuite services. Another interesting side of this is that most Sign in with Google use Google's OpenID and only check for the domain of the email when a user is signing up or signing in. This was what made the attack dangerous.

To get a valid support+1@domain.com email, I first added no-reply@google.com as a CC to the ticket by replying to the automated reply. Once that was done, I went to Google and signed up. For the first name and last name, I put the hash of the ticket. This was done because when sending a verification code, Google sends an email to support+1@domain.com with the hash in the email body under Hi {firstname} {lastname}. After each successful tests, I had a verified @domain.com to be used in most Sign in with Google pages.

Exploiting with the verified email

Internal Domains

After getting a verified email in Google, I wanted to see if I could use this in any place that was internal or sensitive for the company (Gitlab). One of the sites is https://prometheus.gitlab.com/ which as noted requires a @gitlab.com domain to sign in.

However, it is important I make a clear distinction between two Oauth designs that can exist: In some sites, the Oauth uses a simple sign in that checks if the domain uses to sign in matches the company domain (gitlab.com) and checks if the user account exists in the system (Prometheus). In some services like Prometheus, user accounts are auto-provisioned which means a user with @gitlab.com that does not exist in the system can still login with limited permissions. In some cases, companies use GSuite based Oauth. Even in these cases, you see the same Google login, but the login procedure in the back is different. The login function will check if the email exists in the GSuiet users list of that organization. Any email verified through Zendesk exploit will not be in GSuite org because you can only verify non-existing email in Google. If the company uses GSuite SSO you will not be able to exploit the sign in process.

Going back go Gitlab, I tried to sign in to Prometheus with the account I had. Fortunately (for me), Gitlab was using the regular Oauth which then allowed me to login to their Prometheus instance (image not supplied for security reasons).

Atlassian Instance

In some cases, I could not find any internal domain to login with. One of the company tested had recently migrated all their Oauth to GSuite SSO so none of my attempts to login to internal domains worked. However, in some cases, I found that companies were running their JIRA instance through Atlassian's cloud feature. Instead of installing JIRA locally on your own server, you can run JIRA out of box directly on Atlassian's own domain (atlassian.net). Most of these instances were found to be allowing anyone with the company domain to signin and view the JIRA tickets. For this company, even though all the instances were moved to GSuite SSO, they forgot to move their JIRA login to GSuite SSO which allowed me to login and view the JIRA tickets in the system.

Internal domains with Google login and Atlassian are not the only vector with this vulnerability. You can also exploit this on other third party services like Zoom and Asana to gain access to company restricted resources including recorded videos (Zoom) and internal project management (Asana). We will cover the limitations of this attack in the next section once we cover the Help Scout exploitation.

Starting the Attack - Help Scout

Getting into Slack instance(s)

One of the most exciting findings that Inti had on his support system test was getting into company's slack instance. When he tested Zendesk, he found out that if at anytime you could signup on a company's Zendesk instance with no-reply@slack.com you could join their Slack instance because Slack sent invitations through the no-reply email. Once this was found, Slack also decided to protect their customers by adding a random alphanumeric number after no-reply so an email could be something like no-reply-940819dh9uhfy87sgf@slack.com and attacker will have no clue how to guess that. With Help Scout, things got quite interesting when exploiting Slack instances.

Before we get into the Slack exploitation portion it is important to highlight some features of Slack that makes this attack easier. Slack just like other services provides different plans for companies to purchase from. Free is one of the most simplest one with not much enterprise tools like SSO integrated on it. Slack instances can also have a domain restriction that allows them to limit who can join the Slack automatically. It [Slack] also provides a feature to find all the workspaces you can join, are invited to and are part of. This is useful if you have a lot of Slack workspaces that you joined and lost track of one specific instance. In this research, we exploited each of these features + Help Scout to gain access to internal Slack instances of companies. It is important to highlight, we never joined any of the Slack instance for obvious reasons. We only joined one Slack instance but that was after an explicit written permission from the company as they watched us signing up to make sure we did not act malicious.

For this specific exploit, company means the target that we found were vulnerable to Help Scout attack. Once I emailed the support@company.com, I got an autoreply and found out that the company uses Help Scout. Once I knew it was Help Scout, I got the hash token and started my exploit.

1. Creating my own Slack instance

So the first phase of this exploitation will be to create our own Slack instance. We will get into details of why this is the case soon. When creating a Slack instance, for the workspace name I provided the hash token of the company in the name. Once I had done that, I went to Slack's invite feature and invited the support@ email to my Slack instance. Once I had done this, my Slack instance is now added on the list of instances that this email has been invited to.

2. Lets find the instances!

Slack has a really nice feature at https://slack.com/signin/find. This allows you to get list of all Slack instances you are part of, you are invited to and you can signup to by just providing your email. Slack will send you all this detail on an email once you submit your email. Because of the first phase, our instance is considered one of the invited Slack groups. When Slack sends the email with the list of instances you can access, it lists the out by their name and URl along with a hyperlink to join. Because our Slack instance is in the list, it will get added to the email as well. Notice that for the name of the instance, we have the ticket hash token. What this means is that the complete email that Slack sends to support@company.com, is now threaded/added to my support ticket.

3. How do I get that email though?

Now that we have the Slack lists, we still have to be able to see it. Currently it is the embedded on the ticket as a public comment but unlike Zendesk, Help Scout does not automatically send the content to my attacker email. Also as I noted above on the Help Scout research phase, Slack's email will now be considered the "original" sender because the most recent email to that ticket came from Slack's email. So to make sure I get the next set of email/replies on that ticket and see the previous email threads, I have to make sure my attacker email is the one in the "original" sender option. To do so, I reply to the first email that the support sent and say something generic like "Thank you". Then we wait.

4. Exploitation

In order to understand how to exploit this, let me go through some Slack terminology on that email. This is what the threaded Email might look like:

In the image, there is one specific hyperlink called Join. This hyperlink gets a simple hyperlink with a token that allows you to signup directly. In cases with Join you can directly get access to private Slack without much work.

Slack also has a hyperlink in these kind of emails called Launch. It is a one time use magic link that allows you to directly login to the user account. This hyperlink will only exist if the support@ email is a registered user on the Slack instance. At the same time, this magic link also has some limitations to it. Launch magic link is only active for 24 hours. So if the support agent does not reply to the ticket within 24 hours of exploitation the magic link will not work. I overcame this kind of issue in some slack instances by using support+1@company.com if the company allowed anyone with @company.com to register.

Attached below is the video that shows an exploit through Help Scout in full phase.

Take out from this research

Encountering misunderstanding

One common pattern that we saw when researching support systems is that most companies using them were not aware of some features and how they could be exploited. All the features we used (ticket hash in Zendesk and ticket hash in Help Scout) are features that companies unknowingly use everyday but never realized the threat it could bring to them. In some cases, we were asked if we exploiting a 0-day on Zendesk because we were able to gain access to really sensitive materials with this. However, in each case like that, we had to explain to the company why it was not Zendesk's fault and had to point them to correct resources to explain why this worked the way it did. Similarly, some companies also reached out to Zendesk to ask them what was going on because some of them were confused and worried about how this worked. To make sure everyone was in the same page, we also sent an explanation to Zendesk (through HackerOne's support team) so that Zendesk knew what the exploit did.

What I learned from this research was to be patient and work with companies in fixing this vulnerability. This was a really unique case for me because usually I find vulnerabilities like RCE, SSRF, IDOR etc and it had been a while since I did some really big research on third party systems (my last big one was on Sendgrid and Google's GSuite). I am really thankful for companies being patient with us as we worked with them. Because this was a new attack for us, we found multiple ways to exploit this and kept updating companies on how this could be leveraged for gaining critical internal access. In most cases, all companies were really appreciative of the work and really happy with the level of the research done. One particular company pushed out a fix within 1 hour of the report and conducted a security team wide meeting to discuss about it further. For another company, we actually got to work with the engineer on-call to discuss potential fixes and the pros and cons of those fixes.

Working in a group

I don't usually hack in groups a lot but this was a great learning experience for me. As a team, we learned how to best manage our resources and find companies that were vulnerable. For example, to make sure we knew different avenues of exploitation, we wrote down detailed explanation of attacks that we could do and shared with each other. Once this was done, we made a target list of companies that we could test. I knew some engineers at these companies so it made it easier for us to reach out to them when we were able to exploit the issue. Working together not only helped us exploit this issue properly, but also write a well formatted report that companies we reported to were able to understand. In the end, for us what mattered was that companies clearly understood the exploitation and the impact. This has also made me as a hacker really appreciate how group hacking makes it fun to hack and exploit something. We were not only able to find these vulnerabilities and report them but also were able to found unique ways of exploitation if our original attacks did not work. For example when testing Gitlab, we tried finding services they used like domain on Atlassian.net but were not able to access it. That is when we decided to test some internal Gitlab domains and see if we could login to them. This tweet sums up how it felt working in a group:

https://twitter.com/uraniumhacker/status/1098437702544805888

Statistics

We tested about 25 companies for this and found 20 to be vulnerable. In all cases, we were able to gain some internal access. Here are some applications we accessed (with permissions):

  • Slack
  • Asana
  • Internal domains
  • Zoom
  • JIRA & Confluence through Atlassian.net
  • Priv esc in the company's own domain with a registered @company.com account.
  • Internal support portal in Zendesk that was only open for internal employees. Contained JIRA tickets for some API issues.
  • Dropbox

Zendesk fixes and limitations

After companies emailed Zendesk about the vulnerability reports we sent, Zendesk decided to limit the easy exploitation by blocking any no-reply@ emails and sending them to Spam directly. This does not mean all companies are secure. There are third party services that send verification email from anything other than no-reply@ so these emails will still end up being added on the Inbox. That is on the readers to find new systems where this could work on.

Help Scout Fixes

Help Scout has not pushed a major fix for this issue because it requires changing the whole platform structure. For now we recommend adding a email filter blocking no-reply@ emails and any emails that seems automated (verification emails, forgot password emails etc). Also informing support agents to not reply on possible interesting/malicious emails is another security posture to establish.

Example report(s):