Menu
The Chia Plot
  • Blog
  • How-To
  • About
  • Contact
  • Security
  • Discord
The Chia Plot
Server Room

What Chia Pools can do to protect themselves from online attacks

Posted on September 2, 2021September 2, 2021 by Chris Dupres

Yesterday a couple of Chia Pools were hit with a Denial of Service (DoS) attack and experienced some degree of downtime on their Chia nodes, and probably a ton of stress. Both PoolChia and FlexPool were running their pool Full Nodes in Amazon’s AWS. A couple of other pools also saw some spikes in traffic, but nobody else I spoke to was taken offline.

But what can the pool’s do to protect themselves? The first thing is, do not attach a public IP to the Full Node. If at all possible put the Node on a private network and forward requests back to it. If you have inbound ports NAT’d back to a private IP you are in a bit better position, but not significantly unless your firewall has some intelligence to it for dropping malformed packets. Most Chia farmers will be vulnerable to a DDOS, potentially, due to the nature of the software.

This is about to get very technical, with diagrams and everything, so if that’s something that will put you to sleep then now is a good time to get back.

Now, I haven’t seen any logs of what happened specifically to the pools, so the following is going to be an educated guess based on experience dealing with issues like this. The Chia Full Node client is, at it’s heart, an http web server. It uses TLS and TLS auth for communication with other nodes. You can check this out yourself by using NMap or another network scanning tool, but below you will see the results from my Intrusion Detection System and NMap output for sessions on port 8444.

Chia Port NMAP results
Chia Port NMAP results
Chia Session Details
Chia Session Details

The two most common types of DOS attacks that hit web servers like this are Layer 7 and Layer 4 (HTTP flood and SYN flood, in the most general sense). CloudFlare has a good writeup about these at a high level. A SYN flood attack can be handled by most normal firewalls assuming the volume doesn’t overwhelm their physical connections. If it does happen what you will is a whole pile of new half open connections on your firewall waiting for them to complete. This will consume firewall resources until it crashes or cannot accept new connections. This is the most traditional type of attack and the primary reason you should always run everything behind a firewall.

The other possibility is that it was an HTTP flood attack, on layer 7. This involves creating a huge number of HTTP requests that cost the server most processing time to deal with than the attack to create. This will frequently use a botnet, but doesn’t have to, and will keep the application layer of the server so busy that it cannot perform normal tasks. This is the most common form of targeted DOS attack at this point because it is much more difficult to mitigate at a network level. This is the kind of attack I have seen most frequently against this site, and why I pay for a Web Application Firewall.

My guess is that the specific DOS yesterday was layer 7, because I’m fairly certain AWS would stop a SYN flood at the network level, but I cannot be sure. Either way, the preferred solution would address both issues. In their post on the attack FlexPool once again places the blame on the Chia Full Node code, which is Python wrapped C and C++ from my understanding. While possible that their upcoming Golang Full Node could be more performant, there is no magical programming language that prevents flood attacks. The solution to attacks like this is architectural, not programmatic.

The key is to not expose the full node operating your pool directly to the internet, but it needs to be connected to other nodes in order to actually do its job. There are going to be 3 main ways to accomplish this, each with each one costing more in infrastructure and maintenance. The biggest roadblock to deploying truly secure pool infrastructure is going to be cost. When I spoke to J. Eckert at Chia Network about this issue he told me that they had all their stuff behind Cloudflare, and that should be standard. I did not argue with him, but that is not really a feasible solution for most people. Sure, the pool endpoints and other 443/https services can easily be put behind Cloudflare and you can get a WAF with it for $20/month. But to use a custom TCP reverse proxy with DDOS protection requires an Enterprise account which is going to be very expensive, hundreds or thousands of dollars a month expensive.

The first, and easiest – and should be the minimum – solution to this problem is to simply stick a TCP reverse proxy in front of your Full Node and allow no direct traffic to it from the internet. It should not even have a public IP address. All the communication from your node will be to the internal address of the proxy and all the other nodes will talk to the external connection. Even better if it has a Web Application Firewall module with it so you can address common layer 7 attacks. There are a number of open source solutions here, and while complicated is almost certainly worth it and not so complicated that pools will not be able to deploy.

Pool Infrastructure 1
Pool Infrastructure 1

The second solution is a bit more extensive and will require configuring a second full node. One to talk to the outside world, still behind a TCP reverse proxy and WAF, and one for your pool application to talk to. This would probably require some reconfiguration of the existing setup, and is a little more complicated but it would give your pool a small buffer if the first node gets taken out by an attack. This setup still has some risk, but it almost certainly going to be easier to sort out the first node while the pool is talking to the second, still safely up and just not talking to anyone. You can also, in an emergency, allow your protected node to reach outbound to other nodes if your first node is under a sustained attack in order to keep the pool up.

Pool Infrastructure 2
Pool Infrastructure 2

The following is a little extreme. It is the correct way to architect a multilayer application like this, but the costs can be significant. It can get even more wild than this with zero-trust solutions, but that requires so much additional infrastructure I don’t consider it a relevant option for even the largest pools. The following diagram shows a load-balanced set of full nodes. This ensures a highly available front end connection to the wider Chia network and allows for an attack, or a typical maintenance window, to take one node out without any disruption to operations. It also includes a back end set of load balancers to ensure that the Application node can access a Public Node at any time. This will ensure proper availability for a 24/7/365 service.

Pool Infrastructure 3
Pool Infrastructure 3

This is expensive. It might not happen. As we have discussed before the pools are running at very slim margins. Its possible Space Pool should adopt the 3rd architecture to ensure constant uptime but security costs money. It will be slightly cheaper for pools running on their own hardware, but anyone running on public cloud infrastructure will need to be careful about increasing operational costs too high with the price of XCH so low. But security cannot be an afterthought, and it has to be built right into the design.

Related

5 thoughts on “What Chia Pools can do to protect themselves from online attacks”

  1. Bill Nickless says:
    September 2, 2021 at 7:43 pm

    Hi Chris,

    Fundamentally, it seems that the mistake of the pool operators was to rely solely on Full Nodes that had TCP/8444 exposed to the public Internet. If they hadn’t allowed any inbound connections then wouldn’t they have been immune from any distributed denial of service attack?

    Yes, https://github.com/Chia-Network/chia-blockchain/wiki/Resolving-Sync-Issues—Port-8444 recommends every Full Node be exposed to the public Internet. That recommendation makes sense from the Chia Network perspective (as a whole). But as we’ve seen, that has the downside that each exposed Full Node is liable to Denial of Service attack.

    Perhaps another resilient Pool operator architecture would have a Full Node per cloud provider Availability Zone with private IP addresses and outbound-only connectivity to the Internet (NAT for IPv4, outbound-only filtering for IPv6). Call these the “hidden” Full Nodes.

    To be good Chia Network contributors they could run another couple Full Nodes with TCP/8444 exposed. Those publicly-exposed Full Nodes would be configured to always accept connections from the Pool operator’s “hidden” Full Nodes (c.f. config.yaml’s exempt_peer_networks). But if the Pool operator’s publicly-exposed Full Nodes struggle under a DDoS attack, the “hidden” Full Nodes could continue to get service from Full Nodes elsewhere that are not subject to or are better weathering the attack.

    My concern about your proposed architectures is that they don’t explicitly show that the Private IP Chia Full Nodes could reach out to the public Internet and thereby have access to a wider set of Full Nodes, increasing availability resilience.

    Reply
    1. Chris Dupres says:
      September 2, 2021 at 7:53 pm

      Thats a tough one because I don’t know how reliable a connection they need to other nodes in order to operate. And it would protect against random attacks against exposed 8444 services but wouldnt stop a targeted attack against an unprotected gateway.

      Reply
      1. Bill Nickless says:
        September 6, 2021 at 2:41 pm

        I should have been more clear about my assumptions when I wrote my comment above. Specifically, I was assuming that both the exposed and “hidden” Full Nodes were running in a public cloud infrastructure like Amazon AWS, Microsoft Azure, or Google Cloud Platform.

        Granted, if the internet-facing gateway of a “hidden” Full Node was a home cable modem, then yes it could be possible to identify and DDoS that gateway. But given the tens or hundreds of terabits/second of Internet bandwidth provisioned to major cloud service providers, I wish even the largest botnet “good luck” trying to take out the internet-facing gateways of those cloud service providers behind which the “hidden” Full Nodes would be reaching out to other Full Nodes. And, of course, many of the Full Nodes to which they would be reaching out would be operated by other customers of that same public cloud provider, further reducing their exposure to an external DDoS attack.

        Reply
  2. Ethan Waldo says:
    September 3, 2021 at 1:08 pm

    One middle-ground approach would be to write an eBPF to reject malformed Chia node requests.

    Reply
    1. Chris Dupres says:
      September 3, 2021 at 1:09 pm

      You need a waf or additional services running on the node for that to work though. I suspect anyone that can do that is probably not in this position.

      Reply

Leave a Reply Cancel reply

Advertisement

Recent Posts

  • Crypto is burning down – Chia seems fine
  • Chia CAT upgrade fiasco part 2 – Was I wrong?
  • WTF just happened?? CAT1 to CAT2 “upgrade”
  • The era of the Chia NFT is upon us
  • Chia Blockchain 1.4.0 released – NFTs and DIDs oh my
  • Discussion
  • Facts About Farmers
  • How-To
  • Information
  • News
  • pools
  • Security
  • Trademark
  • Trading
  • Uncategorized

Dark Mode Switch

©2021 The Chia Plot - Donate XCH / MRMT / SBX @ xch1p4440d6zwu9ryta2vx073lq2ge3s29d37kskz6t34jp085e8srjqnk0gcr
We use cookies on our website to give you the most relevant experience by remembering your preferences and repeat visits. By clicking “Accept All”, you consent to the use of ALL the cookies. However, you may visit "Cookie Settings" to provide a controlled consent.
Cookie SettingsAccept All
Manage consent

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. These cookies ensure basic functionalities and security features of the website, anonymously.
CookieDurationDescription
cookielawinfo-checkbox-advertisement1 yearThe cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Advertisement".
cookielawinfo-checkbox-analytics11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional11 monthsThe cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
JSESSIONIDsessionUsed by sites written in JSP. General purpose platform session cookies that are used to maintain users' state across page requests.
viewed_cookie_policy11 monthsThe cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.
Functional
Functional cookies help to perform certain functionalities like sharing the content of the website on social media platforms, collect feedbacks, and other third-party features.
CookieDurationDescription
na_id1 year 1 monthThis cookie is set by Addthis.com to enable sharing of links on social media platforms like Facebook and Twitter
na_rn1 monthThis cookie is used to recognize the visitor upon re-entry. This cookie allows to collect information on user behaviour and allows sharing function provided by Addthis.com
na_sc_e1 monthThis cookie is used to recognize the visitor upon re-entry. This cookie allows to collect information on user behaviour and allows sharing function provided by Addthis.com
na_sr1 monthThis cookie is set by Addthis.com. This cookie is used for sharing of links on social media platforms.
na_srp1 minuteThis cookie is used to recognize the visitor upon re-entry. This cookie allows to collect information on user behaviour and allows sharing function provided by Addthis.com
na_tc1 year 1 monthThis cookie is set by the provider Addthis. This cookie is used for social media sharing tracking service.
ouid1 year 1 monthThe cookie is set by Addthis which enables the content of the website to be shared across different networking and social sharing websites.
Performance
Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.
CookieDurationDescription
d3 monthsThis cookie tracks anonymous information on how visitors use the website.
Analytics
Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc.
CookieDurationDescription
__gads1 year 24 daysThis cookie is set by Google and stored under the name dounleclick.com. This cookie is used to track how many times users see a particular advert which helps in measuring the success of the campaign and calculate the revenue generated by the campaign. These cookies can only be read from the domain that it is set on so it will not track any data while browsing through another sites.
_ga2 yearsThis cookie is installed by Google Analytics. The cookie is used to calculate visitor, session, campaign data and keep track of site usage for the site's analytics report. The cookies store information anonymously and assign a randomly generated number to identify unique visitors.
_gat_gtag_UA_199099757_11 minuteThis cookie is set by Google and is used to distinguish users.
_gid1 dayThis cookie is installed by Google Analytics. The cookie is used to store information of how visitors use a website and helps in creating an analytics report of how the website is doing. The data collected including the number visitors, the source where they have come from, and the pages visted in an anonymous form.
CONSENT16 years 4 months 5 daysThese cookies are set via embedded youtube-videos. They register anonymous statistical data on for example how many times the video is displayed and what settings are used for playback.No sensitive data is collected unless you log in to your google account, in that case your choices are linked with your account, for example if you click “like” on a video.
Advertisement
Advertisement cookies are used to provide visitors with relevant ads and marketing campaigns. These cookies track visitors across websites and collect information to provide customized ads.
CookieDurationDescription
advanced_ads_browser_width1 monthThis cookie is set by Advanced ads plugin.This cookie is used to measure and store the user browser width for adverts.
anj3 monthsNo description available.
CMID1 yearThe cookie is set by CasaleMedia. The cookie is used to collect information about the usage behavior for targeted advertising.
CMPRO3 monthsThis cookie is set by Casalemedia and is used for targeted advertisement purposes.
CMPS3 monthsThis cookie is set by Casalemedia and is used for targeted advertisement purposes.
CMRUM31 yearThis cookie is set by Casalemedia and is used for targeted advertisement purposes.
CMST1 dayThe cookie is set by CasaleMedia. The cookie is used to collect information about the usage behavior for targeted advertising.
DSID1 hourThis cookie is setup by doubleclick.net. This cookie is used by Google to make advertising more engaging to users and are stored under doubleclick.net. It contains an encrypted unique ID.
i1 yearThe purpose of the cookie is not known yet.
IDE1 year 24 daysUsed by Google DoubleClick and stores information about how the user uses the website and any other advertisement before visiting the website. This is used to present users with ads that are relevant to them according to the user profile.
KADUSERCOOKIE3 monthsThe cookie is set by pubmatic.com for identifying the visitors' website or device from which they visit PubMatic's partners' website.
KTPCACOOKIE1 dayThis cookie is set by pubmatic.com for the purpose of checking if third-party cookies are enabled on the user's website.
mc1 year 1 monthThis cookie is associated with Quantserve to track anonymously how a user interact with the website.
test_cookie15 minutesThis cookie is set by doubleclick.net. The purpose of the cookie is to determine if the user's browser supports cookies.
uid1 year 1 monthThis cookie is used to measure the number and behavior of the visitors to the website anonymously. The data includes the number of visits, average duration of the visit on the website, pages visited, etc. for the purpose of better understanding user preferences for targeted advertisments.
uuid3 monthsTo optimize ad relevance by collecting visitor data from multiple websites such as what pages have been loaded.
uuid23 monthsThis cookies is set by AppNexus. The cookies stores information that helps in distinguishing between devices and browsers. This information us used to select advertisements served by the platform and assess the performance of the advertisement and attribute payment for those advertisements.
VISITOR_INFO1_LIVE5 months 27 daysThis cookie is set by Youtube. Used to track the information of the embedded YouTube videos on a website.
YSCsessionThis cookies is set by Youtube and is used to track the views of embedded videos.
yt-remote-connected-devicesneverThese cookies are set via embedded youtube-videos.
yt-remote-device-idneverThese cookies are set via embedded youtube-videos.
Others
Other uncategorized cookies are those that are being analyzed and have not been classified into a category as yet.
CookieDurationDescription
__gpi1 year 24 daysNo description
adImpCountpastNo description
C3UID5 yearsNo description available.
C3UID-9245 yearsNo description
fc5 months 27 daysNo description available.
pfpastNo description
pxs5 months 27 daysNo description available.
SAVE & ACCEPT
Powered by CookieYes Logo