Steem Status: 100% Operational

2개월 전

Steem Operational.jpg

Hello Steemians, we apologize for the interruptions in service you experienced yesterday which are now resolved. Thanks to the blockchain’s built-in safety measures, at no time were funds at risk. The interruptions were due to an issue that arose out of the process of transitioning to Hardfork 20. In our last post we announced that we published our Hardfork 20 release and that witnesses and exchanges should begin running version 0.20.0 of steemd at their earliest convenience. When a witness runs the new code, it signals that they are voting in favor of those changes. If a super-majority of witnesses are running Hardfork 20 on September 25th, the blockchain will fork onto the new chain.

Blockchain Halting

Until the date of the Hardfork, both versions produce blocks on the same chain. This gives the witnesses time to test the code as well as cast their vote for the new changes by running the new version. Yesterday, that chain unexpectedly forked when the 0.20.0 nodes produced blocks that did not match the consensus of nodes running version 19 of steemd. This activated one of the many safeguards built into the blockchain which pause block production due to the presence of some unforeseen error. These safeguards guaranteed that no important information, including token balances, was at risk.

The Problem

The specific cause of the problem was a bug in the HF20 code. A constant was changed that should not have been, and this caused the initial fork. Your witnesses did a fantastic job of quickly spotting the suspicious behavior and began reverting to a stable build of the Steem blockchain. This exposed another bug that resided in our fork database logic that was present due to the infrequency that this logic is activated, and which caused the minority fork to halt as well.

Our blockchain developers worked extremely hard to identify the problems quickly and issue a patch that would enable the witnesses to resume block production. They assessed that the 19 fork (version 0.19.X of steemd) was not the problem and advised the witnesses to revert to that version and restart their nodes. 19 of the witnesses responded by restarting their nodes, returning stability to the network. Steemit Inc.’s production nodes are now fully functioning as is steemit.com.

Next Steps

We have already performed a post-mortem on the event and have drafted an attack plan for developing the tools and resources which will ensure that events like this will not happen again. But before we can get to work on those, we’ll be spending the rest of today investigating the fork database code in order to ensure that it’s prepared for the hardfork date of September 25th.

The Cost of Innovation

When we launched the Steem blockchain it was with the goal of creating a protocol that could power real applications that real people use every day. That goal, along with the governance mechanism we chose, has enabled us‒as a community‒to understand what changes need to be made and act on that information by upgrading the software through hardforks. So far we have successfully hardforked 19 times in just 2 years.

This rapid rate of iteration is one of Steem’s uniquely valuable attributes and is virtually unheard of in the blockchain space, which itself is still in a highly experimental phase of development. The cost of this rapid innovation is incidents like the one that occurred yesterday, and the fact that so many people were unable to use the apps they love, is a testament to how successful Steem has been at fostering the development of functional applications that real people use every single day. Yesterday also demonstrated, once again, that our community has the ability to work together to handle these issues so that we can continue to push blockchain technology forward at a more rapid pace.

Thank You

We want to express our gratitude to Steem’s amazing community of witnesses, developers, and users. Blockchain-halting events are pretty much as bad as it gets for a blockchain, but everyone worked together to ensure service was returned as rapidly as possible. It’s a testament to the power of this community that such an event was handled so effectively. Many people outside of Steemit Inc.‒especially the witnesses‒shared vital information that enabled us to respond more rapidly, and return the platform to proper operation.

You don’t lose if you get knocked down; you lose if you stay down. - Muhammad Ali

Steemit Team

Authors get paid when people like you upvote their post.
If you enjoyed what you read here, create your account today and start earning FREE STEEM!
STEEMKR.COM IS SPONSORED BY
Sort Order:  trending

Not all is fixed

Far as I can tell, Nodes that exchanges use are down. This means no deposits and withdrawals. One can send STEEM to the exchange, but the exchanges won't know and user's STEEM won't show up on the exchange.

Here is a list of Broken Exchanges

  • Binance
  • Bittrex
  • Poloniex
  • Huobi
  • Bithumb
  • Openledger DEX
  • HitBTC
  • Gopax
  • UpBit

What is Steemit Inc doing about this problem?

·

https://market.rudex.org/ works well, with 0% trading fee for STEEM/SBD btw

In a course I took about creating product loyalty, we were taught that having a problem and handling it well often creates more loyalty, than not having a problem at all.

Yesterday was a showcase in how many people are engaged and interested in the site, even those who voice frustration were concerned and interested in the outcome.

I know I could be over communicating this message, but I really do see improvement in the communication coming from SteemIt, Inc.

Thank you for the information.

·

As a point of improvement, I believe at least https://steemit.com/ temporarily should have been displaying some information during such an outage.

·

All the more reason why top 20's should be developers, or have someone on their team who is! Nice work everyone!

·
·

Exactly. And the servers should also be hosted by the witness himself/herself or somebody in the team.

·
·
·

That is a no-brainer. I'd be stunned to hear it was otherwise...is it otherwise?

·
·
·
·

It is otherwise. :) There is a large witness that rolls out witness servers for other witnesses.

·
·
·

steem power inequality asides, it's everyone's imperative to vote for witnesses that they feel is best suited for the job.

·
·
·
·

the voting mechanism should be more transparent within the steem website. it has been 4 month that I am using steem and I still don't know how and where to vote....

·
·
·
·
·
·
·
·
·
·
·

thanks! was about to link there too.

·

we were taught that having a problem and handling it well often creates more loyalty, than not having a problem at all.

Is that the same as someone who deliberately creates a problem, and then handles it well?

(thinking cap on)... in history, who do we know who does that? ::chuckle::

·
·

Actually, when you are breaking new ground and developing new industries, mistakes happen. You deal with them, learn, and move on.

That is not the same thing at all as purposely creating/pointing out problems and pretending to fix them. Which is also a strategy used by many. Especially politicians and companies that profit from fixing things. :) The impact is close though.

·
·
·

Politicians. Hmm... I was waiting for that to appear. Yes. I agree with you.

·

Thank you Steemit, for making my point with every blockchain seizure.

I can't wait for the other "edge cases" to manifest themselves as more "functionality" is bolted on to the shuddering PoS chain.

I know, I know - everyone loves Steemit and anything to the contrary is blasphemy.

I'll see myself out.

·
·

It isn't blasphemy at all. It is just really weird to hang aroundthe place if you don't like it and don't trust it and aren't willing to assume the risks.

I am often confused. I don't like myspace, but I never, ever, log in to tell them what I don't like about it.

·
·
·

Seriously? I hang out at myspace every day and tell them what they are doing wrong. Hopefully, one day they will get the message!

·
·
·
·

I see someone is emotionally invested.

If you can't handle some criticism of Steemit, then what are you going to do when it seizes up like a rusty lawnmower next time?

I'm sure it will be instructive.

·
·
·
·
·
·
·
·

haha

·
·
·

Why not enjoy the decline? Being proven right is its own reward, after all.

Also, why would you care? Isn't Steemit glorious and amazing - so much so that you don't have time to reply to me? What? It isn't? Well, whaddaya know...

·
·
·
·

I reply to most of my comments, not all. check it.

·

Well said @whatsup!

·

I agree!!

It was nice having the Steemit Twitter page to stay up to date on the status!

·
·

Thanks for mentioning the twitter page, I was unaware

Posted using Partiko iOS

·

While we'd rather the incident hadn't happened, such events do show off the strength of the ecosystem. I think people use these incidents to gauge the degree of antifragility within a system. They are demonstrations of its ability to grow stronger from volatility. Good communications can highlight this aspect of the system.

The engagement and interest you mention is, IMO, the secret sauce of any such system. So many projects out there think that the solution to all of their problems are algorithmic-first, and if they can just design the right algorithms, that will solve all of their problems when in reality networks of people solve problems. The most elegantly designed "on-chain governance" will crumble if no one uses the software and likely will collapse when they do use it because then it will be forced to interact with unpredictable and irrational (not in a bad way) human actors. The only way to develop the right algorithms, build the right software, and establish functional governance mechanisms, is to build a network of engaged and interested people who are constantly kicking the tires, providing feedback, and influencing change in a positive direction. We don't just have that, the Steem community is years ahead of the competition. People often make the mistake of thinking that Steem should be defined as what it has been in the past, when the real power of Steem is what we will work together to make it in the future.

·
·

in reality networks of people solve problems.
@andrarchy

That’s well said.

Also see:

How to always win on Steemit!
·
·

Bitcoin has also had one major outage which needed manual intervention from the miners - https://github.com/bitcoin/bips/blob/master/bip-0050.mediawiki

Bitcoin used to have a centralized alert system, where urgent messages could be broadcast over the Bitcoin network by anyone having the private keys for the alert system. This alert system has later been removed from the Bitcoin Core client. I think maybe it would make sense to set up some alert system where any of the top 20 active witnesses could send an alert. I don't believe false alerts would be a big problem, since such an alert message (hopefully) would cause the witness to lose votes.

·
·
·

Thanks for the feedback. I don't think witness responsiveness has ever been, or was in this case, an issue but it certainly might be worth considering. It's important to remember that any time that is spent developing one solution is time not spent developing another. As Steve Jobs famously said, "People think focus means saying yes to the thing you've got to focus on. But that's not what it means at all. It means saying no to the hundred other good ideas that there are."

·
·
·
·

I wish the success of the platform! Hoping you release more youtube videos Andrew; you're a great spokesman.

·
·
·
·
·

Thanks! I'll work on it!

·
·
·
·
·

I wish the success of the platform! Hoping you release more youtube videos Andrew; you're a great spokesman.
@kevbot

Yes. @andrarchy
Why not create a video explaining how this problem was fixed?

·
·
·
·

True. The alert-system is probably not worth pursuing - but I do believe it's rather important that nodes stays up and serves read-only-requests even if the block production is halted.

·
·
·

Many witnesses have alarms set up if they have missed blocks.

·

I was happy that there the problem did arose. We are the only blockchain that encounters problem because a) there are so many users using the blockchain,
b) The inevitability of stumbling bugs at rate of upgrades and forks we are expereincing.

It's something that we in fact can be proud. As you said, it's better than not facing any problem at all.

·

Well said boss

Posted using Partiko Android

It seems like all the nodes went offline and all interfaces to Steem (like steemit.com) got broken. Wouldn't it be a lot better if the nodes could continue serving content up until the "uncle block", and that the interfaces could issue a status message, something like "blockchain temporary frozen due to technical problems - posting and voting won't work"?

·

imho, it maybe a lot more complicated than that, but yeah sure there could be more sophistication but that requires prioritization and surely more available quality resources and that will all happen in due time

·
·

It's not the first time the network freezes, and I believe it's not the last time either. As I see it, this is a major trust issue - all of a sudden, both busy, steemit, steempeak, eSteem, etc stops working - and people are even concerned that their steem funds may be lost.

I haven't studied the software, but it should not be a great problem to change the node behaviour from "shutdown" to "stop producing blocks and ignore all newly produced blocks".

Showing the end-users nice error messages explaining that the network has gone down into a "fail-safe read-only-mode while some technical problem is investigated" is of lesser importance, but that's a job done by busy.org, steemit.com, steempeak.com, eSteem, etc.

·
·
·

i'm with you, thanks, but your middle paragraph is imho not that simple or even sensible, because imho if a catastrophic error occurs (which did) NO activity should be allowed, ALL user activity should cease hence shutdown

·
·
·
·

In my opinion, no matter how catastrophic the error is, anyone ought to be allowed to inspect the historical blockchain, that includes browsing old articles and replies on sites like steemit.com, as well as checking the wallet. This is important for user confidence, and also for search engines, etc.

As no new blocks are produced, it will of course be impossible with any actions that are to be recorded in the blockchain, that includes placing orders on the internal market, voting, writing replies or posts, etc. Ideally some alert should be shown even before the reader considers voting on anything, but a generic "an error has occurred" when trying to vote on something should eventually suffice. I believe the latter is roughly how steemit.com works today if an action cannot be recorded in the blockchain for whatever reason; one gets some error message in red text.

·
·
·
·
·

agreed, it will possibly all happen when STEEMIT is declared out of beta and big investors allow for / deliver the needed quality resources - hopefully sooner than later!

I actually think this was a very good thing to have happen.

We have a lot of smart people around here and to read how all came together to offset a potentially devastating event provides a great deal of confidence.

Bugs are a matter of course in the programming world. Having them is not the problem, what is done once they are found is. It was reassuring to see the team get the chain going in a rather quick period of time. Down for a number of hours seems like a lot but think about how long Microsoft leaves bugs operating.

The fact that many, including myself, were lost without STEEM shows how committed we are as a community. This is something that might be unmatched in this arena at this point.

I am disappointed @abit wasn't mentioned. He was a total rockstar yesterday and without him, things would have taken far longer.

The first thing I did when I could get back on Steemit was to vote for @abit. We need more witnesses that can show up and make a difference.

·

Good point you make - I heard in various chats how awesome @abit was - I will probably re-vote him as witness. Thought he was not really active anymore.

·

Is that @abit another of your stupid accounts?

·
·
·
·
·

You do a very good job making Steemit smaller and smaller. You can go brag about it to your bosses. :-)

·
·

That will show em...

·
·
·

so you and bernie and marky are 3 of a kind - good to know

and nextgencrypto = helpie + all the other shit accounts LOL

·
·
·
·

oh i see you are a loving person yourself. :) awesome

Until reading this post, and the comments, I didn't realize what a good thing it was for the site to crash. Now I know that crashes like that are healthy, and we should probably have them at least once a week.

·

Well observed, @vantocan! Probably you are a European. We analyze every word along its full meaning and also read the white space in between the lines of sentences. Not a long time ago, the months of delay of the SMT has also been celebrated as a big victory from the heroes of the Steemit Inc., NY. That's fault management as usual in PR, unfortunately not on real intellectual eye level for Europeans. Every PR–scentence from NY sounds like parents are talking to their children. It's a matter of culture.
You don't have to wonder much about, that your intelligence feels a little bit insulted. It is nothing serious at all, just Public Relation!

The specific cause of the problem was a bug in the HF20 code. A constant was changed that should not have been, and this caused the initial fork. Your witnesses did a fantastic job of quickly spotting the suspicious behavior and began reverting to a stable build of the Steem blockchain. This exposed another bug that resided in our fork database logic that was present due to the infrequency that this logic is activated, and which caused the minority fork to halt as well.

Thank you for staying on the top of this & keeping us updated. @SteemitBlog

C7B2E46F-4DD7-467F-923B-D85CEB8F073D.gif

This rapid rate of iteration is one of Steem’s uniquely valuable attributes and is virtually unheard of in the blockchain space, which itself is still in a highly experimental phase of development. The cost of this rapid innovation is incidents like the one that occurred yesterday,

AKA

"We put too much priority on SMTs and now everything else is buggy."

I am so happy to be a part of this wonderful community! :)

I was waiting for some report to find out about the events of yesterday, excellent work.

It's good you got it fixed. So the hardfork date was only a target? You were basically trying to hardfork early?

·

No, the hardfork will be activated on September 25 when a majority of the witness are running the new version of the software, but there was a mistake in the part that was supposed to be compatible with the current version.

Installing the new software at the latest moment and activating it immediately would create more chaos.

Thanks for the update.. many of us non-nerds were just sitting around frustrated ..luckily I was able to reach out to a witness (@themarkymark) who informed me what was going on.. I certainly hope everything is fixed..

·

You can also follow @SteemNetwork on twitter for official updates on the Steem network.

·
·

Will do thanks.

Maintaining the September 25th hardfork date isn't ideal.

For political and marketing reasons it might be...

I recommend delaying it an extra 10 days... and rescheduling it for October 5th.... give witnesses a time to role out a little slower, and more stable... and run additional testing...

Knowing we're trying again in only a week seems a little soon.

·

Only up hill from here

Posted using Partiko Android

Just a small bump on the long road ahead.. Great work!

To listen to the audio version of this article click on the play image.

Brought to you by @tts. If you find it useful please consider upvoting this reply.

Upvoted.

DISCLAIMER: Your post is upvoted based on curation algorithm configured to find good articles e.g. stories, arts, photography, health, etc. This is to reward you (authors) for sharing good content using the Steem platform especially newbies.
If you're a dolphin or whales, and wish not to be included in future selection, please let me know so I can exclude your account. And if you find the upvoted post is inappropriate, FLAG if you must. This will help a better selection of post.

Keep steeming good content.
@Yehey

Posted using https://Steeming.com condenser site.

Kudos to Steemit and everyone involved in this fix! I am proud to be a part of this community 🤗
Definitely Resteemed!!!!👍

It's OK .........,

Kudos to the Steemit team (developers & witnesses) who worked tirelessly to ensure service is restored!

Posted using Partiko Android

You don''t know what you got till it's gone. I don't understand the magic the team of programmers do in the background to keep this thing working. But good job to everyone involved and thanks for the update.

Posted using Partiko Android

You don't lose if you get knocked down; you lose if you stay down.
Muhammad Ali

Best quote, by far...and that's why I show up everyday.

It's also why I will continue to hodl SP; as a direct result of this community which supports each other.

Respectfully,
Rebecca

Thank you.

Thanks for

·

Thanks for your support!

Good work, you guys seem resilient to hop back up from trouble.

And it seems that the community isn't salty but in the contrary more helpful than I thought 😄

Posted using Partiko Android

I must say I felt totally peaceful with this.
Is not the very first time this Blackout happens and Devs & Witnesses always find a solution.

Cheers for the Chain!

...our community has the ability to work together to handle these issues...

Yes. Yesterday was a testament to this. And it's been said before, but really good job communicating about this so clearly :-) Thanks for all the good work keeping steemit and the community alive :-)

please upvote my comment as compensation for the hf20 mishap....

just kidding..

·
·
·

so that's a firm NO on the upvote ?

·
·
·

Sorry, company policy :)

·
·
·
·

can I talk to your manager ?

@ned
can you quit your dancing and take my call ?

Great to see an official update on this and I am glad it was resolved reasonably quickly. As more and more DApps come online (inc SMT's) this sort of down time will become harder for independent business who rely on the blockchain to tolerate.

I do feel that communication to other stakeholders (ie the general public, non top 20 witnesses, DApp holders) could have been much better however. This issues with this was expressed well by @therealwoolf in his post this morning so I won't repeat all his points here however I hope the team gets a chance to read it.

https://steempeak.com/witness-category/@therealwolf/the-day-the-blockchain-stood-still

Okay. I'm glad that the problem was identified and that the blockchain came up as quickly as it could. For me, that was 11 hours. Don't know what it meant to others, but it could have been less.

Realistically, I might have lost two to three STEEM from it being down. No big deal.

Maybe it meant more to others. I don't want to downplay it's significance for anyone else.

I guess the main thing that sits wrong is the inevitability of it all. Yes, the blockchain is highly complex. Yes, an error in coding is bound to happen. Yes, it would take way too long to test everything that could possibly go wrong, even if you could anticipate everything that could go wrong.

But what is it about these incidences (this is the second blockchain stoppage since I've been here—don't know if there were any or not before January 1, 2018), that is meant to instill confidence that they won't happen again? The fact that the community came together and resolved the issue? I'm glad it happened and things weren't prolonged. I'd be much happier if it never happened.

But that apparently can't be. So, when the next unlikely, unique, shouldn't happen anomaly takes place, and then the next, and the next, how often do you think this can go on before you lose the confidence of the users?

And maybe it's not so a big a deal now, with 50,000-60,000 daily users that are largely scraping out earnings through their social media efforts, but what happens when it becomes SMTs and after that, entire industries built on the STEEM blockchain, as in a worldwide digital economy?

As I've commented elsewhere, I hope the inevitability of this, sooner than later, becomes totally avoidable. That the impossible to foresee or catch or prevent now becomes absolutely possible, if not automatic. That means resources and money I suppose, and technology and knowledge that doesn't yet exist. Maybe that will be much later than sooner and we'll just all have to cross our fingers that another stoppage won't happen for a while longer.

It's great we've got people who can troubleshoot and resolve things. I'm glad people can work together. It's got to be frustrating for all of those involved since it can largely be a thankless job when something like this happens. I'm just hoping the commitment is there to avoid the likelihood of an all day (in my case) downtime, and if there are alternatives for allowing people to work, cache it somewhere or whatever the proper terminology might be, and then get it going on the blockchain when it's up and running again would be great. I don't know what the solution is. I just know there needs to be something better than it's impossible or it's inevitable and great work guys when it's restored.

I hope I've been able to communicate what I'm feeling and trying to say. I'm not trying to ruffle feathers or point fingers. I'm just trying to express a point of view in hopes that it will be heard and that someone will understand it. If anything could be done about it now or in the new future, that would be ultimate, but right now I'd settle for someone getting what I'm saying and being okay with the fact that I said it.

·

Imagine somebody just invented the steam engine (see what I did there?) and people are worried it might derail or explode. It's due to arrive at the station at 3pm and at 4pm there's no sign of it.
If you're just a traveler on the platform you're going to worry that any number of terrible things have happened.
This post is the train finally arriving at the station, and the conductor announcing that somebody pulled on the emergency brake, which did what it was designed to do and brought the train to a screeching halt.
Most of the delay was the investigation by the engineers as they made sure they'd thoroughly investigated the cause of the problem, and double checked the timetable to make sure that restarting after the delay wouldn't put the engine on a collision course with any other trains.

·
·

Hey, @mattclarke.

I have no problem with everything that occurred the moment the bug/bugs came to play. I'm good with all that.

I'm more concerned with what happened to cause the emergency brake (ie the bugs). I'm understanding that it's impossible to account for every instance of why the brake might be pulled. Right now there aren't that many trains out there, at least not running the same length of track, nor are that many people waiting for the train, so the inconveniences might not be so bad.

Soon, however, we're hoping for many more people to be at the station waiting for the train to arrive (ie velocity), and we're anticipating that there will be many more trains providing service on that stretch of track (ie SMTs, entire industries). We're also understanding there will be other track built and trains on them (competitors) that will take away customers if we can't provide on time service all the time.

The one that's going to be able to get the most people from and to their destinations on time is going to be the one which stands.

·

As we said in the post, we have gained a lot of knowledge from this incident and know a lot of tools we want to develop to ensure this won't happen again. While the process might be unpleasant for you, and for that we apologize, this process of dealing with the unknowns and developing solutions is precisely how problems are solved for good. Building blockchains is still a new thing. Building blockchains that power real applications is something only we do. And building such a blockchain and updating it frequently is yet another thing only we do. We have no one to look to for guidance, no established model for anything. If someone tells you they can develop bug free code in such an environment, they're just a liar. What we can promise is that we will never release code that puts your important information at risk, and we have demonstrated that over and over again. This event was a consequence of that commitment as the blockchain stopped producing blocks specifically to safeguard what matters most. Those who come after us will be able to benefit from our solutions, but we do not have the same privilege. I am sorry we live in a world where we have to be the one's blazing the trail on this, but that's also why we are so committed to what we do. We've been doing this for over 2 years and our engineers and witnesses still respond at all hours and put in countless hours to fix the problems that arise. If you don't think that it is commitment, I don't know what is. And yes, for the people who do the real heavy lifting (i.e. not me) it is totally thankless.

·
·

Hey, @andrarchy.

I'm glad to hear all of this. It is all pioneering on your part, and the analogy you gave me the last time of building the airplane as it's flying has stuck with me. It does describe what you're doing, and attempting to do. Something that is basically impossible to do (at least building a commercial jet in flight would be).

I'm glad to hear about the dedication and expertise employed after the fact. I've seen it in action. I know it exists. I'm glad to hear about the tools that you all want to develop to prevent such things from happening.

I guess since it's nearly ten years since Bitcoin rolled out I figured there might be a little more on blockchain technology out there, along with project management, product testing, etc. I thought there would be more to it as far as common practices go. If not specifically for the blockchain, at least adapting from other industries.

I know the STEEM blockchain is unique in many aspects from other blockchains that aren't designed to do what it does and will do. I can see the uncharted territory there, but I was expecting there would be something else to draw on, rather than it basically being put it together as you go. That's more than just building the blockchain. That's also needing to build all the procedures and protocols surrounding how it gets built. I figured there would be a lot of that in place by now.

I've never heard of anyone saying they can create bug free code. Developers are pretty united on that front, even ones who think they can do it better. So, wasn't trying to say that at all.

·
·
·

Of course we use established project management techniques.

Bravo team Steem! You guys are awesome!

I appreciated the robot graphic on the main page, which let me know this problem was recognized, and being worked on. Thank you for all your hard work guys!

·

Thanks for the feedback! Good to know!

The scariest part is not knowing what is happening while everything is down. Maybe I am in the minority, but I am not well connected with many of the higher ups in the know. So I have to assume the worst and cross my fingers that everybody's accounts are safe.

I am glad it is back, and the glitches are being ironed out.

Hopefully fixing the fork database will prevent this kind of downage again. It is concerning that witnesses testing the new fork can cause the main blockchain to hit a stop error.

Having some communication representatives available from official sources would also be appreciated in the future. Rumors start flaring up like wildfire because most of us have no direct communication lines, and that is bad for value of Steem on the exchanges. I heard there was a twitter post, and that was it. Maybe this can be improved too. I'm sure some of the witnesses would be happy to be trusted with spreading official news through popular discord channels, and on their app sites.

·

You can follow @SteemNetwork for live updates. We understand how much you care about Steem and the Steem applications you rely on and are continuously working to develop better communication systems so that you can remain up to date, while still allowing the engineers who have to work together to diagnose the problem and develop a fix to do their jobs.

·
·

Yesterday during the 12hour / 12second issue, this was actually not true. I found it sad that all of the information about the hardfork that I would have considered relevant and helpful was not accessible through normal Steem front-ends. I am not saying its bad to eat your own dogfood, but if there's suddenly no dog-food to eat then its a hungry pack of wolves... or something.

Perhaps it makes sense to consider IPFS'ing important announcements about things like hardforks as a type of insurance against this kind of thing happening in the future.

Anyway, thanks to all the awesome witnesses who got us back online, their devs and the whole steem community. Great to be here.

·

I definitely agree with this, but I will point out that myself and many others were indeed spreading the word through a number of the servers where we're active. We do our best to make sure we can explain as clearly as we can wherever we can, while juggling the background stuff, but you're absolutely right that we can't reach every community or even most people as effectively as official channels.

Having compounding errors is one of those things that testing might miss, no matter how much is done — in this case, almost a month of running testnet, plus two simulated hardforks, and a week or so of quite a few witnesses running the new code on mainnet all managed to sneak by before the perfect storm of nodes and transactions hit to kick this all of. It definitely sucks, and goes to show we can basically never sleep... but I'm bolstered working on a chain designed to stop before it starts double spending or compromising wallets. If there's a community that you're in that you feel needs a liaison, please reach out to me on steem.chat or discord and I'm happy to add it to the places I roll through when I'm updating.

I agree that the communication has improved considerably and thanks to all the new staff who was hired (I’m assuming new people are doing much of the outward facing communication).... we are noticing a difference so keep it up.

Posted using Partiko iOS

Thanks for being so transparent about the issues and working hard to resolve them. The Steemit platform really is unique in the blockchain space and you guys are really pushing it to the next level.

Keep up the great work!

Thanks for being on top of the situation

In my country we say, that in the difficult moments that we can see the true friends. For those of us who believe in the steemit project, we always hope that everything that happens is to make us stronger. Good energy for everyone and for better times to come.
Good vibes.

·

Very well said. As someone who has been invested in this community since the beginning, I have seen how true that is many times now. It's easy to come in here, benefit from the advancements that have been made and the risks that other people have taken, only to jump ship the moment one stops receiving rewards or encounters a temporary setback. There's a reason why they say that friends stick with you through through both thick and thin.

·
·

That is why sometimes when difficult situations happen, you can take the positive aspect. Because we can see the true face of the people. I have little time on this platform, but as I said in several opportunities, since I started, it was like love at first sight and here I am learning and trying to collaborate as I can, besides betting on a long-term investment, while still being clear that to achieve things, you have to do it with effort and work there is no other way, because I do not believe in easy money, to obtain profits from something you always have to work hard. I was pleased to read your response, sincere thanks for your time.
Good vibes @andrarchy

Thank you to all involved. Awaiting the 20.x fix.

Is there such a thing as a Steem status webpage?

Realmente en los 7 meses que tengo dentro de la plataforma , me parece que es una plataforma estable, casi no he frecuentado caídas como la de ayer, pero fue muy rápida su resolución.

Much appreciated 'team', keep up the good work!

Where is my hardfork...? I know you've got yours...

nygma_comment.jpg

Any idea why Steem auto doesn't work anymore?

·

You would have to join their discord for updates. https://discord.gg/jpN3RwF
They may run their own RPC node and it may be down currently or replaying blocks. I'm not sure the setup. You would have to ask the owner of steemauto. He is often active in Steem.chat as well.

·
·

Who is the owner of Steem auto?

Apology well accepted, we all appreciate your good work. Please keep it up

Thank you for keeping us informed. This is what team leaders do! Thank you for coming out in front of some rumors that hurt the platform. Similarly, I do not know if you know that after yesterday there have been some pernicious phishing that seek to destabilize the platform and some accounts. Greetings

·

We are aware of the situation, but thanks for checking

So we had a unexpected hard fork, damn no wonder everything crashed. At least the stop mechanism ensured that errors were resolved.
Thank you for explaining the reason for the delay and issues yesterday it was hard to figure out when Steemit itself is down when explaining to a friend DAPPs :p

Thx for the update and I am so glad Steem is up and running again, looking forward to HF 20 :)

I was actually expecting to see the HF20 being completed already. I saw some earlier comments on Steemit,Inc never delivering on time. So delivering too early could have been awesome.

I'd be great if witnesses had their own alert system so that they can be woken up from their beds and get into action fast. The cost of decentralization is that everybody lives at a different time zone.

Good Luck With Everything!

·

Most of us do have alert systems, and we were woken up and got to work right away as soon as the issue was detected.

·
·

Most of us do, and
We were woken up and got
To work right away.

                 - timcliff


I'm a bot. I detect haiku.

·

Actually executing a hardfork early is the one thing you don't want to do early! I get what you're saying, and in most cases you would be right, but Hardforks are scheduled in at the blockchain level. The blockchain team released the Hardfork code last week, which you could consider "early," but inside that code was the specification that it would occur on the 25th. The only way to change that would be to go back and change the hardfork code, which would only give the witnesses less time to review the code and exchanges less time to fire up their own version of steemd 0.20.0. This very incident proves why it's so important not to rush hardforks and the firm date ensures that everyone can switch over at the same time.

·
·

Thanks for clarifying.

·
·

Just out of curiosity, what is undergoing the HF exactly - Steem, the Steem blockchain, or Steemit? Many simply do not understand the gist of it all. If the crypto Steem were HF'ing, wouldn't that mean more crypto (like BTC to BCH)?

·
·
·

All of that information is discussed in recent @steemitblog posts. The short version is that the Steem blockchain is hardforking, but thanks to Steem's governance mechanism, when Steem hardforks it is more like an "upgrade" and so no "sister-chains" like BTC or BCH are created. Steem moves to a new fork and the old fork is abandoned. Cryptocurrencies themselves do not fork, only blockchains fork. New "sister-currencies" are only created when a blockchain forks, but the old fork of the chain continues. So BTC exists because someone created a new fork of Bitcoin while the old fork also continued to run. Since there were 2 blockchains they needed two different cryptocurrencies. Since only one fork of Steem will continue after the hardfork, the currency of that blockchain will remain STEEM. This hardfork contains no changes to the STEEM currency.

·
·
·
·

Thanks for such a concise explanation. I know I learned something today!

Thanks for your development
In case of any downtime on the steem, you should add some information about that downtime to the steemit.com
IT IS IMPORTANT

Thanks for being on top of the situation

Thank you for this update.

Posted using Partiko iOS

It was disconcerting to think that my blog was removed and no longer existed ... there needs to be a general broadcast page that says “ Steemit is experiencing Momentary Network Problems and we are working to fix them” ... I think I had a small heart attack thinking that I did something wrong and my page as deleted ... just my opinion.

Posted using Partiko iOS

Good to have an explanation, so thanks for that!

Also a nice thing that we now at least have the "tinkering robot" as the default page when there are issues causing the site to be offline.

As a non-nerd, I want to point to the support of the large number of Discord communities that have formed all by themselves, aside from any planned "communities" by Steemit, Inc. They enable "regular folks" to get some information — often from the technically inclined — in "plain English." And that speaks highly of the cohesiveness of the core of the Steemit community.

I did not come across even a single "I've had it!" rage-quitting type post/comment there.

=^..^=

·

Absolutely.

Thanks for taking care of all this. The community is amazing.

Thank you for this information and professional presentation. This lends credibility to the platform when you have a clear an thorough explanation to the recent events that all steem members are concerned about. Thank you.

howdy there steemitblog! Thanks so much for this information and explanation of what went down and why, greatly appreciated!

Congratulations @steemitblog!
Your post was mentioned in the Steemit Hit Parade in the following category:

  • Comments - Ranked 5 with 95 comments

Amazing quote at the end. Well done team!

Posted using Partiko Android

Thanks for the update. @steemitblog is doing a good job these days of keeping us informed of what is happening around the place.

I think the confidence of those on the platform regarding the direction of Steem is certainly growing. This should spread to those not yet invested.

Many thanks to all the witnesses who worked tirelessly to run steemit once again :)

Could we can face similar issue on implementation of HF20 velocity on 25 September.

·

The bug that occurred has now been patched, so it is now impossible for the same thing to happen.

Great job Steemit team and all the witnesses!!!

I know I poke my nose where it don’t belong but consider my interest to get the correct information my way of intercepting any accusations that may be thrown around, I aim to defuse and reinstall trust nothing else!

Good to hear that you guys have the safeguards in place to ensure the safety of everyone’s accounts. Going forward could we not install a system that automatically returns back to the last stable version in a situation like this?

Posted using Partiko iOS

·

It's not quite as simple as that, but effectively those are the types of solutions we're working on.

·
·

Perfect, yes I air on the side of simplicity... how it functions is a little over my head but I understand the need for a system for these situations and it’s good to hear that your on it!

Posted using Partiko iOS

No one said it was going to be easy and well done to all those involved in fixing the problem. Good luck with HardFork 20. Stephen

Eagerly waiting for HF20

So happy I can enjoy the idiot spam comments of @badcontent on my posts again! Good Job!

Well played :)

I am not a developer, I don't understand about some specific details yet of blockchain, but I find this post very interesting. I'd also like to share something with you, guys, that was happening to me last week. I am not sure if your test for the new hardfork was being run the last week, but this is what happened to me:

I know that curation rewards only reflects in our wallets after redeeming or Transfer To Balance. My partner has access to my account in his Smart Phone, but last week he didn't redeem any of my curation rewards, and what I saw is that each of my curation rewards were being added to my SP before and after redeeming. It was like: I have 245,.03. The next time I see the "Transfer To Balance" button I have 245.004 before redeeming, and after redeeming 245.005.

I don't know if you have another case like this. I am almost sure about this fact. I didn't know you were running a test. I don't know if it could have happened to any amount of people.

Anyway, it's not a bad thing for me, but I had the chance to see that happening. I don't know if it represents an important fact.

Thanks for keeping us informed, @steemitblog.

One thing that should be improved is the comunication.

Steemit.com only showed tge message "ups something went wrong" , ¿on my system, the frobt end ,the blockchain? It took me a while to figure out what was going on

The final Muhammad Ali’s quote it’s like the cherry 🍒 on the cake: really nice and appropriated!

Thanks guys for all your kind effort! 🙏🏻

A huge hug from @amico! 🤗

Posted using Partiko iOS