Thoughts about the questionnaire and new proposals

2년 전

A few days ago a draft of the questionnaire was released and some users provided some feedback and also some critics. We considered and discussed the points that were made in the comments but it was obvious that some users misunderstood how the scoring system worked. In our opinion when releasing the questionnaire it should have been specified that because of devs requirements, the questionnaires started from a value of 100 and we had to substract points instead of adding, that is the reason why there were no positive questions. Probably that point caused the most outrage and it was just a perception issue.
Others mentioned that there were too many questions regarding the contribution post and that it was a bad idea to start from these questions. Some users asked to be more punitive on average to mediocre translations that contain several errors but reward more work well done and error free.
We came out with two possible questionnaires, the first option (Q1), is something implementable in the current utopian framework. The option (Q2) may not be currently implementable that it's what we feel should be done.

First example questionaire (Q1):
Q1 Google Sheet example link

Second example questionaire (Q2):
Q2 Google Sheet example link

The two examples are in the same file, but on different tabs

Q1

The first option is compatible with the current framework in Utopian: it starts from a maximum score of 100 and it subtracts or adds values based on the bonus / malus obtained from the answers to the questions. Now we know that the final score can go beyond the value of 100 so we added also positive questions.

Limits:

In order to reward quality, we must assign greater weight (score) to the questions regarding the quality of translation. Thus, it is not possible to differentiate in a balanced way the economic remuneration for translations of different volumes. For example, a translation of 2000 words should not reward twice as much as a 1000 words translation. On the other hand, if we assign more points to the volume of the translation, the quality loses weight and a long but average translation could receive more than an excellent but short translation.

Meaning of the questions

1. How do you rate the overall accuracy of the translated text?
2. How many major mistakes were found in the translated text? (that can be change the meaning of the text)
3. How many minor mistakes were found in the translated text? (that do not change the meaning of the text)
4. Has the same style of translation been maintained for this project? If more translators are working on this project, are they respecting the original style?
5. How would you rate the formatting, grammar, writing style and overall presentation of the contribution post?
6. Did the translator provide in the contribution post all the information needed to fully evaluate the translation? (For example, did he specify if he needed to research the definition of unfamiliar terms and which tools he used in the translation?)
7. What was the total volume (words) of the translated text in this contribution (excluding duplicate phrases and non-translatable content)?

Question 1: It represents an overall judgment of the translation also taking into account factors not specified in the following questions.

Questions 2 e 3: This is a more specific question, it requires a minimum count to identify the proportion between the errors committed and the length of the processed text, distinguishing the weights in serious and minor errors.

Question 4: It is important that the style is consistent between the different parts translated, such as when you decide not to translate technical or medical words. The consistency of methods and style is also important when different translators work on the same project or on one already started by others. Please note that if you find issues in the previously translated strings you should translate them again avoiding the same mistakes.

Question 5: This question refers to the post presentation. It has less weight than the questions relative to the quality of the translation but it gives more granularity to the final score. Nonetheless, a clumsy post, with several grammatical mistakes, would seriously undermine the professionalism of the translator.

Question 6: Second and last question about the contribution post. Here we are asking if the translator has provided all the necessary information for the correct evaluation of his work, such as: number of translated words, project name, github link, crowdin link, name of translated files (if applicable) and other if necessary.

Question 7: The length of the translation is another important parameter, a standard post is minimum 1000 words. We have introduced bonuses for longer posts and panalities for shorter ones.

Note: The scoring system is a short blanket, it is difficult to give due weight to both quality and length, we tried to mediate.

Q2

Our second proposed questionnaire may not be implementable (it would require some changes in the utopian voting bot). But, we believe it would provide a score that reflects better the quality of the translations and it multiplies the result by a coefficient proportional to the volume of translated words. We would like to highlight two values: the score, which indicates the quality of the translation regardless of its volume and the actual vote that it will receive, which considers also to the amount of translated text. This allows us to discriminate between quality represented by the score and to tune the reward taking into consideration the volume of text translated.
Why is this important? To avoid that short but excellent translations are skipped by utopian voting bot only because there were longer translations in the cue.

Please:

When providing feedback/criticism it would be helpful if you could also provide constructive proposals instead of just saying "I don't like it". We will have a chance to discuss this on Monday in the utopian discord server.



If you would like to delegate to Davinci.Witness you can do so by clicking on the following links:
50 SP, 100SP, 250SP, 500SP, 1000SP, 5000SP.
Be sure to leave at least 50SP undelegated on your account.

votaXdavinci.witness.jpgCC0 Image - Creative Commons, thanks to @mrazura for ITASTEM logo.
CLICK HERE AND VOTE FOR DAVINCI.WITNESS
DAVINCI WITNESS TRAIL
If you are the owner of an Open Source project on Crowind platform and you want to translate it with our help and the support of @utopian-io please join our discord server and chat with us.

DAVINCI WITNESS DISCORD SERVER
Authors get paid when people like you upvote their post.
If you enjoyed what you read here, create your account today and start earning FREE STEEM!
STEEMKR.COM IS SPONSORED BY
ADVERTISEMENT
Sort Order:  trending

After going through all the comments on elear's post, and reading about the questionnaire issue, I think finding a consensus is impossible since everyone has different standards and point of views, there are strict people who emphasize a lot on the quality and accuracy of the contribution and I tend to be one of them, and there are other people who tend to have a minimalistic approach, others are more permissive …. On the other hand, we have the standards that utopian wants to implement. There are just many ideas, and many contradictions going on here, there is also a huge lack of information about what utopian wants and what people want because from what I saw, many people don’t know that utopian requested many things before about the presentation, word count, granularity and so forth.

I think people need to understand that there is no perfect solution to this, and at the need of the day, a decision will have to be made, not because it’s the right and best solution, but just because it’s the fairest possible solution. I believe that this is not the end, and we can keep working for a better model in the future, to make it even more accurate and professional. But the sure thing is that if we want to take a step farther in this, more resources will be needed, we will also need developers and people well versed in mathematics to do the proper equations for scaling … for now, we just need to work with what we have and can control, but as I already said, I am pretty sure that a better solution(new project) will be set in the future, a solution that equals the professional existing tools that are available online for companies.

Thank DaVinci, for this quick update! We’re definitely moving forward.

Q1

  1. Did the translator provide in the contribution post all the information needed to fully evaluate the translation? (For example, did he specify if he needed to research the definition of unfamiliar terms and which tools he used in the translation?)

Is this question really necessary? Are there really translators who consistently don’t put these info in their post and are they so many that we need to put this in the questionnaire that will be used for anyone henceforth.
Personally, I would consider a post without these info to be unreviwable and I’d have the translator edit it before even moving on to the translation itself.

Q2

D1 How would you rate the accuracy of the translated text?
D2 How would you rate the legibility of the translated text?

Do we really need both these questions? I would assume that an accurate translation is also a translation that is legible. A perfect word-for-word translation that doesn’t flow at all in the target language is not an accurate translation.

D6 On a 10 points scale how would you rate the difficulty of translating the text in this contribution(with 10 being the highest)?

I still stand by the previous suggestion to assign standard difficulty levels to all projects, rather than leave this to an LM’s perception. Especially on a 1-to-10 scale. This will end up punishing translators with stricter LMs even more than the current system does.

D7 How would you rate the internationalization efforts shown by the translator while translating this project?

I don’t understand this question at all.


I still stand by the suggestions I made in my post from yesterday. A good set of questions, for me, would be as follows:

  • Does this post present the work done in a personal, engaging, or anyway outstanding format? Yes / No
  • How would you rate the grammar, syntax, and overall linguistic level of this post? Good / Poor
  • What was the volume of the translation outlined in this post (excluding duplicate strings and non translatable words)? Whatever breakdown is feasible as desirable
  • What project was this post about? List of all available projects
  • Was the translation outlined in this post significantly more difficult than the rest of the project? Yes / No
  • How do you rate the overall accuracy of the translated text? Excellent / Very good / Good / Poor
  • How many major mistakes were found in the translated text? (that can be change the meaning of the text) Scale of 1 to 5 every 1000 words
  • How many minor mistakes were found in the translated text? (that do not change the meaning of the text) Scale of 1 to 10 every 1000 words

Personally, I agree with some previous comments made (I believe by @scienceangel on @elear’s post) that translations which would be graded lower than Poor in quality or with more than 5 major mistakes or 10 minor mistakes every thousand word should receive a zero score and no upvote at all. I don’t think the questionnaire needs to take those cases into consideration.
Particularly because if the translation ends up being so terrible, it’s the LM who ends up doing the real translation work, with the translator still getting a bigger upvote.

·

Regarding the accuracy and legibility, each word can have several meanings so it's not enough to add one of the meanings in the translation but it also has to make sense in the context of the text, hence the question about legibility. Having these two questions increase the granularity of the questionnaire. About the difficulty question, we think the questionnaire will have to be accompanied by a document or guideline. There we could make a table with all the suggested difficulties for each project. Anyway now that question has much less impact than before. The question D7 considers the feedback from one of the asian teams, for them is not always straight forward to translate words that are common for us.

·
·

each word can have several meanings so it's not enough to add one of the meanings in the translation but it also has to make sense in the context of the text, hence the question about legibility

That’s my point, though. If it doesn’t make sense in context, the translation is not accurate. I can’t imagine a case where accuracy and legibility wouldn’t go together because I consider legibility as part of accuracy, where translation is concerned. Although, of course, the opposite might be true: a translation could be 100% wrong but written in impeccably legible form. But if it’s a bad translation it shouldn’t be scored, regardless of legibility.

In our opinion when releasing the questionnaire it should have been specified that because of devs requirements, the questionnaires started from a value of 100 and we had to substract points instead of adding, that is the reason why there were no positive questions.

Whoever told you this is the case is misinformed. The score does start as 100, but you can have both positive and negative values for the answers. For example, for the question "How would you rate the overall value of this contribution on the open source community and ecosystem?" in the development category we have the values 5, 0, -12 and -20 for each answer respectively. Other categories also have this, so you could do the same if you want to.

·

You are right, we did not know that the score could go beyond 100 so that it was possible to add positive points.

·
·

As it is rare for everything to be perfect, a score beyond 100 will probably never happen.

Hey, @davinci.witness!

Thank you for your contribution to the crowd. We are the Steemit project dedicated to empowering The Wisdom of Crowds. You can find more about us on our official blog or whitepaper and you can support us by voting for our witness and joining our curation trail on Steemauto . We are also inviting you to join Crowdmind Discord server. Don't forget to use the #crowdmind hashtag and happy crowdsourcing!

Aditional comment: Besides the fact that you are already a great project with core crowdsourcing values implemented in your work, your decision to source the questionary from your community was absolutely brilliant. It will not only help you keep the high level of engagement, but also produce better final solutions because you decided to tap the wisdom of the crowd. As from our beginning, crowdmind supports you!

Best regards
@hidden84

My suggestion is to have a "silver bullet", that will mark that the post must be upvoted, not matter the score.

Let me explain:

Imagine that there is a project with 1.152 words

It gets - 0 points

Another translator is working on a project with 10.000 words, and it's easy to split it in parts with 2.000 words

It gets - 14 points

The first translation, although with the lower score...
Is one more finished project. It's translated well and it deserves the upvote, no matter the score.

·

I am glad you brought it up, that's exactly why we made a second proposal where the volume does not affect the score but it's used as a coefficient to change the voting weight. I am just not sure if it could be implemented. There are also manual votes that can be requested in special cases.

Utopian and DaVinci

Thanks @davinci.witness for your awesome work with the Utopian translations category!

Contribute to Open Source with utopian.io

Learn how to contribute on our website and join the new open source economy.

Want to chat? Join the Utopian Community on Discord https://discord.gg/h52nFrV