Unicode CLDR v41 Beta available for testing

The Unicode CLDR v41 Beta is now available for testing. The beta has already been integrated into the development version of ICU. 

The XML data, JSON data, charts, and specification are available for review. These may change if showstopper bugs are found. We would especially appreciate feedback from non-ICU consumers of CLDR data. Feedback can be filed at CLDR Tickets.

The release is scheduled for April 06, 2022.

CLDR v41 is a limited-submission release. Most work was on tooling, with only specified updates to the data, namely Phase 3 of the grammatical units of measurement project. The required grammar data for the Modern coverage level increased, with 40 locales adding an average of 4% new data each. Ukrainian grew the most, by 15.6%. 
The tooling changes  are targeted at the v42 general submission release. They include a number of features and improvements such as progress meter widgets in the Survey Tool. 

Finally, the Basic level has been modified to make it easier to onboard new languages, and easier for implementations to filter locale data based on coverage levels.

The following table shows the number of Languages/Locales in this version. (See the v41 Locale Coverage table for more information.)

Level	Languages	Locales	Notes
Modern	89	361	Suitable for full UI internationalization
Moderate	13	32	Suitable for full “document content” internationalization, such as formats in a spreadsheet.
Basic	22	21	Suitable for locale selection, such as choice of language in mobile phone settings.
Total	124	414	Total of all languages/locales with ≥ Basic coverage.

Beyond the member organizations of the Unicode Consortium, many dedicated communities and individuals regularly contribute to updating their locales, including:

• Modern: Cherokee, Cantonese, Scottish Gaelic,  Sorbian (Lower), Sorbian (Upper)
• Moderate: Asturian [nearly Modern], Breton, Faroese, Fulah (Adlam), Kaingang, Nheengatu, Quechua, Sardinian
• Basic: Bosnian (Cyrillic), Interlingua, Kabuverdianu, Māori, Romansh, Tajik, Tatar, Tongan, Uzbek (Cyrillic), Wolof

For details, see the Unicode CLDR v41 Release Note.
The next version of CLDR, version 42, is slated to start General Submission on May 18, 2022.

Unicode CLDR provides key building blocks for software supporting the world’s languages. CLDR data is used by all major software systems (including all mobile phones) for their software internationalization and localization, adapting software to the conventions of different languages.

---

Over 144,000 characters are available for adoption to help the Unicode Consortium’s work on digitally disadvantaged languages

Update on the Internationalization & Unicode Conference

This is an update on the annual Internationalization & Unicode Conference. As some of you know, Object Management Group (OMG), our events and logistics partner for the annual Internationalization and Unicode Conference (IUC), is moving in a different strategic direction.

We decided to mutually end the partnership and are now in the process of transferring the various resources from OMG to the Unicode Consortium.

ACKNOWLEDGMENTS

We would like to take this opportunity to thank the OMG team, especially Mike Narducci and Carol David, for their support and dedication in making IUC such a mainstay for the global internationalization community.

Unicode would also like to thank the dedicated group of volunteers who worked with Rick McGowan on the program committee. Some of them have been on the committee from the early days even before we began working with OMG in 2006. This speaks to the strong commitment by the individuals as well as the organizations supporting their involvement over the years.

THE WAY FORWARD

While the ending of this partnership creates some challenges, it is also an opportunity to reshape how Unicode approaches community building and training. And given how the meeting and event landscape continues to evolve, it is a great time to explore best practices and apply lessons learned from other meetings and groups.

To that end, Unicode staff and a small group of volunteers convened late last year and will continue meeting in the coming 60-90 days to create the future IUC.

REQUEST

The Unicode Consortium is always looking to improve its conference. We recognize IUC as a key opportunity each year for knowledge-sharing, community building, and evangelization and want your help to shape the future IUC. Please give us your input and ideas by EOD on Friday, March 11th in one of these brief questionnaires.

(1) Survey for previous attendees
(2) Survey for those who have yet to attend

NEXT STEPS

Once we have additional community input and an update on our specific plan, we will share that information with the broader community via this blog and other channels, including on the meeting website at www.unicodeconference.org.

In the meantime, thanks for your time and ideas!

---

Over 144,000 characters are available for adoption to help the Unicode Consortium’s work on digitally disadvantaged languages

Avoiding Source Code Spoofing

Unicode has convened a group of experts in programming languages, tooling, and security to provide guidance and recommendations on how to better handle international text in source code, as well as providing code to help implementations.

Recent reports have highlighted problems in the review of source code containing non-ASCII Unicode characters (the so-called “Trojan Source exploit”). A person reviewing a submission of source code could be fooled into thinking that the code was okay, when it was actually malicious. The basic problem occurs when the actual text is different from what the reader perceives it to be, based on what is displayed. This can result either from the presence of characters used in right-to-left scripts (such as Arabic or Hebrew) that can change the visual ordering of text, or from the presence of characters that look like others (also known as “confusables”).

The problems here are not solely a security issue: text with different writing directions or confusable characters can be hard to work with. Finding a solution here is important from both security and usability points of view. Developers of source code editors or compilers should not be required to have a deep knowledge of Unicode to provide good user experience and robust security mitigations.

Unicode’s mission is to allow everyone to use their own languages on computers and mobile devices. The above issues are part and parcel of a character set that covers all the writing systems of the world – and have been documented in the Unicode Standard since its very first version in 1991. Unicode’s past efforts have focused on misleading URLs and identifiers, and correct visual ordering of plain text. And while much of this material is relevant to source code, this group of experts will now collect, curate, and supplement that early documentation with concrete recommendations to support source code editors and compilers.

While it may seem that it is easiest to simply go back to limiting source code to only ASCII characters, ASCII-only environments make it much harder to write and maintain software that can be used all over the world – a fundamental requirement for modern software. Moreover, this approach disadvantages software developers who use languages other than English.

More details on the source code spoofing issue, the proposed plan, and formation of this group are found in document L2/22-007R2.

---

Over 144,000 characters are available for adoption to help the Unicode Consortium’s work on digitally disadvantaged languages

Unicode CLDR v41 Alpha available for testing

The Unicode CLDR v41 Alpha is now available for testing. The alpha has already been integrated into the development version of ICU. We would especially appreciate feedback from non-ICU consumers of CLDR data. Feedback can be filed at CLDR Tickets.

Alpha means that the main data and charts are available for review, but the specification, JSON data, and other components are not yet ready for review. Some data may change if showstopper bugs are found. The planned schedule is:

• Mar 09 — Beta (data)
• Mar 23 — Beta2 (spec)
• Apr 06 — Release

CLDR v41 is a limited-submission release. Most work was on tooling, with only specified updates to the data, namely Phase 3 of the grammatical units of measurement project. The required grammar data for the Modern coverage level increased, with 40 locales adding an average of 4% new data each. Ukrainian grew the most, by 15.6%.

The tooling changes are targeted at the v42 general submission release. They include a number of features and improvements such as progress meter widgets in the Survey Tool.

Finally, the Basic level has been modified to make it easier to onboard new languages, and easier for implementations to filter locale data based on coverage levels.

The following table shows the number of Languages/Locales in this version. (See the v41 Locale Coverage table for more information.)

Level	Languages	Locales	Notes
Modern	89	361	Suitable for full UI internationalization
Moderate	13	32	Suitable for full “document content” internationalization, such as formats in a spreadsheet.
Basic	22	21	Suitable for locale selection, such as choice of language in mobile phone settings.
Total	124	414	Total of all languages/locales with ≥ Basic coverage.

Beyond the member organizations of the Unicode Consortium, many dedicated communities and individuals regularly contribute to updating their locales, including:

• Modern: Cherokee, Cantonese, Scottish Gaelic, Sorbian (Lower), Sorbian (Upper)
• Moderate: Asturian [nearly Modern], Breton, Faroese, Fulah (Adlam), Kaingang, Nheengatu, Quechua, Sardinian
• Basic: Bosnian (Cyrillic), Interlingua, Kabuverdianu, Māori, Romansh, Tajik, Tatar, Tongan, Uzbek (Cyrillic), Wolof

Unicode CLDR provides key building blocks for software supporting the world’s languages. CLDR data is used by all major software systems (including all mobile phones) for their software internationalization and localization, adapting software to the conventions of different languages.

---

Over 144,000 characters are available for adoption to help the Unicode Consortium’s work on digitally disadvantaged languages

Unicode 15.0 Alpha Review

The repertoire for Unicode 15.0 is now open for early review and comment. During alpha review the repertoire is reasonably mature and stable, but is not yet completely locked down. Discussion regarding whether certain characters should be removed from the repertoire for publication is welcome. Character names and code point assignments are reasonably firm, but suggestions for improvement may still be entertained.

This early review is provided so that reviewers may consider the character repertoire issues prior to the start of beta review (currently scheduled to start in late May, 2022). Once beta review begins, the repertoire, code points, and character names will all be locked down, and no longer be subject to changes.

Feedback for the alpha review should be reported under PRI #442 using the Unicode contact form by April 5, 2022.

---

Over 144,000 characters are available for adoption to help the Unicode Consortium’s work on digitally disadvantaged languages

Enhancements to Unicode Regular Expressions

A new revision of UTS #18, Unicode Regular Expressions is now available.

Regular expressions are a key tool in software development. Back in 2000, few regular expression engines supported Unicode, even at a basic level. UTS #18 set out to raise the bar, describing how regular expression engines could be adapted to deal with Unicode correctly and completely. Since that time, major programming languages and libraries have adopted level 1 features (supporting all Unicode literals, basic character properties, subtraction, intersection, ...), and some also adopted some level 2 features (full character properties, grapheme clusters, ...).

The main focus in this release is on handling the complement of properties of strings. The distinction is drawn between code point complement and full complement, followed by explicitly defining the complement operator [^...] to be code point complement, and providing the reasons for doing so in an annex. The important difference between [A--B] and [A&&[^B]] is outlined — setting out the reasons why the latter is insufficient to represent set difference.

For the EBNF in general, and for character classes with strings in particular, examples were added and the text clarified. A new annex provides examples for how character classes can be parsed.

---

Over 144,000 characters are available for adoption to help the Unicode Consortium’s work on digitally disadvantaged languages

Unicode 14.0 Paperback Available

The Unicode 14.0 core specification is now available in paperback book form with an original cover design by Sophia Tai. This edition consists of a pair of modestly priced print-on-demand volumes containing the complete text of the core specification of Version 14.0 of the Unicode Standard.

Each of the two volumes is a compact 6×9 inch US trade paperback size. The two volumes may be purchased separately or together, although they are intended as a set. Please visit the separate description pages for Volume 1 and Volume 2 to order each volume in the set. The cost for the pair is US $36.72, plus shipping and any applicable taxes.

These volumes do not include the Version 14.0 code charts, nor do they include the Version 14.0 Standard Annexes and Unicode Character Database, which are all freely available on the Unicode website.

Purchase The Unicode Standard, Version 14.0 - Core Specification Volume 1 and Volume 2.

---

Over 144,000 characters are available for adoption to help the Unicode Consortium’s work on digitally disadvantaged languages

The Most Frequently Used Emoji of 2021

The Unicode Emoji Mirror Project

92% of the world’s online population use emoji — but which emoji are we using? The Unicode Consortium, the not-for-profit organization responsible for digitizing the world’s languages, gathers information about how frequently emoji are used. Looking at patterns of usage helps to determine what new emoji should be added to the Unicode Standard. As part of this effort, we are making that data available to the public. 

The new Unicode Emoji Frequency page lists the Unicode v12.0 emoji ranked in order of how frequently they were used in 2021 and what has changed since 2019. Check it out for more analysis, insights and patterns that illustrate our collective experience during a global pandemic.

#UnicodeEmojiMirror

---

Over 144,000 characters are available for adoption to help the Unicode Consortium’s work on digitally disadvantaged languages