What type should I take for text that can be any charset? (C++)

Question

0.00/5 (No votes)

See more:

I have methods that currently take const char* and an encoding. Like so:

C++

foo(const char* text,text_encoding::utf8)

That's fine for utf8 and latin1, etc
But what if you want utf16 (wchar_t?) or utf32 (no intrinsic type, but i use int32_t)

What if there are others, like some kind of binary businesses that's utf8? My encoders don't care - they take a void*

Should I take a const void* here? I really hate it. But I also hate the idea of limiting the types it can be via a union because I just don't know what types of encodings people may want to use down the road.

I mean, I could do like typedef void* encoded_text_t; (I don't like that name, but I'd take suggestions).

Does anyone have a better idea?

Keep in mind, the STL is off limits to me. I'm on embedded.

What I have tried:

not really applicable here.

I've put everything in the first box.

Posted 16-Aug-24 7:15am

honey the codewitch

Add a Solution

Comments

PIEBALDconsult 16-Aug-24 14:00pm

No matter what you do, someone will still try to pass you a wombat. :D

honey the codewitch 16-Aug-24 14:03pm

Right? If this was LVGL I'd just take void* and call it a day, because that's how they handle every situation like this, but that's straight C, not C++, and also it squicks me out. I just don't like taking void* for what is essentially text. I could live with typedefing it if there are no other good options, but I need a name. text_t? nah. string_t? too confusing (std::string exists) meh.

2 solutions

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Iids · Answer 1 · 2024-08-17T01:42:00

Solution 1

To store text that can be any charset in C++, you should use std::string.

This data type is designed to handle strings more flexibly compared to the char data type which can only store one character.

Posted 17-Aug-24 1:42am

Iids

Comments

honey the codewitch 17-Aug-24 10:22am

A) Keep in mind, the STL is off limits to me. I'm on embedded.

C) can std::string produce utf32 codepoints? No it cannot

D) Just no

Richard MacCutchan 17-Aug-24 10:24am

OP clearly stated that she cannot use STL.

honey the codewitch 17-Aug-24 11:05am

I've settled on the typedef for now. It's less than ideal. I'm considering a union, but that gets weirder to use than a cast. A void pointer theoretically works, and obviates the need for a cast when passing in the argument - I just hate it.

Richard MacCutchan 17-Aug-24 11:12am

Do you ever get that feeling that just as you reach what you thought was the end of the rabbit hole, there is a bend which leads down into deeper darkness? :(

honey the codewitch 17-Aug-24 11:12am

I'm literally right there now, and I just posted in the lounge about it. LOL

KarstenK 19-Aug-24 3:44am

Be aware that multibyte string are using double space and needing other function. So better stick to my advice or risk a lot of trouble.

honey the codewitch 19-Aug-24 9:10am

I'm not using tchar or wchar_t or C or C++ functions to deal with multibyte characters.

I have encoder classes that take a void* and a length, and they return a series of UTF32 codepoints. Whether it's latin1, utf8, or something else. No wchar needed or wanted.

KarstenK 20-Aug-24 7:24am

stick to my solution because it also handles this problem. Type casting is evil when there isnt any check after it but the machine code. :-/

honey the codewitch 20-Aug-24 7:26am

Absolutely not. Your solution is Microsoft specific. Your solution does not yield UTF32 codepoints. In fact, your solution can barely handle any encodings at all. It's not a good solution.

KarstenK · Answer 2 · 2024-08-17T10:20:00

Solution 2

Depending on your library and compiler you should use some macro for the character identifier. Microsoft is using therefor the

C++

TCHAR

macro which can resolve to char or wchar_t depending on the compiler settings.
That need some change in most string functions like using

C++

_tcscpy

instead of strcpy.

So some string library which can handle it would be very handy.

Posted 17-Aug-24 10:20am

KarstenK

Comments

honey the codewitch 17-Aug-24 16:42pm

oof. I'm trying to keep things C++ wherever possible. I do use the preprocessor, but seldomly. I could use a macro but I don't think that puts me in a better position than a typedef, TBH.

KarstenK 19-Aug-24 3:42am

this is some typedef but it also is some typedef for the API like the strcpy. So it works!!!