DOSBox is for DOS programs only.
64-bit versions of windows can still run win32 programs.

Yes, I know. My program is an ms-dos program. Run in win32 cmd. but not in win64 cmd - because uses a dos-extender. Problem his - need to call OpenSSL. In win32 no problem, in win64 inside DOSBox - no luck.
Exist any alternative to OpenSSL that works inside DOSBox?

It's not a DOS program, it's a windows console based program. There is a difference.
The OpenSSL.exe itself is telling you that it's a windows program when you try to run it under DOSBox. Just run it directly from windows CMD instead.

Maybe this is something for you ? http://www.columbia.edu/~em36/ntvdmx64.html

Your best bet is to use the win64 native version, if possible.

But if you insist on using dosbox to launch win32 program, you can try HX. It can launch *some* win32 console apps (and even some GUI apps):
https://github.com/Baron-von-Riedesel/HX

There are some small test programs included, best to check with them first.

OpenSSL is open source but distributed without binaries. But there could also be a DJGPP port out there, somewhere...