Are you using DOSBox 0.74-3 or a build of current SVN source? If the former, then please try the latter.

Thank you for your reply. I am using 0.74-2 actually so will try the latest one. The SVN one if I can find it.
EDIT: In 0.74-3 the problem persists.

Not sure if thats what you meant, but I tried this SVN DOSBox-X v0.84.0 (2022/05/31) and it does solve the mouse problem, thank you!
I would prefer to stay with the official version, hopefully the fix will find itw way into 0.74-4 when its out.

No, that is an unofficial fork -- it does not represent SVN, and it does not indicate if the issue is resolved in official source.
You can get an SVN build here: https://yesterplay.net/dosboxece/
Use the most recent non-ECE build, which as of today is: "DOSBox r4477.7z"

Nice, this build does solve the issue too. Alas for some reason it completely broke my mouse routines (and its just the basic stuff of INT 0x33). The cursor now goes haywire in weird directions.
Was there a change of the built-in mouse driver? It seems to either jump off the screen (the default sensitivity 100) or slowly change direction after moving the mouse like a meter on my table (sensitivity 5). And does not work at all when the mouse is locked.
It does work much better in the unofficial build with sensitivity 100 so I dont want to bother you unnecessarily. Unless you want to solve the issue, then I can of course test and report.

It would help if you maybe could share the code that you wrote? I doubt that DOSBox is so broken that int 0x33 doesn't work anymore, as that is how 99% of the software will use the mouse...

Of course, I can share the code. Its just that the code works compiled and both on a real pc and virtual pc even in the IDE. And also in the unofficial build of DOSBox.

1#include <stdio.h>
2#include "mouse.h"
3
4typedef struct {
5	int x, y;
6	unsigned char lmb, rmb, mmb;
7} MOUSE;
8
9MOUSE Mouse;
10
11int MOUSE_Detect(void)
12{
13	asm {
14		mov ax, 0x00
15		int 0x33
16	}
17
18	Mouse.lmb = 0;
19	Mouse.rmb = 0;
20	Mouse.mmb = 0;
21	Mouse.x = 0;
22	Mouse.y = 0;
23
24	if (_AX == 0) return(0);
25	else return(1);
26}
27
28void MOUSE_Update(void)
29{
30	asm {
31		mov ax, 0x03
32		int 0x33
33		mov [Mouse.x], cx
34		mov [Mouse.y], dx
35		mov [Mouse.lmb], bx
36		and [Mouse.lmb], 0x01
37		mov [Mouse.rmb], bx
38		and [Mouse.rmb], 0x02
39		mov [Mouse.mmb], bx
40		and [Mouse.mmb], 0x04
41	}
42}
43
44void MOUSE_SetPosition(int x, int y)
45{
46	asm {
47		mov ax, 0x04
48		mov cx, [x]
49		mov dx, [y]
50		int 0x33
51	}
52}
53
54void MOUSE_SetRange(int x1, int y1, int x2, int y2)
55{
56	asm {
57		mov ax, 0x07
58		mov cx, [x1]
59		mov dx, [x2]
60		int 0x33

61
62		mov ax, 0x08
63		mov cx, [y1]
64		mov dx, [y2]
65		int 0x33
66	}
67}
68

Then I simply draw a bitmap on the mouse coordinates with

1VESA_DrawImage640(Mouse.x, Mouse.y, image, 20, 20);

not even erasing it afterwards so I can see how it travels.

Its not easy to explain. The code is the same. I tried about the same mouse moves in official DOSBox and then in the unofficial build (the second image):
https://imgur.com/a/mfW9enO
https://imgur.com/a/olgxS9g

Both have mouse sensitivity 100.

The erratic mouse movement seems more like an issue with SDL than DOSBox, so instead of the SDL.dll included with the SVN build you can try substituting the one from 0.74-3.

I tried a version of SDL.dll from 2010 which is about 1/3 smaller than the one that comes with the SVN build and no dice, its the same. Can't test with the one that comes with the unofficial build as the file is not there, its probably linked with the main .exe.

What is your OS?

Can you attach a DOS executable that demonstrates the issue? I figure you can, since you're compiling in BC++.

Sure, I tested the executable now on my home PC with DOSBox 0.74-3 and it works perfectly. It must be something with my computer at work I guess. I will try again on Monday.

Just tested again on my work PC, the movement is no longer jerky, I dont get it. The BX problem is still somewhat active, the program works just for the first time (when executed from the IDE), the following runs it just ends immediatelly (the LMB is still acting as if pressed). But when I have a pause after initializing the mouse and click at that time, it seems to reset the BX register or something and it sometimes works then, for one run again. I guess thats as far as I can get. Will have to see how it behaves on the real DOS machine when I get there in a few weeks.

SVN has a fix to ignore button press and release events that are out of sequence, and a fix for clearing the button press and release counters when the reset function is called, which could relate to what you refer to as "the BX problem", and that is why I suggest using SVN. I do not know if the fixes were adopted and correctly implemented in unofficial forks.

As for the strange coordinate behavior you described, I don't know what might cause that. I have seen instances where certain apps running in the task tray caused odd mouse behavior in DOSBox, so you could try removing anything like that from the equation.

Sorry for teh confusion, the erratic movement was due to using remote desktop on my laptop and running the program on my work PC this way. I know, I know, but when running it in the virtual box the same way it worked okay, so it did not occur to me as the possible cause.

I tried it on my real old PC and it works without a hitch there so it must be the out of sequence clicks you describe.