A couple of other notes - this may only be important for setup:
It looks like, at least for now, only 2 HDDs are recognized by the machine. This may be because the 39320 isn't actually supported by 98SE and the drives are just passing through with Int 13h.

One of the things I like in this setup is that I can quickly just prep an image and sling it to the machine as a whole SCSI disk. It's making the driver setup very easy in machines that have no other peripherals. I don't need to find a CDROM, floppy, any of that crap. If it needs something before network is up, it's a reboot away.

JodieC wrote:

Adaptec PSCSI cards

When you say "PSCSI" do you mean "parallel SCSI"? I don't think that "PSCSI" is a conventional abbreviation for that, is it?

I found an interesting discussion here about someone having issues with FreeBSD target mode, and this actually included a lot of helpful info for getting started
https://forums.freebsd.org/threads/help-with- … -example.56828/

I found that a few weeks ago myself, but unfortunately have not had the success that you have! Anyway thanks for your post, it's good to know that this does work in FreeBSD 12.0 for someone so that I don't have to go back and try some of the older versions that I had seen success reports with.

options AHC_TMODE_ENABLE=0x127

I think you mixed up hex and decimal there, if you want it to be 127 shouldn't it be 0x7F?

I found your post in your "Ca 2000 PC - Albatross - PIII 1GHz, GF2, iSCSI XP" thread where you said:

JodieC wrote:

I reworked the FreeBSD PSCSI emulation using the CTL tools in FreeBSD.
The storage block device is emulated via PSCSI through the Adaptec 39160s on each end.

This seems much, much, much more robust and reliable than the scsi_target userland example. Doesn't hurt when it's decent for speed.

That sounds good! For now since I know nothing about CTL tools I'm happy to just have one command to run, but I assume CTL would mean the ability to maybe emulate a SCSI CD-ROM drive too.

Edit: Was it difficult? Are you planning to publish and/or contribute a patch for it?

I'm using an Adaptec 2940 and when my initiator machine (which I think has an Adaptec 2920) is POSTing and gets to the SCSI BIOS initialisation, I get a kernel crash due to a page fault with lots of spam in the 'dmesg' output in the core.txt file:

1ahc0: Someone reset channel A
2(targ0:ahc0:0:1:0): targdone 0x9747c00
3ahc0: Someone reset channel A
4(targ0:ahc0:0:1:0): ahc0: Someone reset channel A
5targread
6(targ0:ahc0:0:1:0): targread ccb 0x9747c00 (0x2827a460)
7(targ0:ahc0:0:1:0): targreturnccb 0x9747c00
8cam_debug: targfreeccb descr 0x9736e20 and
9cam_debug: freeing ccb 0x9747c00
10ahc0: Someone reset channel A
11ahc0: Someone reset channel A
12(targ0:ahc0:0:1:0): write - uio_resid 4
13(targ0:ahc0:0:1:0): ahc0: Someone reset channel A
14Sending inline ccb 0x10 (0xffbfe658)
15(targ0:ahc0:0:1:0): getccb 0x1cc6ef00
16(targ0:ahc0:0:1:0): ahc0: Someone reset channel A
17Changing abort for 0x28270000 to 0x1c4fd000
18(targ0:ahc0:0:1:0): sendccb 0x1cc6ef00
19(targ0:ahc0:0:1:0): ahc0: Someone reset channel A
20targreturnccb 0x1cc6ef00

etc.

I gather that with improper termination you can get signals reflecting, and I might not have gotten that correct, so maybe this is causing some unrealistically huge number of resets or other commands to be received. The Adaptec 2920 I'm using as an initiator doesn't even have a hotkey to enter its BIOS so I can't check on the termination settings there, nor ask it to scan the bus as in your screenshot 🙁

doshea wrote:

JodieC wrote:

Adaptec PSCSI cards

When you say "PSCSI" do you mean "parallel SCSI"? I don't think that "PSCSI" is a conventional abbreviation for that, is it?

Yes just parallel SCSI. I'm probably just making up the PSCSI acronym. I since found that LIO uses the term pscsi as pass-through SCSI so there's likely to be an overrun in acronyms.

I think you mixed up hex and decimal there, if you want it to be 127 shouldn't it be 0x7F?

Right. I fixed it later but I haven't updated my post.

That sounds good! For now since I know nothing about CTL tools I'm happy to just have one command to run, but I assume CTL would mean the ability to maybe emulate a SCSI CD-ROM drive too.

Edit: Was it difficult? Are you planning to publish and/or contribute a patch for it?

It's all off the shelf stuff, CTL is much easier and can load easily at boot. I do plan to publish my ctl.conf file shortly. I'm running into a bug where my Win98SE box just constantly resets the bus and CTL is unhappy about that. I started over with a fresh FreeBSD 12.0 box, recompiled kernel and I'm starting to install 98SE today.

I'm using an Adaptec 2940 and when my initiator machine (which I think has an Adaptec 2920) is POSTing and gets to the SCSI BIOS […]
Show full quote

I'm using an Adaptec 2940 and when my initiator machine (which I think has an Adaptec 2920) is POSTing and gets to the SCSI BIOS initialisation, I get a kernel crash due to a page fault with lots of spam in the 'dmesg' output in the core.txt file:

Copy code to clipboard

1ahc0: Someone reset channel A
2(targ0:ahc0:0:1:0): targdone 0x9747c00
3ahc0: Someone reset channel A
4(targ0:ahc0:0:1:0): ahc0: Someone reset channel A
5targread
6(targ0:ahc0:0:1:0): targread ccb 0x9747c00 (0x2827a460)
7(targ0:ahc0:0:1:0): targreturnccb 0x9747c00
8cam_debug: targfreeccb descr 0x9736e20 and
9cam_debug: freeing ccb 0x9747c00
10ahc0: Someone reset channel A
11ahc0: Someone reset channel A
12(targ0:ahc0:0:1:0): write - uio_resid 4
13(targ0:ahc0:0:1:0): ahc0: Someone reset channel A
14Sending inline ccb 0x10 (0xffbfe658)
15(targ0:ahc0:0:1:0): getccb 0x1cc6ef00
16(targ0:ahc0:0:1:0): ahc0: Someone reset channel A
17Changing abort for 0x28270000 to 0x1c4fd000
18(targ0:ahc0:0:1:0): sendccb 0x1cc6ef00
19(targ0:ahc0:0:1:0): ahc0: Someone reset channel A
20targreturnccb 0x1cc6ef00

etc.

I gather that with improper termination you can get signals reflecting, and I might not have gotten that correct, so maybe this is causing some unrealistically huge number of resets or other commands to be received. The Adaptec 2920 I'm using as an initiator doesn't even have a hotkey to enter its BIOS so I can't check on the termination settings there, nor ask it to scan the bus as in your screenshot 🙁

I would bring up your FreeBSD 12 box, not run the scsi_target, and see if you get reset spam on the bus in dmesg when you boot the other machine. The other lines about the ccbs are fine. You won't need them when it's up and running, you can recompile without CAMDEBUG turned on.

Just writing up the CTL stuff

My ctl.conf

1target eui.943d07ce77ecc404 {
2	port ahc0
3	port ahc1
4	lun 0{
5		path /root/floppyimg
6	}
7	lun 1{
8		path /dev/da1
9	}
10	lun 2{
11		path /dev/cd0
12	}
13}

The target EUI doesn't matter, I made it with `openssl rand -hex 8` . CTL requires an EUI, IQN or NAA number here though.

1sysrc ctld_enable=YES

Compile the kernel with the targ drivers (mine looks like this)

1include GENERIC
2ident SCSIKERNEL
3
4device scbus
5device da
6device targ
7device targbh

Here's what my CTL command outputs look like on boot:

1root@bsd3:~ # ctladm portlist -v
2Port Online Frontend Name     pp vp
30    NO     camsim   camsim   0  0  naa.5000000d40cfc701
4  Target: naa.5000000d40cfc700
5  All LUNs mapped
6      port_type=8
71    YES    ioctl    ioctl    0  0  
8  All LUNs mapped
9      port_type=4
102    YES    tpc      tpc      0  0  
11  All LUNs mapped
12      port_type=8
133    YES    camtgt   ahc0     0  0  
14  LUN 0: 0
15  LUN 1: 1
16  LUN 2: 2
17      port_type=2
184    YES    camtgt   ahc1     0  0  
19  LUN 0: 0
20  LUN 1: 1
21  LUN 2: 2
22      port_type=2
23root@bsd3:~ # ctladm lunlist
24(7:1:0/0): <FREEBSD CTLDISK 0001> Fixed Direct Access SPC-5 SCSI device
25(7:1:1/1): <FREEBSD CTLDISK 0001> Fixed Direct Access SPC-5 SCSI device
26(7:1:2/2): <FREEBSD CTLDISK 0001> Fixed Direct Access SPC-5 SCSI device
27root@bsd3:~ # ctladm devlist -v
28LUN Backend       Size (Blocks)   BS Serial Number    Device ID       
29  0 block                  4096  512 MYSERIAL0000     MYDEVID0000     
30      lun_type=0
31      num_threads=14
32      file=/root/floppyimg
33      ctld_name=eui.943d07ce77ecc404,lun,0
34      scsiname=eui.943d07ce77ecc404,lun,0
35  1 block              31457280  512 MYSERIAL0001     MYDEVID0001     
36      lun_type=0
37      num_threads=14
38      file=/dev/da1
39      ctld_name=eui.943d07ce77ecc404,lun,1
40      scsiname=eui.943d07ce77ecc404,lun,1
41  2 block                     0  512 MYSERIAL0002     MYDEVID0002     
42      lun_type=0
43      num_threads=14
44      file=/dev/cd0
45      ctld_name=eui.943d07ce77ecc404,lun,2
46      scsiname=eui.943d07ce77ecc404,lun,2

I'm trying to boot from floppy img but I may have to just make a 98 system disk on a FAT disk and boot from that. No big deal if I have to, but I thought I would try the lowest intervention way possible first!

JodieC wrote:

I would bring up your FreeBSD 12 box, not run the scsi_target, and see if you get reset spam on the bus in dmesg when you boot the other machine.

I've done that a few times, and just get a single "Someone reset channel A" message each time.

I noticed in /usr/src/sys/conf/NOTES (if I recall the path correctly) that there is a setting to make something (CAM? the target layer?) only respond after X number of milliseconds, so I'm considering trying that to see if maybe it helps me to get an idea of what is going on. I kind of wish I had some kind of SCSI protocol analyser, like Wireshark for parallel SCSI. I assume such things exist(ed) in very limited numbers for those with lots of money!

Thanks for the CTL information! I'll have a read through it later.

I would like to add a small note that I was able to pass through an ISO from my VM to the SCSI bus, as a CDROM, and it was bootable to the client.

Since my SCSI card is set to boot ID 7 lun 0, I removed everything and set the lun 0 to be the cdrom. You have to specify a device type of CDROM (5) when making the CTL lun

1ctladm create -b block -t 5 -o file=/mnt/os/Microsoft/WinXP/WinXPwSP3.iso

With that, my ISO was bootable on the initiator.
I haven't had much luck with 98SE yet because I guess not all 98SE disks are bootable to begin with. Even the one I have that shows bootable in the initiator's SCSI BIOS won't actually boot at boot time. The SCSI BIOS detects the boot sector on the CDROM but then at boot time it says "DISK BOOT FAILURE, INSERT SYSTEM DISK AND PRESS ENTER"

The biggest difference I can see, looking at the ISOs with `isoinfo -d -i $imagename` is that the 98SE disk looks like this:

1    Eltorito defaultboot header:
2        Bootid 88 (bootable)
3        Boot media 2 (1.44MB Floppy)

and the XP disk looks like

1    Eltorito defaultboot header:
2        Bootid 88 (bootable)
3[b]        Boot media 0 (No Emulation Boot)
4[/b]

Going through my other bootable images, my 98SE disk and Red Hat 5.2, 6.1 and 6.2 are the only ones with Boot media 2. sigh.

I know I just lamented my journey in reasons why it wouldn't boot. However, I found that since I had the FDC controller disabled, it would not boot from Boot media 2. Who knew?
It booted from the emulated SCSI CDROM now, however, once it gets to the PCI bus scan after the AIC-7890/7891, it's hanging.
So I have more to dig into.

Very cool.

I do something similar with Fibre-Channel, FreeBSD and ZFS and/or RAM backed volumes (mdconfig). I also have a Fibre-Channel router/bridge that should allow me to use plain old SCSI like your setup.

The reason I use ZFS volumes is snapshots, rollback and clones (it takes less than one second make a clone of a LUN). Kind of like you can do on a VM. (remember to use volmode=dev for zfs volumes)

Messed up? Just rollback to an automatic snapshot 10 minutes ago.

Trying out 10 different chipset drivers? Just make 10 clones after installing Windows but before installing the drivers (only the differences take up space).

I have not yet tried the SCSI bridge part yet. But I do have a machine with Windows 98 that boots directly of a SAN LUN using a PCI-Express Fibre Channel HBA. It does not have a driver for the HBA so it just uses the "INT 13" access that the HBA provides via its BIOS option ROM. Benchmarks are strange and inconsistent (see screenshot). There are however drivers for Windows XP and up, and that works great.

I also use this for my main workstation that normally runs Linux and FreeBSD, but I do have a Windows 7 LUN that I sometimes use to play The Witcher 3. I just shut down, switch the LUN to the Windows 7 one and boot (my network is setup in such a way that there is no network access when running windows).

I also have snapshots so I can rollback to before installing drivers, patches, etc.

I used this setup to do 40 almost fully automated installs of Windows 98 controlled from a shell script. Trying about 40 different combinations of parameters to Windows 98 install setup.exe.
The script just put "setup.exe" with the specified parameters in autoexec.bat in each LUN before booting the machine.

All that was missing was the ability to cut power to the machine from the script. I have since then solved the power control issue with a network controlled APC PDU.
If I had a VGA capture setup (working on it) I could start a script to run an experiment and view the results later as video files. Off-cause the LUN of each variant is snapshotted at the end, so I can just choose any one of them later, and boot that setup. Automating Windows 98 stuff beyond the install is a bit hard, but it is doable. It is also fairly easy to emulate a PS/2 or USB keyboard from something like an Arduino, if the need should arise.

Since I built that setup I have acquired some PCI HBAs that should allow me to do this on older machines. But still no drivers and just "INT 13" access. When I have time to play with the FC/SCSI bridge/router, it should allow me to present the SAN LUNs via. plain old SCSI. Then I can hopefully use normal SCSI controllers with drivers and everything.

It has been a few years since I really played with this on anything older than my main workstation.

Another benefit with this setup is when running FreeBSD or Linux, is that they are not tied to the hardware the same way as Windows.
This allows me to boot a LUN that is normally used by a physical machine as a VM (bhyve). I use this for my GF's workstation that runs Xubuntu. It can be booted as a VM during the night for automatic patching. It is fairly easy to detect from a script during boot if it is running as a VM, and then do any patching needed, and then have it shut it self down after patching. It is off-cause important to make sure the physical machine is not running and also to cut off FC access to the LUN while it is running as a VM. I also use snapshots before and after patching.

mil wrote:

I also have a Fibre-Channel router/bridge that should allow me to use plain old SCSI like your setup.

I had a brief look around for such a product recently and it didn't seem like there's anything available that is relatively cheap. Do you know of any, or does one just need to get a lucky find on eBay or something?

The reason I use ZFS volumes is snapshots, rollback and clones (it takes less than one second make a clone of a LUN). Kind of like you can do on a VM. (remember to use volmode=dev for zfs volumes)

Messed up? Just rollback to an automatic snapshot 10 minutes ago.

Trying out 10 different chipset drivers? Just make 10 clones after installing Windows but before installing the drivers (only the differences take up space).

I've done something similar on Linux with qcow2 disk images, which you can expose as a local block device using qemu-nbd. I haven't gotten to the point of seeing whether I can expose those via FC though.

This all sounds like a nice setup, being able to do snapshots of a physical Windows machine would be great!

doshea wrote:

I noticed in /usr/src/sys/conf/NOTES (if I recall the path correctly) that there is a setting to make something (CAM? the target layer?) only respond after X number of milliseconds, so I'm considering trying that to see if maybe it helps me to get an idea of what is going on.

Unfortunately I was just thinking of this:

1# SCSI_DELAY: The number of MILLISECONDS to freeze the SIM (scsi adapter)
2#             queue after a bus reset, and the number of milliseconds to
3#             freeze the device queue after a bus device reset.  This
4#             can be changed at boot and runtime with the
5#             kern.cam.scsi_delay tunable/sysctl.

and it defaults to 5000ms anyway (verified using sysctl).

Anyway thanks, using CTL was slightly more successful (and also looks more promising in terms of features!). Here is what showed up in /var/log/messages when the initiator machine booted:

kernel: ahc0: Someone reset channel A
syslogd: last message repeated 998 times

I guess 999 resets isn't good for that scsi_target daemon but CTL can handle it.

The SCSI BIOS on the initiator was able to identify that there was a target device there on FreeBSD, but then when I booted into (very old) Linux on the initiator and loaded the kernel module for the SCSI card, after some more of those reset messages, the module reported timeouts and no devices showed up in /proc/scsi/scsi. I think it's time to try some different HBAs on both ends.

Incidentally I get these messages when FreeBSD boots:

kernel: Configuring Target Mode
kernel: (ctl0:ahc0:0:6:0): Lun now enabled for target mode
kernel: (ctl1:ahc0:0👎ffffffff): Lun already enabled
kernel: (ctl1:ahc0:0👎ffffffff): ctlferegister: Enable LUN failed, status 0x3e
kernel: ctlfe_online: CAM error LUN Already Enabled for Target Mode (0x3e) returned from cam_periph_alloc()
kernel: ctlfe_onoffline: ahc0 has no valid WWNN/WWPN
kernel: ctlfe_onoffline: ahc0 (path id 0) failed enable target role: 0x16
kernel: .

I noticed "failed enable target role" first and was concerned, but then it looks like for some reason it's just trying to enable it more than once so maybe this is normal? It obviously works to some extent.

I noticed "failed enable target role" first and was concerned, but then it looks like for some reason it's just trying to enable it more than once so maybe this is normal? It obviously works to some extent.

Mine does that too, it's because of the AHC_TMODE_ENABLE in the kernel config.

I'm not sure why your bus would reset 999 times, that's pretty nuts. I'd definitely get a new initiator.

The huge number of resets seems to be related to the SCSI HBA I'm using for the initiator, since with different ones I only get 1 or 2, but I don't think that they are actually a problem.

I did have some success, and it seems like my problems are related more to what software is running on the initiator than on the HBA itself. Generally the worst things to run are Linux (I tried Red Hat Linux 7.3 or Fedora Core 1) where I either can't see the device or get fatal issues under high load, or FDISK under DOS which also has fatal issues.

On the system I already mentioned with the AHA-2920, as I already said "The SCSI BIOS on the initiator was able to identify that there was a target device there on FreeBSD" but Linux couldn't see anything, but when I boot into DOS the BIOS provides drive letter(s) for the partitions I already had on the disk, and I can also boot from the virtual disk, so there the issue seems to be with the Linux driver (fdomain).

On another machine with some other (mostly Adaptec PCI) HBAs, I could at least mount the virtual drive under Linux and see the filesystem, but if I tried to use 'dd' to copy the entire virtual disk to /dev/null, FreeBSD started dumping out errors and the Linux system would hang. On the other hand if I booted into DOS or Windows 95 or 98 SE, I could see the disk and run ScanDisk and do a full surface test with no problems. The only issue under those non-Linux OSes was if I ran FDISK it tended to cause similar problems with errors and hangs. If I disabled the SCSI BIOS and loaded ASPI drivers under DOS and ran AFDISK I had problems too.

I did manage to get AFDISK working when I used one particular old ISA HBA, but with that one I still had trouble under Linux, so I haven't found any combination that works completely yet.

I tried a few cards for the target too, it didn't seem to make any difference.

One solution I found to not being able to run FDISK was getting ctld to export the same disk via iSCSI, attaching to the disk on my Linux system, then configuring Bochs to emulate the same geometry that I know the HBA does and running FDISK under Bochs. An alternative would be using Linux's fdisk, enabling DOS compatibility mode, and then forcing the geometry using special commands, but I'd rather let DOS pick sizes that it knows will work anyway.

My end goal is to use this for a Macintosh, but when "Drive Setup 1.5" does a drive scan it causes a FreeBSD kernel crash.

Edit: I'd forgotten to try changing the HBA's settings to slow things down. Changing from wide 40MB/s to narrow 5MB/s and disabling disconnection seems to have helped the issues under high load from Linux at least. I'll need to do some more testing.