net/netdev: correct the ioctl command validation logic

[zhhyu7]

Summary

This PR fixes the validation order in netdev ioctl handlers to ensure correct error codes are returned according to POSIX standards.

Problem

Currently, the ioctl handlers for Bluetooth, IEEE 802.15.4, and packet radio devices check the argument pointer (arg != 0ul) before validating the command type. This causes incorrect error codes to be returned:

• When an invalid command is passed with NULL argument, it returns -EINVAL instead of -ENOTTY
• The validation logic is backwards: command type should be checked first

According to POSIX standards:

• -ENOTTY: The specified request does not apply to the kind of object that the file descriptor references
• -EINVAL: The request or arg parameter is not valid

Solution

This patch reorders the validation logic in three ioctl handlers:

1. netdev_bluetooth_ioctl()
2. netdev_iee802154_ioctl()
3. netdev_pktradio_ioctl()

Changes made:

• Check command validity (_*IOCVALID(cmd)) first
• Then check argument pointer validity (arg != 0ul)
• Return -ENOTTY for invalid commands (not supported by this device type)
• Return -EINVAL for invalid arguments (NULL pointer when data is required)
• Update comments to reflect the correct validation order

Affected Functions

// Before:
if (arg != 0ul) {
    if (_BLUETOOTHIOCVALID(cmd)) {
        // process command
    } else {
        return -ENOTTY;  // Wrong: should be returned for invalid cmd first
    }
}

// After:
if (_BLUETOOTHIOCVALID(cmd)) {
    if (arg != 0ul) {
        // process command
    } else {
        return -EINVAL;  // Correct: invalid argument
    }
}
// Returns -ENOTTY for invalid cmd (set at initialization)

Impact

Correctness: Positive - Returns correct POSIX-compliant error codes.

Compatibility: Positive - Applications relying on proper error codes will work correctly.

Stability: Neutral - No change in stability, only error code correctness.

Performance: Neutral - Same number of checks, just reordered.

Code Quality: Positive - Improves POSIX compliance and code clarity.

Testing

Test Configuration
Host OS: Ubuntu 22.04 x86_64
Target: sim:matter (x86_64 simulator)
CONFIG_WIRELESS_BLUETOOTH=y, CONFIG_WIRELESS_IEEE802154=y, CONFIG_WIRELESS_PKTRADIO=y
Test Cases
Test 1: Invalid Bluetooth command with valid argument
Expected: Returns -1 with errno = ENOTTY
Result: PASS - Correct error code returned

Test 2: Valid Bluetooth command with NULL argument
Expected: Returns -1 with errno = EINVAL
Result: PASS - Correct error code returned

Test 3: Valid command with valid argument
Expected: Command processed normally (returns 0 or valid result)
Result: PASS - Normal operation unchanged

Test 4: IEEE 802.15.4 and Packet Radio validation
Similar tests performed for IEEE 802.15.4 and packet radio ioctl handlers with identical results.

Test Logs

nsh> test_ioctl_validation
Testing Bluetooth ioctl validation:
  Invalid command with valid arg: errno=25 (ENOTTY)
  Valid command with NULL arg:   errno=22 (EINVAL)
  Valid command with valid arg:  ret=0

Testing IEEE 802.15.4 ioctl validation:
  Invalid command with valid arg: errno=25 (ENOTTY)
  Valid command with NULL arg:   errno=22 (EINVAL)
  Valid command with valid arg:  ret=0

Testing Packet Radio ioctl validation:
  Invalid command with valid arg: errno=25 (ENOTTY)
  Valid command with NULL arg:   errno=22 (EINVAL)
  Valid command with valid arg:  ret=0

All tests passed!

Verification Checklist
Code compiles without warnings
Invalid commands return -ENOTTY (not supported)
NULL arguments return -EINVAL (invalid argument)
Valid commands with valid arguments work normally
No regression in existing functionality
POSIX compliance improved
Same behavior across all three affected handlers (Bluetooth, IEEE 802.15.4, Packet Radio)
Comments updated to reflect correct logic