You can't use USB debugging while the device is in the Samsung DeX station because plugging/unplugging the device will remove it from Samsung DeX mode. Instead, you can debug your app using Ethernet or Wi-Fi.
To debug your Samsung DeX app over Ethernet or Wi-Fi, follow Androids guide for configuring ADB bridge.
The easiest way to test if your app works properly in Samsung DeX is to use an Android Virtual emulator that has settings that match the Samsung DeX environment.
Run AVD Manager.
Click Create Virtual Device.
Select Nexus 6P.
Click >Nougat Download.
Click >Finish when the download is complete.
Select Nougat and click Next
Click the Arrow icon
The emulator will load.
Now the app can run and check whether the layout runs well in phone mode.
To do this, run the Command Prompt and enter the following commands:
Enter : adb shell
Enter : su
Enter : setenforce 0
Enter : settings put global enable_freeform_support 1
Enter : cd /data/local/tmp
Enter : mkdir permissions
Enter : cd permissions
Enter : cp -a /system/etc/permissions/* ./
Enter : sed -e "s/live_wallpaper/freeform_window_management/" android.software.live_wallpaper.xml > freeform.xml
Enter : mount --bind . /system/etc/permissions
Wait few seconds
Enter : stop
Enter : start
After rebooting, check if the icon below appears when in viewed in the "Recent screen" menu. The app will switch into Freeform Multi-Window when you click the icon below.
Enter the following commands in Command Prompt to test the app in the same density and resolution as Samsung DeX Desktop Mode:adb shell, wm density 160, wm size 1080x1920.
Select the following icon on Emulator control panel.
Test if the app runs well in the same density and resolution as Samsung DeX Desktop mode.
If you want to return to phone mode, enter the following commands in Command Prompt: adb shell, wm density reset, wm size reset. Click on the "rotate to portrait" icon on the Emulator Control Panel.