Drafted by Alfred, modified on 06/07/203
Apply for a Palmetto account on CCIT using this link.
Once you get the account, contact CCIT again via email to let the administrator give you permission for accessing /project/twei2 folder.
The /project/twei2 is a folder shared by Nextlab. Hence, your personal home folder should be created inside /project/twei2/home. Please name it with your username. For easier access, you could create a soft link inside your real home folder (/home/$USER) using the command:
mkdir -p /project/twei2/home/$USER
ln -s /project/twei2/home/$USER /home/$USER/workspace
If you want to use miniconda, run
/project/twei2/miniconda3/bin/conda init
Do not delete anything else inside the /project/twei2 folder unless you are the author or creator of that file.
Currently, the Palmetto server has some issues with exporting "*.xo" file. Therefore, we have to build Vitis HLS project locally and copy them into the Palmetto Server. A demo project is saved in /project/twei2/SharedFolder/Vitis_multi_binary.
The project has a structure as follows:
.
├── build_oct.pbs
├── figures
│ ├── Connection.png
│ ├── extra.png
│ ├── Head_schematic.png
│ ├── Node_schematic.png
│ ├── schematic.png
│ └── Tail_schematic.png
├── head_ip_2b.ini
├── head_ip.ini
├── host_src
│ ├── ALICE29.txt
│ ├── fileops.cpp
│ ├── fileops.h
│ ├── head.cpp
│ ├── networking
│ │ ├── Makefile
│ │ ├── oct_fpga.hpp
│ │ └── udp_setup.cpp
│ ├── node.cpp
│ └── tail.cpp
├── kernel_src
│ ├── Base_IPs
│ │ ├── cmac_0.xo
│ │ ├── cmac_1.xo
│ │ ├── networklayer.xo
│ │ └── synthesis_results_HBM
│ ├── bit_container_0
│ │ ├── auto_data_pack.xo
│ │ ├── linker.cfg
│ │ ├── Makefile
│ │ ├── packet_switch_rx.xo
│ │ ├── packet_switch_tx.xo
│ │ ├── rxkrnl.xo
│ │ ├── txkrnl.xo
│ │ ├── xcd.log
│ │ └── xrc.log
│ ├── bit_container_1
│ │ ├── auto_data_pack.xo
│ │ ├── linker.cfg
│ │ ├── Makefile
│ │ ├── packet_p2s.xo
│ │ ├── packet_s2p.xo
│ │ ├── packet_switch_rx.xo
│ │ ├── packet_switch_tx.xo
│ │ ├── xcd.log
│ │ └── xrc.log
│ ├── bit_container_2
│ │ ├── auto_data_pack.xo
│ │ ├── linker.cfg
│ │ ├── Makefile
│ │ ├── packet_switch_rx.xo
│ │ ├── packet_switch_tx.xo
│ │ ├── xcd.log
│ │ └── xrc.log
│ ├── post_sys_link.tcl
│ └── templates
│ ├── connectivity_if0.ini
│ ├── connectivity_if1.ini
│ └── connectivity_if3.ini
├── Makefile
├── node_ip_2b.ini
├── node_ip.ini
├── README.md
├── sync_2b.sh
├── sync.sh
├── tail_ip_2b.ini
├── tail_ip.ini
└── vitis_setup.sh
This is a project for building multiple bit-containers simultaneously for OCT FPGA projects (Ref). In this project, three bit-containers are generated, and their IP package file (*.xo) and the corresponding linker file (linker.cfg) are saved in kernel_src/bit_container_* folders. If some common IP packages are required, save them into the Base_IPs folder and refer to them inside each Makefile. The Makefile automatically reads the *.xo files in the folder and builds the bit-container. However, if the IP is not used, remove it from the folder or it will lead to a linking error.
Before building the project, copy the folder to your own workspace with:
cp -r /project/twei2/SharedFolder/Vitis_multi_binary /project/twei2/home/$USER
Then in the main Makefile (the immediate one inside the project folder), rename the project name with anyone you like (the default is demo).
The Palmetto server does not allow you to do any heavy work on the login node. You have to connect to a working node using qsub command. A default job description file build_oct.pbs is created inside the project folder. To build, simply run:
qsub build_oct.pbs
The content of the build_oct.pbs is shown below:
1 #PBS -N build_oct
2 #PBS -l select=1:ncpus=16:mem=128gb,walltime=23:00:00
3
4 cd /project/twei2/home/$USER/Vitis_multi_binary
5
6 export XILINXD_LICENSE_FILE=28017@license6.clemson.edu
7
8 make link TARGET=hw -j8 | tee build_log.log
9
10 exit 0
The first two rows are not comments. They follow the rules of pbs files (Ref). The first row names the job as build_oct; the second row specifies the resources required for the job. Here, it asks for a server machine with a minimum of 16 vcpus and 128GB RAM. The resource requirements automatically evict some low-end servers. The walltime is set as 23 hours, which means the job will be terminated after 23 hours no matter if it is finished or not. Make sure that you give it enough time to finish the task. Typically, building 3 bit-containers takes around 12 hours on the server.
The 4th row goes to the project folder, '$USER' will be evaluated with your username automatically and you don't have to modify it; the 6th row sets up the license server location for Vivado to the bitstream generation when GT transceivers are used. If the project doesn't use Transceivers, it can be removed. The 8th row runs the Makefile with the makefile target as 'link', which means linking the project and generating the bitstream. The 'TARGET=hw' means the bitstream is for real hardware, not for any emulation. '-j8' specifies 8 threads are allowed (the maximum number suggested is one-half of the vcpus you applied for). '|' is the pipe symbol for passing the output to the next shell command, which is a tee command which simply copies the output to the 'build_log.log' file for debugging if there were any error happens.
A queued task will be created and the job ID will be printed after executing the command. You can use the following command to check the status of your job:
qstat -xf <job_id>
Since a Vitis project can take ~50GB of space, it is unwise to save the project file inside the /project/twei2 folder as we pay for the space. Therefore, in the main Makefile, you can see that all temp files, which are mainly Vivado routing temporary files and checkpoints are saved in /scratch/$(USER)/$(PRJ_NAME) folder. The scratch folder is large but can only hold your file for 7 days. If you want to keep the file, copy them to your own home folder (/home/$USER, rather than /project/twei2/home/$USER). All report and log files, however, are saved in the build folder.