devand make sure you’re up to date with the latest changes.
git checkout -b parser-nameyet that’s up to you.
It is probably easier to use the docker-compose stack (and benefit from the hot-reload capbility for uWSGI). Set up your environment to use the debug environment, such as:
$ docker/setEnv.sh debug
Please have a look at DOCKER.md for more details.
You’d want to build your docker images locally, and eventually pass in your local user’s
uid to be able to write to the image (handy for database migration files). Assuming your user’s
$ docker-compose build --build-arg uid=1000
|Empty file for class initialization|
|The meat. This is where you write your actual parser|
|Sample files containing meaningful data for unit tests. The minimal set.|
$ pip install cookiecutter
Then generate your scanner parser from the root of django-DefectDojo:
$ cookiecutter https://github.com/DefectDojo/cookiecutter-scanner-parser
Read more on the template configuration variables.
Parsers may have many fields, out of which many of them may be optional.
Always make sure you include checks to avoid potential
KeyError errors (e.g. field does not exist), for those fields you are not absolutely certain will always be in file that will get uploaded. These translate to 500 error, and do not look good.
Each parser must have unit tests, at least to test for 0 vuln, 1 vuln and many vulns. You can take a look at how other parsers have them for starters. The more quality tests, the better.
To test your unit tests locally, you first need to grant some rights. Get your MySQL root password from the docker-compose logs, login as root and issue the following commands:
MYSQL> grant all privileges on test_defectdojo.* to defectdojo@'%'; MYSQL> flush privileges;
This local command will launch the unit test for your new parser
$ docker-compose exec uwsgi bash -c 'python manage.py test dojo.unittests.tools.<your_unittest_py_file>.<main_class_name> -v2'
Example for the blackduck hub parser:
$ docker-compose exec uwsgi bash -c 'python manage.py test dojo.unittests.tools.test_blackduck_csv_parser.TestBlackduckHubParser -v2'
$ docker-compose exec uwsgi bash -c 'python manage.py test dojo.unittests -v2'
In the event where you’d have to change the model, e.g. to increase a database column size to accomodate a longer string of data to be saved
Change what you need in
Create a new migration file in dojo/db_migrations by running and including as part of your PR
$ docker-compose exec uwsgi bash -c 'python manage.py makemigrations -v2'
If you want to be able to accept a new type of file for your parser, take a look at
dojo/forms.py around line 436 (at the time of this writing) or locate the 2 places (for import and re-import) where you find the string
Formats currently accepted: .xml, .csv, .nessus, .json, .html, .js, .zip.
Of course, nothing prevents you from having more files than the
parser.py file. It’s python :-)
If you want to take a look at previous parsers that are now part of DefectDojo, take a look at https://github.com/DefectDojo/django-DefectDojo/pulls?q=is%3Apr+label%3A%22import+scans%22+
The DefectDojo official documentation lives in the docs folder, https://github.com/DefectDojo/django-DefectDojo/tree/dev/docs Please update
docs/content/en/integrations/import.md with the details of your new parser.