|
Hi,
I've installed the Training module for Yolov5 6.2.
I'm using the GUI to attempt to create a dataset and to train de model.
However any action from the GUI seems to be failing (create dataset, dataset info, cancel, train, resume).
Any clue on what I can do to troubleshoot or get it fixed ?
modified 4-Aug-23 8:20am.
|
|
|
|
|
Can you first try Ctrl+F5 to force a refresh of the UI and to ensure you have the latest code running.
Have you entered dataset names, classes to train on? When you hit create dataset, does the UI change at all? More info will help us help you.
cheers
Chris Maunder
|
|
|
|
|
Hi,
Yes I did all of that.
18:52:38:Client request 'create_dataset' in queue 'trainingyolov5_queue' (...c408e4)
18:52:45:Client request 'status' in queue 'trainingyolov5_queue' (...d6181f)
18:52:48:Client request 'status' in queue 'trainingyolov5_queue' (...ec7217)
18:52:53:Client request 'status' in queue 'trainingyolov5_queue' (...ae9630)
18:52:58:Client request 'status' in queue 'trainingyolov5_queue' (...ba6706)
|
|
|
|
|
You need to use Dog not dog. To see all the Classes click the !
|
|
|
|
|
It actually doesn't make a difference in the result...
|
|
|
|
|
You might have your firewall blocking the images from downloading. Do you see the below folder, the images do take sometime to download
|
|
|
|
|
I had to manually create the folder as they were not created.
But even having done that and changing the permissions doesn't help...
I've attempted to reinstall the module and I receive the below errors:
root@8600315528ae:/app/modules/TrainingYoloV5# bash ../../setup.sh
Hi Docker! We will disable shared python installs for downloaded modules
No schemas installed
(No schemas means: we can't detect if you're in light or dark mode)
Setting up CodeProject.AI Development Environment
======================================================================
CodeProject.AI Installer
======================================================================
Checking GPU support
CUDA Present...No
Allowing GPU Support: Yes
Allowing CUDA Support: Yes
General CodeProject.AI setup
Creating Directories...Done
Installing module TrainingYoloV5
Python 3.8 is already installed
Virtual Environment already present
Checking for Python 3.8...Found Python 3.8.17. present
--2023-08-04 21:16:21-- http:
Resolving security.ubuntu.com (security.ubuntu.com)... 91.189.91.39, 185.125.190.36, 185.125.190.39, ...
Connecting to security.ubuntu.com (security.ubuntu.com)|91.189.91.39|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1321244 (1.3M) [application/x-debian-package]
Saving to: 'libssl1.1_1.1.1f-1ubuntu2.19_amd64.deb'
libssl1.1_1.1.1f-1ubuntu2.19_amd64 100%[================================================================>] 1.26M 1.42MB/s in 0.9s
2023-08-04 21:16:22 (1.42 MB/s) - 'libssl1.1_1.1.1f-1ubuntu2.19_amd64.deb' saved [1321244/1321244]
(Reading database ... 28426 files and directories currently installed.)
Preparing to unpack libssl1.1_1.1.1f-1ubuntu2.19_amd64.deb ...
Unpacking libssl1.1:amd64 (1.1.1f-1ubuntu2.19) over (1.1.1f-1ubuntu2.19) ...
Setting up libssl1.1:amd64 (1.1.1f-1ubuntu2.19) ...
Processing triggers for libc-bin (2.35-0ubuntu3.1) ...
Hit:1 http:
Hit:2 http:
Hit:3 https:
Hit:4 https:
Hit:5 http:
Hit:6 http:
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package libssl1.1.1
E: Couldn't find any package by glob 'libssl1.1.1'
ln: failed to create symbolic link '/usr/lib/libcrypto.so.1.1': File exists
ln: failed to create symbolic link '/usr/lib/libssl.so.1.1': File exists
Checking for CUDA...Not found
Ensuring PIP is installed and up to date...
Looking in links: /tmp/tmp_epp7oc0
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (59.6.0)
Requirement already satisfied: pip in /usr/local/lib/python3.8/dist-packages (23.2.1)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Requirement already satisfied: pip in /usr/local/lib/python3.8/dist-packages (23.2.1)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Installing setuptools...Done
Choosing packages from requirements.txt
Installing Packages into Virtual Environment...ERROR: Exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
status = run_func(*args)
File "/usr/local/lib/python3.8/dist-packages/pip/_internal/cli/req_command.py", line 248, in wrapper
return func(self, options, args)
File "/usr/local/lib/python3.8/dist-packages/pip/_internal/commands/install.py", line 377, in run
requirement_set = resolver.resolve(
File "/usr/local/lib/python3.8/dist-packages/pip/_internal/resolution/resolvelib/resolver.py", line 92, in resolve
result = self._result = resolver.resolve(
File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
state = resolution.resolve(requirements, max_rounds=max_rounds)
File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/resolvelib/resolvers.py", line 397, in resolve
self._add_to_criteria(self.state.criteria, r, parent=None)
File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/resolvelib/resolvers.py", line 173, in _add_to_criteria
if not criterion.candidates:
File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/resolvelib/structs.py", line 156, in __bool__
return bool(self._sequence)
File "/usr/local/lib/python3.8/dist-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in __bool__
return any(self)
File "/usr/local/lib/python3.8/dist-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in <genexpr>
return (c for c in iterator if id(c) not in self._incompatible_ids)
File "/usr/local/lib/python3.8/dist-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 47, in _iter_built
candidate = func()
File "/usr/local/lib/python3.8/dist-packages/pip/_internal/resolution/resolvelib/factory.py", line 206, in _make_candidate_from_link
self._link_candidate_cache[link] = LinkCandidate(
File "/usr/local/lib/python3.8/dist-packages/pip/_internal/resolution/resolvelib/candidates.py", line 293, in __init__
super().__init__(
File "/usr/local/lib/python3.8/dist-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
self.dist = self._prepare()
File "/usr/local/lib/python3.8/dist-packages/pip/_internal/resolution/resolvelib/candidates.py", line 225, in _prepare
dist = self._prepare_distribution()
File "/usr/local/lib/python3.8/dist-packages/pip/_internal/resolution/resolvelib/candidates.py", line 304, in _prepare_distribution
return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
File "/usr/local/lib/python3.8/dist-packages/pip/_internal/operations/prepare.py", line 538, in prepare_linked_requirement
return self._prepare_linked_requirement(req, parallel_builds)
File "/usr/local/lib/python3.8/dist-packages/pip/_internal/operations/prepare.py", line 609, in _prepare_linked_requirement
local_file = unpack_url(
File "/usr/local/lib/python3.8/dist-packages/pip/_internal/operations/prepare.py", line 166, in unpack_url
file = get_http_url(
File "/usr/local/lib/python3.8/dist-packages/pip/_internal/operations/prepare.py", line 107, in get_http_url
from_path, content_type = download(link, temp_dir.path)
File "/usr/local/lib/python3.8/dist-packages/pip/_internal/network/download.py", line 134, in __call__
resp = _http_get_download(self._session, link)
File "/usr/local/lib/python3.8/dist-packages/pip/_internal/network/download.py", line 117, in _http_get_download
resp = session.get(target_url, headers=HEADERS, stream=True)
File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/requests/sessions.py", line 602, in get
return self.request("GET", url, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pip/_internal/network/session.py", line 519, in request
return super().request(method, url, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/cachecontrol/adapter.py", line 48, in send
cached_response = self.controller.cached_request(request)
File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/cachecontrol/controller.py", line 155, in cached_request
resp = self.serializer.loads(request, cache_data, body_file)
File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/cachecontrol/serialize.py", line 95, in loads
return getattr(self, "_loads_v{}".format(ver))(request, data, body_file)
File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/cachecontrol/serialize.py", line 186, in _loads_v4
cached = msgpack.loads(data, raw=False)
File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/msgpack/fallback.py", line 125, in unpackb
ret = unpacker._unpack()
File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/msgpack/fallback.py", line 590, in _unpack
ret[key] = self._unpack(EX_CONSTRUCT)
File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/msgpack/fallback.py", line 590, in _unpack
ret[key] = self._unpack(EX_CONSTRUCT)
File "/usr/local/lib/python3.8/dist-packages/pip/_vendor/msgpack/fallback.py", line 603, in _unpack
return bytes(obj)
MemoryError
Success
Checking for CUDA...Not found
Ensuring PIP is installed and up to date...
Looking in links: /tmp/tmp6m_9ctwh
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (59.6.0)
Requirement already satisfied: pip in /usr/local/lib/python3.8/dist-packages (23.2.1)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Requirement already satisfied: pip in /usr/local/lib/python3.8/dist-packages (23.2.1)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Installing setuptools...Done
Choosing packages from requirements.txt
Installing Packages into Virtual Environment...ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
botocore 1.31.19 requires urllib3<1.27,>=1.25.4, but you have urllib3 2.0.4 which is incompatible.
google-auth 2.22.0 requires urllib3<2.0, but you have urllib3 2.0.4 which is incompatible.
Success
Downloading models-yolo5-pt.zip to /app/downloads/TrainingYoloV5
Downloading Standard YOLO models... already exists...Expanding...Done.
sending incremental file list
sent 100 bytes received 12 bytes 224.00 bytes/sec
total size is 156,824,238 speedup is 1,400,216.41
Module setup complete
|
|
|
|
|
Thanks for that. The issue is the fiftyone package, and we're ditching it because it's truly painful.
Fiftyone downloads a multi-GB CSV then reads that into a mongodb database. They include a very old version of mongodb that uses a very, very old version of libssl. We try and install that during install of the module, but it seems that install is failing for you.
We'll update the docker images next week so libssl1.1.1 is pre-installed and good to go. However, we're also working on a replacement of 51 that will massively reduce the working memory set needed to acquire data and train models.
cheers
Chris Maunder
|
|
|
|
|
I have updated to the latest version of SenseAI and most recent version of the Training module (v1.3).
I'm still unable to create datasets and having training dataset...
Not many logs when it comes down to creating the dataset, so not much evidence I can provide for now.
|
|
|
|
|
At the top right of thee dashboard are you seeing a green "online" message?
cheers
Chris Maunder
|
|
|
|
|
We've finally released our 2.1.10, which was going to be 2.2 but we couldn't wait so slipped this one into the release stream.
The main highlights:
- We've included a training module that allows you to train your own YOLO object detection models with just a couple of clicks. We grab images from Google's OpenImages based on the objects you want to detect, build a dataset for training, then you can build multiple models based off this training set. Just a few clicks and a little patience and you've got your custom models.
- Thanks to Mike Lud, and his boundless generosity and trust, we've added support for the Orange Pi, and included his Object Detection module that uses the Orange Pi's Rockchip NPU.
- Triggers! If you've ever wanted to run a script when an event occurs you can now do that via a triggers.json file. It's rough, it's beta, but it's pretty simple:
{
"Queue" : "objectdetection_queue",
"PredictionsCollectionName" : "predictions",
"PropertyName" : "label",
"PropertyValue" : "car",
"PropertyComparison" : "equals",
"Confidence" : 0.5,
"ConfidenceComparison" : "greaterthan",
"PlatformTasks" : {
"Windows" : { "Command": "cmd", "Args": "/c echo Hi Windows. I see a car", "Type": "Command" },
"Linux" : { "Command": "bash", "Args": "echo Hi Linux. I see a car", "Type": "Command" },
"LinuxArm64" : { "Command": "bash", "Args": "echo Hi Linux. I see a car", "Type": "Command" },
"macOS" : { "Command": "zsh", "Args": "echo Hi Linux. I see a car", "Type": "Command" },
}
}
We need to write this up, but it should be self-explanatory. When a module returns an AI inference result it will return a json object with properties. Specify a queue to watch, and the property name and trigger value. If you specify PredictionsCollectionName the it looks at each item in that collection and checks the property/value. The check is done using the PropertyComparison . equals, notequals, greaterthan, greaterthanorequal etc. If the inference also returns a confidence, then you can specify the confidence value to test and, again, the comparison type. If there's a match then the task you specify in the Platform Tasks is executed depending on the current platform.
- bug fixes, perf improvements, better system info, and a tweaked (but backwards compatible) modulesettings schema
cheers
Chris Maunder
|
|
|
|
|
Chris,
I just did a clean install of v2.1.10 and all of the modules show the version as null and none of the modules will install. Also the training module is not in the module list
|
|
|
|
|
Can you do a Ctrl+F5? Here's what I see
The training module should now be on the list.
cheers
Chris Maunder
|
|
|
|
|
CTRL-F5 has resolved the issue here.
Thanks Chris
|
|
|
|
|
I just loaded the training module and created a person dataset. Dataset Info shows the creation status as not completed but is is completed.
Next I will try training it.
|
|
|
|
|
Hi Chris - just to let you know that the interpreter issue I reported before didn't get fixed in this version. I installed 2.1.10 this morning but alas it died after a short while.
|
|
|
|
|
Sorry, can you please remind me of which issue this is? My brain is starting to leak
cheers
Chris Maunder
|
|
|
|
|
Chris,
Since the clean install had issues with installing modules I did a clean install of v2.1.9 then upgraded to v2.1.10 and the issues with installing modules is fixed except no training module.
Also there is an issue with the Object Detection (YOLOv5 .NET) module count not updating.
modified 4-Aug-23 8:17am.
|
|
|
|
|
Thanks - we'll get that sorted today.
Update: that was a dumb double negative bug. Unfortunately it's a server fix. We may just push out a new build of the current version to save the pain of forcing everyone to upgrade again.
cheers
Chris Maunder
modified 4-Aug-23 9:18am.
|
|
|
|
|
Hi Chris
Nice job on the YOLOTraining module...saves a lot of arse using VS Code and mucking around when you just want to run out a standard trained model.
I've had a couple of runs and it seems to work well, except there's one or two non critical errors logged, eg this one:
If you ever get around to allowing custom image sets and labels created/downloaded from places such as RoboFlow...that would be absolutely awesome as it would save having to use Google Colab workbooks.
Also, just a small RQ..unless I missed it there's no "download of the trained model best.pt" link on the GUI. It's no hardship to got get it from the directory using the OS explorer but if you're ever bored one day wondering how to improve the module user experience ...
|
|
|
|
|
I just tested using a dataset from RoboFlow.
To get the dataset to work you need download the dataset in YOLOv5 format and save the dataset to "C:\Program Files\CodeProject\AI\modules\TrainingYoloV5\dataset" then do some changes to the data.yaml, fist rename it to dataset.yaml. Below is the original yaml and modified yaml
Original data.yaml
train: ../train/images
val: ../valid/images
test: ../test/images
nc: 1
names: ['qr_code']
roboflow:
workspace: project
project: qrcodes-7h389
version: 1
license: CC BY 4.0
url: https://universe.roboflow.com/project/qrcodes-7h389/dataset/1
Modified dataset.yaml
path: C:\Program Files\CodeProject\AI\modules\TrainingYoloV5\datasets\qrcodes
train: ../train/images
val: ../valid/images
test: ../test/images
nc: 1
names:
- qr_code
roboflow:
workspace: project
project: qrcodes-7h389
version: 1
license: CC BY 4.0
url: https://universe.roboflow.com/project/qrcodes-7h389/dataset/1
|
|
|
|
|
Nice!
I shall give this a go. Without a doubt this is probably the most useful new addition I can think of.
|
|
|
|
|
We've updated the module but didn't bump the version since it's just a UI thing that doesn't affect the outcome. You can re-install to get rid of the message, but ensure you go to [app root]/downloads/modules and delete the TrainingYolov5 package before you re-install to ensure no caching.
The download link. Yes, that's missing but we have to do some shuffling to get that to work.
cheers
Chris Maunder
|
|
|
|
|
Chris,
I started playing with the training module I have a couple of suggestions.
- For --bath-size allow -1 to run autobatch also increase the max size to 256, I typically train using a batch size of 128 to 256
- Add --workers argument, I typically train using 16 to 24 workers
- Add --patience argument, I typically train using 5% to 10 % of the numbers of epochs
parser.add_argument('--batch-size', type=int, default=16, help='total batch size for all GPUs, -1 for autobatch')
parser.add_argument('--workers', type=int, default=8, help='max dataloader workers (per RANK in DDP mode)')
parser.add_argument('--patience', type=int, default=100, help='EarlyStopping patience (epochs without improvement)')
|
|
|
|
|
Done. Will be available next release
cheers
Chris Maunder
|
|
|
|
|