Folder scan not providing any results?

thepix

Can anyone shed any light on the scanner produce results on some folders and not others?

Had files in a Library folder to be processed since last night and yet the scanner finds nothing?

Here are the logs from a manual scan trigger:

03-04 13:09:06.072 [INFO] -> Triggering worker: SystemMonitor 2023-03-04 13:09:06.079 [INFO] -> Making request [GET]: http://localhost:5000/api/node/by-address/INTERNAL_NODE 2023-03-04 13:09:06.087 [DBUG] -> Getting directory size (TEMP DIR) 'C:\Users\xxxxxxx\AppData\Roaming\FileFlows\Temp': 0 B 2023-03-04 13:09:06.094 [DBUG] -> Getting directory size (LOGGING DIR) 'C:\Users\xxxxxxx\AppData\Roaming\FileFlows\Logs': 6.39 MB 2023-03-04 13:09:07.782 [INFO] -> Triggering worker: FlowWorker 2023-03-04 13:09:07.786 [INFO] -> Making request [GET]: http://localhost:5000/api/settings/current-config/revision 2023-03-04 13:09:07.812 [INFO] -> No file found to process, status from server: NoFile 2023-03-04 13:09:11.748 [INFO] -> Triggering worker: FlowRunnerMonitor 2023-03-04 13:09:12.343 [DBUG] -> LibraryWorker: Updating Libraries 2023-03-04 13:09:12.346 [INFO] -> WatchedLibrary: Library 'Torrents - Sonarrr' marked for re-scan 2023-03-04 13:09:12.354 [INFO] -> Triggering worker: LibraryWorker 2023-03-04 13:09:12.357 [DBUG] -> LibraryWorker: Library 'Torrents - Sonarrr' calling scan (Scan complete: False) (Library Scan: False (last scanned: 738582.13:09:12.3407501) (Full Scan interval: 60) 2023-03-04 13:09:12.360 [DBUG] -> WatchedLibrary: Scan started on 'Torrents - Sonarrr': D:\Downloads\FileFlows\Torrents\Sonarr 2023-03-04 13:09:12.369 [DBUG] -> LibraryWorker: Library 'Usenet - Sonarr' was scanned recently 00:41:12.0696878 (full scan interval 60 minutes) 2023-03-04 13:09:12.373 [DBUG] -> LibraryWorker: Library 'Usenet - Radarr' was scanned recently 00:46:12.1644538 (full scan interval 60 minutes) 2023-03-04 13:09:12.375 [DBUG] -> LibraryWorker: Library 'Torrents - Radarr' was scanned recently 00:41:12.0565228 (full scan interval 60 minutes) 2023-03-04 13:09:12.379 [DBUG] -> LibraryWorker: Library 'BP' was scanned recently 00:49:12.1825855 (full scan interval 60 minutes) 2023-03-04 13:09:16.036 [INFO] -> Triggering worker: SystemMonitor

reven

There's a setting to log the scanner. Turn that on too see what's going on. It's off by default as the scanner runs often and it creates a log of logging

thepix

reven I'll give it a look.

thepix

I presumme this is the setting you were referring to?

I don't really see any difference to the log results when this is on or off, there seems to be no Full Scan results and only seems to trigger when a WatchLibrary item is identified.

2023-03-05 12:54:48.620 [INFO] -> Triggering worker: FlowRunnerMonitor 2023-03-05 12:54:54.049 [INFO] -> Triggering worker: FlowWorker 2023-03-05 12:54:54.052 [INFO] -> Making request [GET]: http://localhost:5000/api/settings/current-config/revision 2023-03-05 12:54:54.055 [INFO] -> Triggering worker: SystemMonitor 2023-03-05 12:54:54.057 [INFO] -> Making request [GET]: http://localhost:5000/api/node/by-address/INTERNAL_NODE 2023-03-05 12:54:54.060 [DBUG] -> At limit of running executors on 'Internal Processing Node': 1 2023-03-05 12:54:54.063 [DBUG] -> Getting directory size (LOGGING DIR) 'C:\Users\xxxxxxx\AppData\Roaming\FileFlows\Logs': 4.99 MB 2023-03-05 12:54:54.066 [DBUG] -> Getting directory size (TEMP DIR) 'C:\Users\xxxxxxx\AppData\Roaming\FileFlows\Temp': 26.71 MB 2023-03-05 12:54:58.629 [INFO] -> Triggering worker: FlowRunnerMonitor 2023-03-05 12:55:00.468 [INFO] -> Triggering worker: FileFlowsTasksWorker 2023-03-05 12:55:00.531 [DBUG] -> Usenet - Sonarr queueing file: D:\Downloads\FileFlows\Usenet\Sonarr\Django.2023.S01E03.1080p.WEB.H.264-WAYNE\~syncthing~Django.2023.S01E03.1080p.WEB.H.264-WAYNE.mkv.tmp 2023-03-05 12:55:00.546 [DBUG] -> Usenet - Sonarr Dequeued: D:\Downloads\FileFlows\Usenet\Sonarr\Django.2023.S01E03.1080p.WEB.H.264-WAYNE\~syncthing~Django.2023.S01E03.1080p.WEB.H.264-WAYNE.mkv.tmp 2023-03-05 12:55:00.560 [ERRR] -> Failed to calculate fingerprint for file 'D:\Downloads\FileFlows\Usenet\Sonarr\Django.2023.S01E03.1080p.WEB.H.264-WAYNE\~syncthing~Django.2023.S01E03.1080p.WEB.H.264-WAYNE.mkv.tmp': The process cannot access the file 'D:\Downloads\FileFlows\Usenet\Sonarr\Django.2023.S01E03.1080p.WEB.H.264-WAYNE\~syncthing~Django.2023.S01E03.1080p.WEB.H.264-WAYNE.mkv.tmp' because it is being used by another process. at Microsoft.Win32.SafeHandles.SafeFileHandle.CreateFile(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options) at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize) at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize) at FileFlows.ServerShared.Helpers.FileHelper.CalculateFingerprint(String file) in /src/ServerShared/Helpers/FileHelper.cs:line 65 2023-03-05 12:55:00.563 [DBUG] -> New unknown file: D:\Downloads\FileFlows\Usenet\Sonarr\Django.2023.S01E03.1080p.WEB.H.264-WAYNE\~syncthing~Django.2023.S01E03.1080p.WEB.H.264-WAYNE.mkv.tmp 2023-03-05 12:55:00.967 [DBUG] -> LibraryWorker: Library 'Usenet - Sonarr' was scanned recently 00:19:00.0663679 (full scan interval 60 minutes) 2023-03-05 12:55:00.972 [DBUG] -> LibraryWorker: Library 'Usenet - Radarr' was scanned recently 00:19:00.0896660 (full scan interval 60 minutes) 2023-03-05 12:55:00.975 [DBUG] -> LibraryWorker: Library 'Torrents - Radarr' was scanned recently 00:19:00.1119581 (full scan interval 60 minutes) 2023-03-05 12:55:00.979 [DBUG] -> LibraryWorker: Library 'Torrents - Sonarrr' was scanned recently 00:19:00.1503567 (full scan interval 60 minutes) 2023-03-05 12:55:00.981 [INFO] -> Triggering worker: LibraryWorker 2023-03-05 12:55:04.067 [INFO] -> Triggering worker: SystemMonitor 2023-03-05 12:55:04.071 [INFO] -> Triggering worker: FlowWorker 2023-03-05 12:55:04.073 [INFO] -> Making request [GET]: http://localhost:5000/api/settings/current-config/revision 2023-03-05 12:55:04.077 [INFO] -> Making request [GET]: http://localhost:5000/api/node/by-address/INTERNAL_NODE 2023-03-05 12:55:04.079 [DBUG] -> Getting directory size (LOGGING DIR) 'C:\Users\xxxxxxx\AppData\Roaming\FileFlows\Logs': 5 MB 2023-03-05 12:55:04.082 [DBUG] -> At limit of running executors on 'Internal Processing Node': 1 2023-03-05 12:55:04.085 [DBUG] -> Getting directory size (TEMP DIR) 'C:\Users\xxxxxxx\AppData\Roaming\FileFlows\Temp': 26.71 MB 2023-03-05 12:55:05.565 [INFO] -> Cannot read file: D:\Downloads\FileFlows\Usenet\Sonarr\Django.2023.S01E03.1080p.WEB.H.264-WAYNE\~syncthing~Django.2023.S01E03.1080p.WEB.H.264-WAYNE.mkv.tmp 2023-03-05 12:55:05.568 [INFO] -> Cannot write file: D:\Downloads\FileFlows\Usenet\Sonarr\Django.2023.S01E03.1080p.WEB.H.264-WAYNE\~syncthing~Django.2023.S01E03.1080p.WEB.H.264-WAYNE.mkv.tmp 2023-03-05 12:55:05.570 [WARN] -> Failed access checks for file file: D:\Downloads\FileFlows\Usenet\Sonarr\Django.2023.S01E03.1080p.WEB.H.264-WAYNE\~syncthing~Django.2023.S01E03.1080p.WEB.H.264-WAYNE.mkv.tmp These checks can be disabled in library settings, but ensure the flow can read and write to the library. 2023-03-05 12:55:05.573 [DBUG] -> Time taken "00:00:05.0010049" to test can access file: "D:\Downloads\FileFlows\Usenet\Sonarr\Django.2023.S01E03.1080p.WEB.H.264-WAYNE\~syncthing~Django.2023.S01E03.1080p.WEB.H.264-WAYNE.mkv.tmp"

Many files added to the folders past 2:30am have not been caught and processed. Even changing the Scan option in the libraries to "Scan" and rescanning produces 0 results?

reven

The process cannot access the file 'D:\Downloads\FileFlows\Usenet\Sonarr\Django.2023.S01E03.1080p.WEB.H.264-WAYNE\~syncthing~Django.2023.S01E03.1080p.WEB.H.264-WAYNE.mkv.tmp' because it is being used by another process.

FileFlows cannot get write access to this file

thepix

reven this is irrelevant, it's a temp file with a file extension .tmp that shouldn't be processed anyway.

The relevant files are those Date Modified in the images between 02:30 and 12:53 and are not being picked up in the same folder that others have.

reven

well your filter looks wrong then, if youre trying to scan a .tmp file

thepix

reven Again, this isn't exactly relevant to the issue at hand. I've since added an exclusion filter for .tmp files.

The folders with the files that haven't been scanned have the video files in them, same as the folders in the images that did get scanned.

From what I can see in the logs, there is either no Full Scan triggering or if there is it isn't finding anything? It only seems to process files that have been found via WatchLibrary and if it doesn't find it first time around, it never will.

reven

assuming you didnt check "Folders"

how it works, is it will even watch for file system events, and if that file matches the filter and doesnt match the inclusion filter, it will add it to the scan queue.
or on a full scan (if set to scan mode, or every hour), it will find all files in all subfolders matchign the filter and not the inclusion pattern and add it to the queue.

then there is a worker that constantly monitors that queue and dequeues items. when it dequeues an item, thats when it checks if it is already known, if the file can bet read/written to etc.

So this can cause excessive logging since if you have 10,000 files in your files, it will check those 10,000 files each scan. the level of checking will vary, if they are known, its a quick lookup. but still it will log it.

So i was responding to whats in your logs.

thepix

reven When I set the Library up it was the General tab, Detection > 100mb and Advanced just the exclusion filters.

I understand the basics of what should happen, but as the title suggests it doesn't seem to be doing what you've described on a consistant basis for me.

I had it setup last night and one folder didn't seem to either detect on a system watch or thereafter on the hourly Full Scan. So I switched to only add files in the one folder that was working, only to wake up to the fact that new files were added overnight and not detected. Then I added more files later and they were detected, but the others still weren't.

I switched on the scanner log as suggested to see additional info as to what issue may be, but without posting the whole log (which was rows and rows of the same info), I posted the most relevant, yet that still doesn't seem to indentfy why the process isn't working as you describe.

I should add the the folders being watched are active download folders, but I presumme as some files are detected that this shouldn't be an issue and it should know a complete file from incomplete?

reven

it (by default, can adjust this time), will check the filesize, then wait 5 seconds and check the filesize again. if the file size changes, it will skip it from adding to the system this time (next scan should pick it up). This is mostly designed for when a file is being moved into a folder. I wouldnt recommend scanning a downloading folder, rather have a incomplete/complete folder when the download completes. but up to you.

then i suggest you look for a file you know it hasnt found, turn on the excess logging, hit rescan and wait for the scan to complete, then search the logs for that one file to see if it says anything about it.

thepix

reven Issue is I'm copying files from a SeedBox, so in theory the system should know by the 5 sec check that the file is incomplete I guess.

Still it's weird that the Full Scan hours after the file is complete hasn't detected the new files.

I've given new files and it has handled them as expected, it's just these ones previously described. It's as if they've been added to a Blacklist and being refused admission.

thepix

reven Is it possible to let me know if there's a files where the completed scanned file list is kept that I can delete, as I'm still having continued issues with the program not recognising partial vs finished downloaded files.

reven

Just stored in the database file

thepix

reven where exactly is this db file, I've tried the fileflows.sqlite in the Data folder to no avail?

reven

yes, thats the sqlite database file

thepix

reven looked at the db and it just seems to be saved info on the settings, nodes, flows and libraries, not of the finger printed files it check against to determine if as file has previously been scanned or processed?