diff --git a/README.md b/README.md index ae39671..c5b738e 100755 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # RuGiVi - Adult media landscape browser -RuGiVi enables you to fly over your image collection and view thousands of images at once. Zoom in and out from one image to small thumbnails with your mousewheel in seconds. All images are grouped as you have them on your disk and arranged in a huge landscape. RuGiVi can work with hundred thousand of images at once. +RuGiVi enables you to fly over your image and video collection and view thousands of images and video frames at once. Zoom in and out from one image to small thumbnails with your mousewheel in seconds. All images are grouped as you have them on your disk and arranged in a huge landscape. RuGiVi can work with hundred thousand of images at once. RuGiVi integrates fully into the [Fapel-System](https://github.com/pronopython/fapel-system), so you can travel trough your collection and tag images as you fly over them! @@ -30,10 +30,17 @@ Always visible, floating Fapel Tables: ![](img/rugivi4.gif) +RuGiVi works with images... ![](img/2306.png) -Zoom in (screenshot is obviously censored ;-) ) +...and with videos, which it displays as a set of video still frames in the landscape: + +![](img/202401895.jpg) + +(Everything here is obviously censored ;-) ) + +Zoom in... ![](img/0.jpg) @@ -66,26 +73,23 @@ You can also export a world map: # Benefits -* Works with hundreds of thousands of images **at the same time** +* Works with hundreds of thousands of images and videos **at the same time** * Tested with around 700.000 images (see the world map shown here), that's a RuGiVi Pixel size of 4.600.000 x 4.400.000 pixels or 20.240.000 Megapixels or 10.120.000 Full HD Screens to be scrolled through * Dynamic view rendering - screen is updated partially when drawing takes more time * Thumbnails are cached in a database +* Video still frames are cached in a cache directory structure * Works together with the [Fapel-System](https://github.com/pronopython/fapel-system) * Uses PyGame to render everything * With some tweaks it should work under MacOS (but was not tested) - - - # Requirements * Microsoft Windows or Ubuntu Linux (was written on Ubuntu, should work on all big distros). Works probably on MacOS when tweaked * 8 GB RAM or more * Python 3 * PIP Python Package Manager - - +* VLC is recommended for video playback # Installation @@ -95,7 +99,7 @@ You can also export a world map: ### 1. Clone this repository -Clone this repository into a writeable directory. +Download or clone the [latest release](https://github.com/pronopython/rugivi/releases/latest/) into a writeable directory. ### 2. Install python tkinter and pillow @@ -125,7 +129,7 @@ Note that it is advised to backup your databases (see further below). ### 1. Clone this repository -Clone this repository into a writeable directory. +Download or clone the [latest release](https://github.com/pronopython/rugivi/releases/latest/) into a writeable directory. ### 2. Uninstall old version with pip @@ -139,17 +143,20 @@ to uninstall the old version, but leave config file and database intact. Run -`pip uninstall .` +`pip install .` in the repository root (the one containing the `setup.py` file) +### 4. Configure RuGiVi + +Proceed with [Configure RuGiVi](#configure-rugivi) (you need to do this!) ## Windows Installation ### 1. Clone or download this repo -Yes, clone or download this repo now! +Download or clone the [latest release](https://github.com/pronopython/rugivi/releases/latest/) into a writeable directory. ### 2. Install Python @@ -167,7 +174,7 @@ It installs RuGiVi via Python's PIP and creates the start menu entries. ### 4. Configure RuGiVi -Proceed with [Configure RuGiVi](#configure-rugivi) (you need to do this!) +Proceed with [Configure RuGiVi](#configure-rugivi) (you need to do this, because it migrates your config file!) ## Windows Upgrade @@ -178,7 +185,7 @@ Note that it is advised to backup your databases (see further below). ### 1. Clone this repository -Clone this repository into a writeable directory. +Download or clone the [latest release](https://github.com/pronopython/rugivi/releases/latest/) into a writeable directory. ### 2. Run upgrade batch file @@ -188,13 +195,16 @@ Doubleclick to uninstall the old version, but leave config file and database intact and then automatically install the new version. +### 3. Configure RuGiVi + +Proceed with [Configure RuGiVi](#configure-rugivi) (you need to do this, because it migrates your config file!) # Configure RuGiVi Run `rugivi_configurator`: -![](img/2302.png) +![](img/202401125.png) Change the entries to fit your setup. @@ -205,6 +215,9 @@ Change the entries to fit your setup. |Crawler root directory | This is the root directory of all the pictures you want to explore within RuGiVi | |Crawler World DB File | This is the database file the "world" (the position of all files on the screen) is saved to| |Thumb DB File| The Database containing all thumbnails. *This can be several GB in size!*| +|Enable video crawling|When "true", RuGiVi will also parse video files| +|Video still cache directory| The directory where RuGiVi will save video still frames as jpg images| +|Play video with VLC| When "true", RuGiVi will try to use VLC as the video player when opening a video by pressing `n`. Otherwise it will use the system default video player.| |Reverse Scroll Wheel Zoom| Changes the direction for zooming. Set it "true" when using a trackpad| | Status font size | Font size of the grey status area| | FapTable parent dirs | See FapTables | @@ -212,6 +225,8 @@ Change the entries to fit your setup. > :cherries: *You must use a new World DB File or delete the old one when changing root directory* +> :cherries: *rugivi_configurator migrates your old config file when you upgrade from a previous version of RuGiVi. Make sure to press "save and exit" even when you do not change settings yourself!* + Make sure your database files are placed on a SSD drive! # Start RuGiVi @@ -248,12 +263,13 @@ You can watch this process or start traveling through the images world. |1-7 |Zoom levels | |0 |Zoom fit | |j |Jump randomly | -|n |Open image with system image viewer | +|n |Open image with system image viewer; Open video with system video player of VLC | |t |Call [Fapel-System](https://github.com/pronopython/fapel-system) Tagger on selection| |s |Call [Fapel-System](https://github.com/pronopython/fapel-system) Set on selection | |up / down |Open and go through Fapel Table rows | |left / right |Go through Fapel Tables of one row | |g |Open Dialog to go to a spot by coordinates x,y | +|c |Open information dialog | |o |Pause image server (troubleshooting) | |p |Pause crawler (troubleshooting) | |e |Generate and export world map as png file | @@ -267,6 +283,79 @@ Close RuGiVi by closing the main window ("x"). If it works and all databases and Sometimes RuGiVi hangs up during shutdown (a known bug). If so, please go into the command line and kill RuGiVi with Ctrl+C. +# Video handling + +If you have video crawling enabled through rugivi_configurator, RuGiVi will parse video files and add them as a set of video still frames to your world map. + +RuGiVi supports mp4, mkv, webm, mov, rm, avi, flv, wmv and asf video files. + +![](img/202401365.png) + +Here a video is represented by video frames (A) which will be generated by RuGiVi and placed anlongside normal images (B). + + +![](img/202401982.jpg) + +Different video length result in different amount of images. RuGiVi tries to at least get 2 images (20 seconds videos / reels / shorts in the above example). Longer videos get at least 1 frame every 15 seconds. The 92 minutes video in the example above results in about 360 images. + +| Duration | images | +|-------|------| +| few seconds | 2 | +| 30 sec | 3 | +| 1 min | 4 | +| 2 min | 8 | +| 5 min | 20 | +| 10 min | 40 | +| 30 min | 120 | +| 60 min | 249 | +| 90 min | 360| + +## Playback via standard app or VLC media player + +If you open a video by pressing `n`, RuGiVi opens it in your standard video player. + +The video will start from the beginning, no matter which still frame you selected: + +![](img/202401521.jpg) + +If you have the [VLC media player](https://www.videolan.org/vlc/) installed, you can enabled VLC playback in rugivi_configurator. RuGiVi will then open up VLC for playback and seek the video to 2 seconds before the frame you selected: + +![](img/202401487.jpg) + +You can switch off the seeking feature via RuGiVi config file (not supported by the configurator). + +## Remove Letterbox + +RuGiVi removes black borders ("Letterbox") around video frames by default. + +![](img/202401888.jpg) + +In this example all small images of this video within RuGiVi have no black stripes on top and bottom whereas the video itself has these black letterbox stripes. + +You can switch off this removal of black borders in RuGiVi config file (not supported by the configurator). + + +# Information dialog + +Pressing `c` while a file is selected brings up an information dialog about that file: + +![](img/202401254.png) + +The information dialog is especially handy if you want to copy and paste directory or filename information of your selection. Just select the text you want to copy and use the right click menu to copy the text to your clipboard. + +| Information| Description| +|--|--| +|Image file| the original image| +|Video file| the video file the still image you selected in RuGiVi was taken from| +|Parent directory of image/video file| the directory without the filename| +| Position (sec)| the video position in seconds of the selected still image| +|Still image in cache| the path and filename of the still image RuGiVi uses (the location in the cache)| +|Parent directory of still image in cache | the directory without the filename of the still image (the cache sub directory)| +|Selected Spot| the x,y coordinates of the selected spot| +|Ordered Quality|The needed quality to display the picture. Image Server will fetch this quality. (0 Thumb, 1 Grid, 2 Screen, 3 Original)| +|Available Quality|The current loaded quality in memory| +|State| Image Server's state of this image| + # FapTables FapTables are a way to always keep specific media in the view floating over the world view all the time. @@ -379,19 +468,32 @@ You will need 1 GB of RAM for every 150,000 images in your world. Note that you can only run World Overlook one time per RuGiVi Session. You need to restart RuGiVi, if you want to export another map. -# Troubleshooting +# Exclude directories from crawling + +You can edit RuGiVi's config file to exclude subdirectories of the root directory from crawling: +``` +crawlerRootDir=/my/root/dir +crawlerExcludeDirList=/my/root/dir/DoNotCrawl;/my/root/dir/No access here +``` + +Multiple directories are seperated with a `;`. + +> :cherries: *This feature is currently not supported by the configurator. You have to edit the config manually.* + +# Troubleshooting |Problem |Solution | |---------------|-------------------------------------------------------------| | RuGiVi does not start| Try starting RuGiVi via console (with `rugivi`) and look at the error messages. Common reasons is a broken or incorrect config file. You can also try to define new world and thumb database files.| |Screen is black|Probably Edit mode is enabled, press middle mousebutton again| |Images are pixelated or just red | RuGuVi is probably loading a lot of images and the ones you currently want to see are at a later position at the loader queue. Keep an eye on "Queue" in the information box ( press `i` to show it) | +|Images stay red even after waiting|When the original file is an image, the image is missing. If it is a video, the cache file is missing or the cache is broken.| |RuGiVi is slow | Loading takes time and RuGiVi loads a lot of data. See the Tips section for speed tips. | |Fapel-System keys (t and s) crash RuGiVi, despite the Fapel-System being installed | Check RuGiVi Config file if the correct python executable is mentioned under section `[control]`, `pythonexecutable=` | - - +|Crash and `FileNotFoundError: [Errno 2] No such file or directory: 'vlc'` when pressing `n`| You have playback via VLC enabled but it is not installed or the path to vlc binary (vlc.exe on Windows) is wrong. Change vlc binary path in rugivi.conf| +|Config entry missing! Error: The following group / key combination is missing in your RuGiVi config... | The shown config entry is missing in your config file. Please add it manually in your rugivi.conf. Look into rugivi dir of git repo for a default config file. You can also run `rugivi_configurator` to add missing entries.| # Tips @@ -399,13 +501,25 @@ Note that you can only run World Overlook one time per RuGiVi Session. You need * You can speed up everything by using a fast SSD and a lot of RAM. When running inside a virtual machine make sure the machine has enough virtual processors, a lot of RAM and its disk files are on SSDs. -# Known bugs and limitations +# Troubleshooting + +## Debug video still generation and playback + +In the config file you can enable verbose output of CV2 (which is used for video still generation) and VLC (which plays back videos): + +``` +[debug] +vlcverbose=True +cv2verbose=True +``` + +## Known bugs and limitations * Quit does not work everytime. The crawler then runs in an infinite loop. * Sometimes the garbage collection is not run or runs too late and memory is depleted (rugivi can easily grab gigabytes of RAM of course). You then get an out-of-memory error. * Despite not changing anything, RuGiVi does not work on read-only media directories. This is because pygame image loader opens files with write access. * TIFF files may produce warnings which on Windows are opened as separate message boxes. -* Big worlds (> 500,000 images) are not round or rectangular but have the shape of a plus sign ("+"), see big world map example on this page. This is due to the crawler placement algorithm, which is in alpha state. * Audio pops at startup (probably a pygame issue) +* High res versions are unloaded despite being displayed (pictures end up showing as a colored square when zoomed beyond thumb size) # Technical Stuff @@ -447,6 +561,7 @@ Each thumb needs around 3 to 6 kB of RAM, so for world map generation you will n ## Disk space needed +### Database size The thumbs.sqlite file will use the following disk space depending on your world size. | Number of images | approx. thumb db size | @@ -460,6 +575,107 @@ The thumbs.sqlite file will use the following disk space depending on your world The chunks.sqlite will stay around 100 MB. +A video file will gegenerate approximatly 4 images per minute. + +### Video still frame cache size + +| jpg quality | maxsize | MB / 1000 video still frame images | MB / 1000 videos| | +|--|--|--|--|--| +| 65 | 800 | 11.4 | 321 | | +| 65 | - | 16.6 | 467 | default | +| 75 | - | 19.5 | 548 | | +| 95 | 800 | 27.4 | 771 | | +| 95 | - | 46.9 | 1318 | | + +Note that the "*MB / 1000 videos*" column contains my personal experience with my video collection (broad mix of short and long, small and Full-HD videos). If you have tenthousand 4k videos your milage may vary. + +You can change the video still image quality manually in RuGiVi config. Just see further down how to do this. + +## Video still frame cache + + +### Cache design +When you configure a directory as the video still cache in rugivi_configurator, RuGiVi uses it as the base directory for sub directories to store the video still frames in it. + +Every frame you see in the RuGiVi landscape will then be a jpg image file within the cache. + +The typical content of one of the cache directories then looks like this: + +![](img/202401274.png) + +Since all still frames are stored by random names and in random cache folders, every folder contains random still frames. + +> :eggplant: :sweat_drops: *Note: The files within the cache are plain normal jpg images of your beloved videos. So just be aware that the cache contains this content!* + +### Cache image quality settings + +Via the config file you can manually edit and set the quality of the generated still video frames: + +``` +[videoframe] +jpgquality=65 +maxsizeenabled=False +maxsize=800 +``` +| Setting | Description| +|--|--| +|jpgquality| The jpg quality in the typical range from 0 (lowest) to 100 (highest). Recommended is 55-95.| +|maxsizeenabled| When set to "True" then RuGiVi will scale down video still frames when they exceed the maxsize value.| +|maxsize| The size in pixel a video frame is scaled down to (height and width)| + +Note that changed settings only affect newly generated images. + + +### Cache Maintenance + +When you create a new database and do not delete the cache or when RuGiVi crashes without saving the last changes, video still frame image files will remain unused and orphaned in the cache. + +RuGiVi comes with a small commandline tool to clean up the cache. + +When you are **not running RuGiVi**, run + +`rugivi_image_cache_maintenance` + +to gather information about the cache status (*nothing will be changed yet!*): + +``` +Opening config file /home/xxxxxxx/.config/rugivi.conf +Reading Chunks from Database:done +Scanning files of cache:done +Gathering information on files in cache:done +World DB files: 1000 +Cache files : 1530 +Cache size: 60 MB +Finding orphant cache files:done +Orphant files : 530 +NOTHING WAS CHANGED YET! +RUN THIS TOOL AGAIN with '-c' to move orphant files and clean up the cache! +``` + +In this example 530 of 1530 files can be deleted. + +Run it again with the parameter `-c` to actually clean the cache: + +`rugivi_image_cache_maintenance -c` + +All orphant files will be moved to a `trash_` directory: + +``` +530 file(s) moved to trash: /xxxx/xxxxx/rugivi_cache/trash_20240101_120000 +Check these files if they are ok to be deleted and then delete the 'trash_*' folder manually. +``` + +You can now go to the mentioned directory and check and delete the orphaned files manually. + +> :cherries: *No files are deleted by this tool itself! This is a precaution because the tool also has access to your image and video collection.* + +If you get a + +`Warning: There is already a trash folder present: /.../.../trash_20240101_120000` + +the maintenance tool reminds you that there is already a trash folder present which might take up disk space. + + ## Information display Press `i` a few times to show the information display. @@ -486,21 +702,79 @@ Press `i` a few times to show the information display. |Queue| Image Server Queue (both Disk and DB access)| |World Overlook|Only displayed when a map is being generated. Shows the status of the generation process.| +## Crawler World Building Customization + +You can customize how the world is layed out via settings in rugivi.conf (not supported via rugivi_configurator). + +| Setting |Default| Description | +|---|---|--| +|crossshapegrow| `False`|`True` = Shape world like a cross. `False` = Shape world like a ball. Earlier versions of RuGiVi had a bug that resulted in a world shaped like a cross / plus sign instead of a round world.| +|nodiagonalgrow| `True`| `True` = Pictures of a set a placed only top/bottom/left/right, never diagonal.| +|organicgrow| `True`| `True` = Picture sets are more sponge-like| +|reachoutantmode| `True`| `True` = Sometimes sets are grown like a spike reaching out from the center| + -## Backup database files -If you want to backup your config and database files, just make a copy of these files. If you did not change the location in the `rugivi_configurator`, then you find the files here: +## Backup database files and cache +If you want to backup your config and database files and the video still frame cache, just make a copy of these files and directories. If you did not change the location in the `rugivi_configurator`, then you find the files and directories here: -|file | Linux | Windows | + +|file/dir | Linux | Windows | |------------|---------------------|---------------------------| |rugivi.conf | ~/.config/ | C:\Users\\[username]\\AppData\Roaming\RuGiVi | |thumbs.sqlite | ~/.local/share/rugivi/ | C:\Users\\[username]\\AppData\Roaming\RuGiVi | |chunks.sqlite | ~/.local/share/rugivi/ | C:\Users\\[username]\\AppData\Roaming\RuGiVi | +|video still frame cache directory| ~/.local/share/rugivi/cache | C:\Users\\[username]\\AppData\Roaming\RuGiVi\cache| + + + + +## Mockup images mode for testing and development + +RuGiVi includes a feature for testing the crawler. If you enable + +``` +[debug] +... +mockupimages=True +``` +in `rugivi.conf`, RuGiVi will not load images but rather represent every directory and every video file it finds with empty spots. Each directory and each video file will be represented by a different color. The world is build up faster without loading images. + +# 📢 Community Support + +The [GitHub discussion boards](https://github.com/pronopython/rugivi/discussions) are open for sharing ideas and plans for RuGiVi. + +You can report errors through a [GitHub issue](https://github.com/pronopython/rugivi/issues/new). + +Don't want to use GitHub? You can also contact me via email: pronopython@proton.me If you want to contact me anonymously, create yourself a burner email account. # Release Notes +## v0.4.0-alpha + +### added + +- Video file support including still frame cache and external playback for mp4, mkv, webm, mov, rm, avi, flv, wmv and asf video files +- Information dialog +- Subdirectories of the root directory can now be excluded from crawling +- Video still image cache maintenance cli tool +- mockup images mode for testing and development of crawler +- Crawler world building can be customized via config + +### changed + +- Crawler wait times optimized +- Smaller sizes and thumbs now use antialias scaling +- Images are shown down to 5 pixel width/height, color starts at 4 pixels + +### fixed + +- Crawler border correction bug (crawler is now faster) +- Surface draw crash when using peek +- View fetch loop for images bigger than thumb size even ran when zooming out to or smaller than thumb size. Zooming out with `7` now is faster. + ## v0.3.1-alpha ### added diff --git a/img/202401125.png b/img/202401125.png new file mode 100755 index 0000000..66e213d Binary files /dev/null and b/img/202401125.png differ diff --git a/img/202401254.png b/img/202401254.png new file mode 100755 index 0000000..1efa7ae Binary files /dev/null and b/img/202401254.png differ diff --git a/img/202401274.png b/img/202401274.png new file mode 100755 index 0000000..8dd1e35 Binary files /dev/null and b/img/202401274.png differ diff --git a/img/202401365.png b/img/202401365.png new file mode 100755 index 0000000..b051417 Binary files /dev/null and b/img/202401365.png differ diff --git a/img/202401487.jpg b/img/202401487.jpg new file mode 100755 index 0000000..95ffc5e Binary files /dev/null and b/img/202401487.jpg differ diff --git a/img/202401521.jpg b/img/202401521.jpg new file mode 100755 index 0000000..0564318 Binary files /dev/null and b/img/202401521.jpg differ diff --git a/img/202401888.jpg b/img/202401888.jpg new file mode 100755 index 0000000..7ed4dc3 Binary files /dev/null and b/img/202401888.jpg differ diff --git a/img/202401895.jpg b/img/202401895.jpg new file mode 100755 index 0000000..70a5ea0 Binary files /dev/null and b/img/202401895.jpg differ diff --git a/img/202401982.jpg b/img/202401982.jpg new file mode 100755 index 0000000..1b0756a Binary files /dev/null and b/img/202401982.jpg differ diff --git a/img/2302.png b/img/2302.png deleted file mode 100755 index 8c542c8..0000000 Binary files a/img/2302.png and /dev/null differ diff --git a/rugivi/config_file_handler.py b/rugivi/config_file_handler.py index b30172b..5d3bcbb 100755 --- a/rugivi/config_file_handler.py +++ b/rugivi/config_file_handler.py @@ -32,6 +32,8 @@ import os from pathlib import Path import configparser +import sys +from tkinter import messagebox from . import dir_helper as dir_helper @@ -59,15 +61,19 @@ def get_config_parser(self) -> configparser.RawConfigParser: return self.config_parser def get(self, group, key) -> str: + self.check_key(group, key) return self.config_parser.get(group, key) def get_int(self, group, key) -> int: + self.check_key(group, key) return int(self.config_parser.get(group, key)) def get_boolean(self, group, key) -> bool: + self.check_key(group, key) return self.config_parser.get(group, key) in ("True", "TRUE", "true", "1") def get_directory_path(self, group, key, ifEmpty="") -> str: + self.check_key(group, key) path = self.get(group, key) if path == "": return ifEmpty.replace("~", self.homedir) @@ -85,3 +91,14 @@ def write_changed_config(self) -> None: print("Writing changes to config file", self.config_file_path) configfile = open(self.config_file_path, "w") self.config_parser.write(configfile) + + def check_key(self, group, key): + try: + self.config_parser.get(group, key) + except configparser.NoOptionError: + errortext = 'Error: The following group / key combination is missing in your RuGiVi config "'+self.config_file_path+'":\n\n['+group+']\n'+key+'\n\nLook into rugivi dir of git repo for a default config file!' + print("Config entry missing!") + print(errortext) + messagebox.showerror('Config entry missing!', errortext) + sys.exit() + diff --git a/rugivi/crawlers/first_organic/organic_crawler_first_edition.py b/rugivi/crawlers/first_organic/organic_crawler_first_edition.py index 70e6d55..adf85b0 100644 --- a/rugivi/crawlers/first_organic/organic_crawler_first_edition.py +++ b/rugivi/crawlers/first_organic/organic_crawler_first_edition.py @@ -29,6 +29,7 @@ ############################################################################################## # +import math from rugivi.image_service.image_server import ( Frame, ImageServer, @@ -50,6 +51,7 @@ import threading from pathlib import Path from time import sleep +from time import time class OrganicCrawlerFirstEdition: @@ -60,12 +62,21 @@ class OrganicCrawlerFirstEdition: START_SPOT_SEARCH: int = START_SPOT_SEARCH_NEAR_PARENT + SAVE_EVERY_SECONDS = 60 * 10 # 10 minutes + def __init__( self, world: World, image_server: ImageServer, basedir: str, crawler_db_file_path: str, + crawl_videos=True, + excludeDirList=[], + mockup_mode=True, + cross_shape_grow = False, + no_diagonal_grow = True, + organic_grow = True, + reach_out_ant_mode = True, ) -> None: self.db = WorldDatabase(crawler_db_file_path, world, image_server) @@ -79,6 +90,9 @@ def __init__( self.basedir: str = self.db.get_and_put_if_none( WorldDatabase.KEY_BASEDIR, basedir ) + self.excludeDirList=excludeDirList + + self.mockup_mode = mockup_mode self.pause_crawling: bool = False """ can be set externally to True to halt crawling until this is set to False again""" @@ -88,21 +102,25 @@ def __init__( ) # organic grow will sponge-like grow the bioms and not more rectangular - self.ORGANIC_GROW: bool = True + self.ORGANIC_GROW: bool = organic_grow # only grow up/down, left/right and not diagonal # .o. # oxo x=start pos # .o. o=ok - self.NO_DIAGONAL_GROW: bool = True + self.NO_DIAGONAL_GROW: bool = no_diagonal_grow # Ant mode will create diagonal corridors from time to time - self.REACH_OUT_ANT_MODE: bool = True + self.REACH_OUT_ANT_MODE: bool = reach_out_ant_mode self.ANT_CORRIDOR_PROB: float = 1 # 0.5..1 less..more + self.CROSS_SHAPE_GROW = cross_shape_grow + self.running: bool = True self.crawler_loop_running: bool = False + self.crawl_videos = crawl_videos + def run(self) -> None: self.thread = threading.Thread(target=self.crawler_loop, args=()) self.thread.daemon = True # so these threads get killed when program exits @@ -125,25 +143,28 @@ def crawler_loop(self) -> None: WorldDatabase.VALUE_STATUS_CRAWLING, ) - if status == WorldDatabase.VALUE_STATUS_CRAWL_COMPLETED: # TODO craw completed is not used yet! + if ( + status == WorldDatabase.VALUE_STATUS_CRAWL_COMPLETED + ): # TODO craw completed is not used yet! return - border_spots: set = self.db.get( + self.border_spots: set = self.db.get( WorldDatabase.KEY_BORDERSPOTS ) # a set of *all* empty spots directly neighbouring to used spots - if border_spots == None: - border_spots = set() - border_spots.add((0, 0)) + if self.border_spots == None: + self.border_spots = set() + self.border_spots.add((0, 0)) basedir_path = Path(self.basedir) basedir_parent_path = basedir_path.parent.absolute() - dir_and_start_spot: dict = self.db.get(WorldDatabase.KEY_DIRANDSTARTSPOT) - if dir_and_start_spot == None: - dir_and_start_spot = {} + self.dir_and_start_spot: dict = self.db.get(WorldDatabase.KEY_DIRANDSTARTSPOT) + if self.dir_and_start_spot == None: + self.dir_and_start_spot = {} next_save_at_number_of_frames = 10000 + last_save_time = time() ############################################################################## # Crawl directory by directory @@ -160,10 +181,7 @@ def crawler_loop(self) -> None: sleep(1) self.status = "paused" - if self.imageServer.get_queue_size() > 100: - while self.imageServer.get_queue_size() > 10 and self.running == True: - self.status = "sleeping" - sleep(0.2) + self.__pause_for_image_queue() ############################################################################## # Directory known? @@ -172,295 +190,248 @@ def crawler_loop(self) -> None: self.status = "crawling for next dir" current_dir_absolute_path = Path(current_dir).absolute() - if str(current_dir_absolute_path) in dir_and_start_spot: + if str(current_dir_absolute_path) in self.dir_and_start_spot: # already visited according to database + sleep(0.2) + continue + + skip_dir = False + for exdir in self.excludeDirList: + #print(current_dir_absolute_path," ?= ",exdir) + if str(current_dir_absolute_path).startswith(exdir): + print("Excluded dir",current_dir_absolute_path,"is skipped") + skip_dir = True + break + if skip_dir: + sleep(0.2) continue # populating dirAndStartSpot when this is crawler is run for the first time - if len(dir_and_start_spot) == 0: - dir_and_start_spot[str(basedir_path.absolute())] = (0, 0) - dir_and_start_spot[str(basedir_parent_path)] = (0, 0) + if len(self.dir_and_start_spot) == 0: + self.dir_and_start_spot[str(basedir_path.absolute())] = (0, 0) + self.dir_and_start_spot[str(basedir_parent_path)] = (0, 0) ############################################################################## - # Filter files + # HANDLE IMAGES ############################################################################## self.status = "filter files" - f_names = [ + image_f_names = [ file for file in f_names if file.lower().endswith((".jpg", ".jpeg", ".gif", ".png", ".tif")) ] - neededSpots = len(f_names) # TODO only count images - - if neededSpots == 0: - continue - - self.current_dir = os.path.basename(current_dir) + ############################################################################## + # Find empty spots + ############################################################################## - found_enough_empty_spots: bool = False + neededSpots = len(image_f_names) # TODO only count images - found_empty_spots: list = [] + start_spot = None # if something was added, this will be not None - start_spot = None + if neededSpots > 0: - self.status = "finding biome" - while not found_enough_empty_spots: + self.current_dir = os.path.basename(current_dir) - sleep(0.02) + start_spot, found_empty_spots = self.__find_empty_spots( + current_dir, neededSpots + ) + # __find_empty_spots returns with None,None if shutdown is initiated, so follow along if self.running == False: - print("Crawler loop stopped") self.crawler_loop_running = False return - found_empty_spots = [] - ############################################################################## - # Looking for a start spot - # Start spots are selected out of border spots (these always empty) + # Found enough empty spots, now these will be populated with images of dir ############################################################################## - start_spot = random.choice(tuple(border_spots)) + self.status = "creating frames" - if ( - self.start_spot_search_method - == OrganicCrawlerFirstEdition.START_SPOT_SEARCH_NEAR_PARENT - ): + for file_number, filename in enumerate(image_f_names): + file_path = os.path.join(current_dir, filename) - # Find a parent dir of the current dir that is already placed in the world. - # This can also be a grandparent dir etc if the parent dir is empty! - basedir_path = Path(current_dir) - - basedir = Path(self.basedir).absolute() - # error: 2 times going to parent dir! - # parentPath = path.parent.absolute() - basedir_parent_path = basedir_path.absolute() - # Going down the parent line - while len(str(basedir_parent_path)) > len(str(basedir)): - basedir_parent_path = basedir_parent_path.parent.absolute() - if str(basedir_parent_path) in dir_and_start_spot: - break + if not self.mockup_mode: + streamed_image = self.imageServer.create_streamed_image( + file_path, StreamedImage.QUALITY_THUMB + ) + else: + streamed_image = self.imageServer.create_streamed_mockup(file_path, color_string=self.current_dir) - parent_spot = dir_and_start_spot[str(basedir_parent_path)] - (parent_spot_x_S, parent_spot_y_S) = parent_spot + (start_spot_x_S, start_spot_y_S) = found_empty_spots[file_number] - # try x times to find an empty border spot closer and closer to the parent spot - for current_round in range( - 0, 30 - ): # was 300 -> more time needed for that (bad), but 30 is inaccurate - candidate = random.choice(tuple(border_spots)) - (candidate_x_S, candidate_y_S) = candidate - (start_spot_x_S, start_spot_y_S) = start_spot - # changed "or" to "and" - if abs(candidate_x_S - parent_spot_x_S) < abs( - start_spot_x_S - parent_spot_x_S - ) and abs(candidate_y_S - parent_spot_y_S) < abs( - start_spot_y_S - parent_spot_y_S - ): - start_spot = candidate + self.world.set_frame_at_S( + Frame(streamed_image), start_spot_x_S, start_spot_y_S + ) - elif ( - self.start_spot_search_method - == OrganicCrawlerFirstEdition.START_SPOT_SEARCH_NEAR_WORLD_CENTER - ): - for current_round in range(0, 50): - candidate = random.choice(tuple(border_spots)) - (candidate_x_S, candidate_y_S) = candidate - (start_spot_x_S, start_spot_y_S) = start_spot - if abs(candidate_x_S) < abs(start_spot_x_S) or abs( - candidate_y_S - ) < abs(start_spot_y_S): - start_spot = candidate + ############################################################################## + # HANDLE VIDEOS + ############################################################################## - """ - if True: # TODO insert "far away" code, do this from while to while - # also grow not around center but around start point of parent dir!!!!! - for i in range (0,50): - candidate = random.choice(borderSpots) - (cx,cy) = candidate - (sx,sy) = startSpot - if abs(cx) > abs(sx) or abs(cy) > abs(sy): - startSpot = candidate + if self.crawl_videos: + video_f_names = [ + file + for file in f_names + if file.lower().endswith( + ( + ".mp4", + ".mkv", + ".webm", + ".mov", + ".rm", + ".avi", + ".flv", + ".wmv", + ".asf", + ) + ) # TODO do all of the video formats work? All tested? + ] + else: + video_f_names = [] + + for video_filename in video_f_names: + + self.__pause_for_image_queue() + + video_file_path = os.path.join(current_dir, video_filename) + + if str(Path(video_file_path).absolute()) in self.dir_and_start_spot: + # already visited according to database + sleep(0.2) + continue - """ + positions = ( + self.imageServer.video_still_generator.analyze_and_get_positions( + video_file_path + ) + ) ############################################################################## - # See if enough empty spots are next to the start spot + # Find empty spots for video stills ############################################################################## - stack_with_empty_spots_to_check: list = [start_spot] - # stack always holds neighbouring empty spots to be checked - # if they are not that far away so that spots are grouped together - # and not so far away from another + neededSpots = len(positions) - while ( - len(stack_with_empty_spots_to_check) > 0 - and not found_enough_empty_spots - ): - sleep(0.02) - if self.running == False: - print("Crawler loop stopped") - self.crawler_loop_running = False - return + if neededSpots == 0: + continue # next video - ############################################################################## - # Select one of the empty 8 (or less) neighbours of the start_spot - ############################################################################## + self.current_dir = os.path.basename(current_dir) + + video_start_spot, found_empty_spots = self.__find_empty_spots( + current_dir, neededSpots + ) - if not self.ORGANIC_GROW: - currentSpot = stack_with_empty_spots_to_check.pop(0) - else: # organic grow - (start_spot_x_S, start_spot_y_S) = start_spot - currentSpot = random.choice(stack_with_empty_spots_to_check) + # __find_empty_spots returns with None,None if shutdown is initiated, so follow along + if self.running == False: + self.crawler_loop_running = False + return - # calculate if the next empty spot should "reach out", - # that means, should be farest instead of nearest neighbour + if start_spot == None: + # only use video start spot when no images had been added + # (dir has ONLY videos) + start_spot = video_start_spot - reach_out = False + ############################################################################## + # Found enough empty spots, now these will be populated with images of dir + ############################################################################## - if ( - self.REACH_OUT_ANT_MODE - and neededSpots - len(found_empty_spots) != 0 - ): - reach_out = ( - random.random() - < ((neededSpots - len(found_empty_spots)) / neededSpots) - * self.ANT_CORRIDOR_PROB - ) - - for current_round in range(0, 50): - candidate = random.choice(stack_with_empty_spots_to_check) - (candidate_x_S, candidate_y_S) = candidate - (ux, uy) = currentSpot - # if abs(cx-sx) < abs(ux-sx) or abs(cy-sy) < abs(uy-sy): - - # was both "or", now "and" - if reach_out: - # changed "or" to "and" - if abs(candidate_x_S - start_spot_x_S) > abs( - ux - start_spot_x_S - ) and abs(candidate_y_S - start_spot_y_S) > abs( - uy - start_spot_y_S - ): - currentSpot = candidate - else: - # changed "or" to "and" - if abs(candidate_x_S - start_spot_x_S) < abs( - ux - start_spot_x_S - ) and abs(candidate_y_S - start_spot_y_S) < abs( - uy - start_spot_y_S - ): - currentSpot = candidate + self.status = "creating frames for video file" - stack_with_empty_spots_to_check.remove(currentSpot) + for position_number, position in enumerate(positions): + ( + uuid_name, + file_path, + ) = ( + self.imageServer.image_cache.generate_new_cache_uuid_and_filename() + ) + if not self.mockup_mode: + streamed_image = self.imageServer.create_streamed_image( + file_path, StreamedImage.QUALITY_THUMB + ) + else: + streamed_image = self.imageServer.create_streamed_mockup(file_path, color_string=video_file_path) - # the current spot is taken... - found_empty_spots.append(currentSpot) + streamed_image.set_extended_attribute("video_still_uuid", uuid_name) + streamed_image.set_extended_attribute("video_file", video_file_path) + streamed_image.set_extended_attribute("video_position", position) - if len(found_empty_spots) >= neededSpots: - found_enough_empty_spots = True - break + (start_spot_x_S, start_spot_y_S) = found_empty_spots[ + position_number + ] - # ... and now look for its neighbours - (start_spot_x_S, start_spot_y_S) = currentSpot + self.world.set_frame_at_S( + Frame(streamed_image), start_spot_x_S, start_spot_y_S + ) + if video_start_spot != None: ############################################################################## - # Fill stack with all empty neighbours (up to 8 neighbours) + # remember current position for video ############################################################################## - for x_S in range(start_spot_x_S - 1, start_spot_x_S + 2): - for y_S in range(start_spot_y_S - 1, start_spot_y_S + 2): - if start_spot_x_S == x_S and start_spot_y_S == y_S: - # not the center spot (not a neighbour) - continue - - if ( - self.NO_DIAGONAL_GROW - and abs(x_S - start_spot_x_S) - - abs(y_S - start_spot_y_S) - == 0 - ): - continue - if (x_S, y_S) in found_empty_spots: - continue - if (x_S, y_S) in stack_with_empty_spots_to_check: - continue - if self.world.get_frame_at_S(x_S, y_S) != None: - continue - else: - stack_with_empty_spots_to_check.append((x_S, y_S)) - - ############################################################################## - # Found enough empty spots, now these will be populated - ############################################################################## - - self.status = "creating frames" + self.status = "remember video start spot" + # remember startSpot of this dir + video_file_path = Path(video_file_path).absolute() + self.dir_and_start_spot[str(video_file_path)] = video_start_spot - for file_number, filename in enumerate(f_names): - file_path = os.path.join(current_dir, filename) - streamed_image = self.imageServer.create_streamed_image( - file_path, StreamedImage.QUALITY_THUMB - ) - - (start_spot_x_S, start_spot_y_S) = found_empty_spots[file_number] - - self.world.set_frame_at_S( - Frame(streamed_image), start_spot_x_S, start_spot_y_S - ) + if start_spot != None: - ############################################################################## - # correct border spots (add new border spots of the new now filled spots and - # remove the border spots that now contain new frames) - ############################################################################## - - self.status = ( - "correcting border, " + str(len(found_empty_spots)) + " spots to check" - ) - # correct border spots - for currentSpot in found_empty_spots: - (start_spot_x_S, start_spot_y_S) = currentSpot - sleep(0.00003) + ############################################################################## + # remember current position and (sometimes) save everything to db / disk + ############################################################################## - # go through all 8 neighbours and check changes in "border status" - for x_S in range(start_spot_x_S - 1, start_spot_x_S + 2): - for y_S in range(start_spot_y_S - 1, start_spot_y_S + 2): - if self.world.get_frame_at_S(x_S, y_S) != None: - if (x_S, y_S) in border_spots: - border_spots.remove((x_S, y_S)) # no border anymore - else: - if (x_S, y_S) not in border_spots: - border_spots.add((x_S, y_S)) # a new border spot + self.status = "remember start spot" + # remember startSpot of this dir + basedir_path = Path(current_dir).absolute() + self.dir_and_start_spot[str(basedir_path)] = start_spot + + if ( + self.world.count_frames() > next_save_at_number_of_frames + or last_save_time + + OrganicCrawlerFirstEdition.SAVE_EVERY_SECONDS + < time() + ): + next_save_at_number_of_frames = ( + self.world.count_frames() + 10000 + ) + last_save_time = time() + self.__save_to_db(self.border_spots, self.dir_and_start_spot) + + if start_spot != None: - ############################################################################## - # remember current position and (sometimes) save everything to db / disk - ############################################################################## + ############################################################################## + # remember current position and (sometimes) save everything to db / disk + ############################################################################## - self.status = "remember start spot" - # remember startSpot of this dir - basedir_path = Path(current_dir).absolute() - dir_and_start_spot[str(basedir_path)] = start_spot + self.status = "remember start spot" + # remember startSpot of this dir + basedir_path = Path(current_dir).absolute() + self.dir_and_start_spot[str(basedir_path)] = start_spot - if self.world.count_frames() > next_save_at_number_of_frames: - next_save_at_number_of_frames += 10000 - self.__save_to_db(border_spots, dir_and_start_spot) + if ( + self.world.count_frames() > next_save_at_number_of_frames + or last_save_time + OrganicCrawlerFirstEdition.SAVE_EVERY_SECONDS + < time() + ): + next_save_at_number_of_frames = self.world.count_frames() + 10000 + last_save_time = time() + self.__save_to_db(self.border_spots, self.dir_and_start_spot) - self.status = "fetching next dir" + self.status = "fetching next dir" - if self.running == False: - print("Crawler loop stopped") - self.crawler_loop_running = False - return + if self.running == False: + print("Crawler loop stopped") + self.crawler_loop_running = False + return self.db.put( WorldDatabase.KEY_STATUS, WorldDatabase.VALUE_STATUS_CRAWLING, ) - self.__save_to_db(border_spots, dir_and_start_spot) + self.__save_to_db(self.border_spots, self.dir_and_start_spot) self.status = "crawl completed, chunk database finalized" self.crawler_loop_running = False print(self.status) @@ -490,3 +461,279 @@ def __save_to_db(self, borderSpots, dirAndStartSpot): """ self.db.save_world_to_database() + + def __pause_for_image_queue(self): + if self.imageServer.get_queue_size() > 500: + while self.imageServer.get_queue_size() > 100 and self.running == True: + self.status = "sleeping" + sleep(0.2) + + def __find_empty_spots(self, current_dir, neededSpots): + + found_enough_empty_spots: bool = False + + found_empty_spots: list = [] + + list_of_unsuccessful_border_spots = [] + + start_spot = None + + self.status = "finding biome" + while not found_enough_empty_spots: + + sleep(0.02) + + if self.running == False: + print("Crawler loop stopped") + self.crawler_loop_running = False + return None, None + + found_empty_spots = [] + + ############################################################################## + # Looking for a start spot + # Start spots are selected out of border spots (these are always empty) + ############################################################################## + + # fallback: fill start_spot no matter what will happen + start_spot = random.choice(tuple(self.border_spots)) + + #print("Trying start spot",start_spot) + + if ( + self.start_spot_search_method + == OrganicCrawlerFirstEdition.START_SPOT_SEARCH_NEAR_PARENT + ): + + # Find a parent dir of the current dir that is already placed in the world. + # This can also be a grandparent dir etc if the parent dir is empty! + basedir_path = Path(current_dir) + + basedir = Path(self.basedir).absolute() + # error: 2 times going to parent dir! + # parentPath = path.parent.absolute() + basedir_parent_path = basedir_path.absolute() + # Going down the parent line + while len(str(basedir_parent_path)) > len(str(basedir)): + basedir_parent_path = basedir_parent_path.parent.absolute() + if str(basedir_parent_path) in self.dir_and_start_spot: + break + + parent_spot = self.dir_and_start_spot[str(basedir_parent_path)] + (parent_spot_x_S, parent_spot_y_S) = parent_spot + + # try x times to find an empty border spot closer and closer to the parent spot + for current_round in range( + 0, 30 + ): # was 300 -> more time needed for that (bad), but 30 is inaccurate + sleep(0.0002) + candidate = random.choice(tuple(self.border_spots)) + #print("border spot candidate",candidate) + if candidate in list_of_unsuccessful_border_spots: + #print("Border candiate",candidate, "already checked and it is not good") + continue + #print("Border candiate",candidate) + (candidate_x_S, candidate_y_S) = candidate + (start_spot_x_S, start_spot_y_S) = start_spot + # changed "or" to "and" + if self.CROSS_SHAPE_GROW: + if abs(candidate_x_S - parent_spot_x_S) < abs( + start_spot_x_S - parent_spot_x_S + ) and abs(candidate_y_S - parent_spot_y_S) < abs( + start_spot_y_S - parent_spot_y_S + ): + start_spot = candidate + elif math.dist((candidate_x_S,candidate_y_S),(parent_spot_x_S,parent_spot_y_S)) < math.dist((start_spot_x_S,start_spot_y_S),(parent_spot_x_S,parent_spot_y_S)): + start_spot = candidate + + elif ( + self.start_spot_search_method + == OrganicCrawlerFirstEdition.START_SPOT_SEARCH_NEAR_WORLD_CENTER + ): + for current_round in range(0, 50): + sleep(0.0002) + candidate = random.choice(tuple(self.border_spots)) + #print("border spot candidate",candidate) + if candidate in list_of_unsuccessful_border_spots: + continue + (candidate_x_S, candidate_y_S) = candidate + (start_spot_x_S, start_spot_y_S) = start_spot + if abs(candidate_x_S) < abs(start_spot_x_S) or abs( + candidate_y_S + ) < abs(start_spot_y_S): + start_spot = candidate + + """ + if True: # TODO insert "far away" code, do this from while to while + # also grow not around center but around start point of parent dir!!!!! + for i in range (0,50): + candidate = random.choice(borderSpots) + (cx,cy) = candidate + (sx,sy) = startSpot + if abs(cx) > abs(sx) or abs(cy) > abs(sy): + startSpot = candidate + + """ + + ############################################################################## + # See if enough empty spots are next to the start spot + ############################################################################## + + stack_with_empty_spots_to_check: list = [start_spot] + # stack always holds neighbouring empty spots to be checked + # if they are not that far away so that spots are grouped together + # and not so far away from another + #print("checking enough empty spots") + while ( + len(stack_with_empty_spots_to_check) > 0 + and not found_enough_empty_spots + ): + #print("size of stack_with_empty_spots_to_check:",len(stack_with_empty_spots_to_check)) + sleep(0.0002) + if self.running == False: + print("Crawler loop stopped") + self.crawler_loop_running = False + return None, None + + ############################################################################## + # Select one of the empty 8 (or less) neighbours of the start_spot + ############################################################################## + #print("select one empty neighbour") + if not self.ORGANIC_GROW: + currentSpot = stack_with_empty_spots_to_check.pop(0) + else: # organic grow + (start_spot_x_S, start_spot_y_S) = start_spot + currentSpot = random.choice(stack_with_empty_spots_to_check) + + # calculate if the next empty spot should "reach out", + # that means, should be farest instead of nearest neighbour + + reach_out = False + + if ( + self.REACH_OUT_ANT_MODE + and neededSpots - len(found_empty_spots) != 0 + ): + reach_out = ( + random.random() + < ((neededSpots - len(found_empty_spots)) / neededSpots) + * self.ANT_CORRIDOR_PROB + ) + + no_of_same_result = 0 + if neededSpots > 100: + max_same_result = 1 + else: + max_same_result = 3 + if reach_out: + max_same_result = 8 + for current_round in range(0, 50): + if no_of_same_result >= max_same_result: + break + candidate = random.choice(stack_with_empty_spots_to_check) + #print("current_round", current_round,"currentspot:",currentSpot,"candidate", candidate) + if candidate == currentSpot: + no_of_same_result += 1 + continue + else: + sleep(0.0001) + (candidate_x_S, candidate_y_S) = candidate + (ux, uy) = currentSpot + # if abs(cx-sx) < abs(ux-sx) or abs(cy-sy) < abs(uy-sy): + + # was both "or", now "and" + if reach_out: + # changed "or" to "and" + if abs(candidate_x_S - start_spot_x_S) > abs( + ux - start_spot_x_S + ) and abs(candidate_y_S - start_spot_y_S) > abs( + uy - start_spot_y_S + ): + currentSpot = candidate + no_of_same_result = 0 + else: + no_of_same_result += 1 + else: + # changed "or" to "and" + if abs(candidate_x_S - start_spot_x_S) < abs( + ux - start_spot_x_S + ) and abs(candidate_y_S - start_spot_y_S) < abs( + uy - start_spot_y_S + ): + no_of_same_result = 0 + currentSpot = candidate + else: + no_of_same_result += 1 + + stack_with_empty_spots_to_check.remove(currentSpot) + + # the current spot is taken... + found_empty_spots.append(currentSpot) + + if len(found_empty_spots) >= neededSpots: + found_enough_empty_spots = True + break + + # ... and now look for its neighbours + (start_spot_x_S, start_spot_y_S) = currentSpot + + ############################################################################## + # Fill stack with all empty neighbours (up to 8 neighbours) + ############################################################################## + #print("add all empty adjacent spots of the neighbour") + for x_S in range(start_spot_x_S - 1, start_spot_x_S + 2): + for y_S in range(start_spot_y_S - 1, start_spot_y_S + 2): + if start_spot_x_S == x_S and start_spot_y_S == y_S: + # not the center spot (not a neighbour) + continue + + if ( + self.NO_DIAGONAL_GROW + and abs(x_S - start_spot_x_S) - abs(y_S - start_spot_y_S) + == 0 + ): + continue + if (x_S, y_S) in found_empty_spots: + continue + if (x_S, y_S) in stack_with_empty_spots_to_check: + continue + if self.world.get_frame_at_S(x_S, y_S) != None: + continue + else: + stack_with_empty_spots_to_check.append((x_S, y_S)) + + if not found_enough_empty_spots: + list_of_unsuccessful_border_spots.append(start_spot) + + ############################################################################## + # correct border spots (add new border spots of the new now filled spots and + # remove the border spots that now contain new frames) + ############################################################################## + + self.status = ( + "correcting border, " + str(len(found_empty_spots)) + " spots to check" + ) + # correct border spots + for currentSpot in found_empty_spots: + (start_spot_x_S, start_spot_y_S) = currentSpot + #sleep(0.00003) + sleep(0.00002) + # go through all 8 neighbours and check changes in "border status" + for x_S in range(start_spot_x_S - 1, start_spot_x_S + 2): + for y_S in range(start_spot_y_S - 1, start_spot_y_S + 2): + if self.world.get_frame_at_S(x_S, y_S) != None or (x_S, y_S) in found_empty_spots: + if (x_S, y_S) in self.border_spots: + #print("removing border spot",(x_S, y_S)) + self.border_spots.remove((x_S, y_S)) # no border anymore + else: + if (x_S, y_S) not in self.border_spots: + self.border_spots.add((x_S, y_S)) # a new border spot + + #for bs in self.border_spots: + # if self.world.get_frame_at_S(bs[0],bs[1]) != None: + # print("BORDER SPOTS CONTAIN NON EMPTY SPOT:",bs) + + #print("found empty spots",found_empty_spots) + #print("border spots", self.border_spots) + + return start_spot, found_empty_spots diff --git a/rugivi/dialogs.py b/rugivi/dialogs.py index c5741db..a8964c5 100755 --- a/rugivi/dialogs.py +++ b/rugivi/dialogs.py @@ -31,7 +31,7 @@ import abc -from tkinter import Tk +from tkinter import Button, Menu, Misc, Text, Tk from tkinter import Label from tkinter import Entry import tkinter.simpledialog @@ -80,6 +80,28 @@ def setResult(self) -> Any: pass +class TkinterRightClick: + def __init__(self, root) -> None: + self.root = root + self._make_textmenu() + + def _make_textmenu(self): + self.the_menu = Menu(self.root, tearoff=0) + self.the_menu.add_command(label="Copy") + + def _show_textmenu(self, event): + e_widget = event.widget + self.the_menu.entryconfigure( + "Copy", command=lambda: e_widget.event_generate("<>") + ) + self.the_menu.tk.call("tk_popup", self.the_menu, event.x_root, event.y_root) + + def bind_textmenu(self): + # to be called AFTER all Entry + Text elements are placed + self.root.bind_class("Entry", "", self._show_textmenu) + self.root.bind_class("Text", "", self._show_textmenu) + + class Dialog_xy_tkwindow(tkinter.simpledialog.Dialog): def body(self, master) -> Any: @@ -126,3 +148,38 @@ def setResult(self) -> Any: if str(result) == "()": result = "" self.result = result + + +class Dialog_copy_clipboard_tkwindow(tkinter.simpledialog.Dialog): + def __init__(self, parent, text, title=None) -> None: + self.text = text + super().__init__(parent, title) + + def body(self, master) -> Any: + + rcm = TkinterRightClick(master) + + self.t1 = Text(master, height=20, width=100) + self.t1.insert(tkinter.END, self.text) + self.t1.grid(row=0, column=0) + + rcm.bind_textmenu() + + return self.t1 # initial focus + + def apply(self) -> None: + pass + + +class Dialog_copy_clipboard(TkinterWrapper): + def __init__(self, title, text) -> None: + self.title = title + self.text = text + super().__init__() + + def createDialogWindow(self) -> Dialog_copy_clipboard_tkwindow: + dialog = Dialog_copy_clipboard_tkwindow(self.root, self.text, self.title) + return dialog + + def setResult(self) -> Any: + self.result = None diff --git a/rugivi/image_service/abstract_cache_filename_generator.py b/rugivi/image_service/abstract_cache_filename_generator.py new file mode 100644 index 0000000..8286bbd --- /dev/null +++ b/rugivi/image_service/abstract_cache_filename_generator.py @@ -0,0 +1,42 @@ +#!/usr/bin/python3 +# +############################################################################################## +# +# RuGiVi - Adult Media Landscape Browser +# +# For updates see git-repo at +# https://github.com/pronopython/rugivi +# +############################################################################################## +# +# Copyright (C) PronoPython +# +# Contact me at pronopython@proton.me +# +# This program is free software: you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the +# Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# +############################################################################################## +# + +import abc + +class AbstractCacheFilenameGenerator: + + + def __init__(self) -> None: + pass + + @abc.abstractmethod + def generate_new_cache_uuid_and_filename(self) -> (str,str): # type: ignore + pass diff --git a/rugivi/image_service/abstract_streamed_media.py b/rugivi/image_service/abstract_streamed_media.py index f748fb9..2b81776 100644 --- a/rugivi/image_service/abstract_streamed_media.py +++ b/rugivi/image_service/abstract_streamed_media.py @@ -63,7 +63,8 @@ class AbstractStreamedMedia: def __init__(self) -> None: self.state = AbstractStreamedMedia.STATE_NEW - self.original_file_path: str = None # type: ignore + self.original_file_path: str = None # type: ignore # always the path to an image (user file or cache image surrogate) + self._extended_dictionary = None self.aspect_ratio: float = 1.0 # width * aspectRatio = height self.width = 0 @@ -135,3 +136,16 @@ def get_ordered_quality(self) -> int: def set_ordered_quality(self, quality) -> None: self._ordered_quality = quality + + def set_extended_attribute(self, key, value): + if self._extended_dictionary == None: + self._extended_dictionary = {} + self._extended_dictionary[key] = value + + def get_extended_attribute(self, key): + if self._extended_dictionary == None: + return None + if key in self._extended_dictionary: + return self._extended_dictionary[key] + else: + return None \ No newline at end of file diff --git a/rugivi/image_service/image_cache.py b/rugivi/image_service/image_cache.py new file mode 100755 index 0000000..1f8a68e --- /dev/null +++ b/rugivi/image_service/image_cache.py @@ -0,0 +1,71 @@ +#!/usr/bin/python3 +# +############################################################################################## +# +# RuGiVi - Adult Media Landscape Browser +# +# For updates see git-repo at +# https://github.com/pronopython/rugivi +# +############################################################################################## +# +# Copyright (C) PronoPython +# +# Contact me at pronopython@proton.me +# +# This program is free software: you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the +# Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# +############################################################################################## +# + +import os +import pathlib +import uuid + +from rugivi.image_service.abstract_cache_filename_generator import AbstractCacheFilenameGenerator + +class ImageCache(AbstractCacheFilenameGenerator): + + + + def __init__(self, cache_base_dir) -> None: + self.cache_base_dir = cache_base_dir + + def generate_path_for_uuid(self,uuid_name): + full_path = self.__get_path_for_uuid(uuid_name) + pathlib.Path(full_path).mkdir(parents=True, exist_ok=True) + + def generate_new_cache_uuid_and_filename(self) -> (str,str): # type: ignore + uuid_name = uuid.uuid4().hex + self.generate_path_for_uuid(uuid_name) + full_path = self.__get_path_for_uuid(uuid_name) + return uuid_name, self.__append_filename_to_path(full_path,uuid_name) + + def __get_path_for_uuid(self, uuid_name) -> str: + path_level_1 = uuid_name[:2] + path_level_2 = uuid_name[2:4] + full_path = os.path.join(self.cache_base_dir, path_level_1, path_level_2) + #full_path_and_filename = os.path.abspath(os.path.join(full_path, uuid_name)) + return str(full_path) + + + def __append_filename_to_path(self, full_path,uuid_name)->str: + return os.path.abspath(os.path.join(full_path, uuid_name + ".jpg")) + + def get_filename_for_uuid(self, uuid_name) -> str: + full_path = self.__get_path_for_uuid(uuid_name) + return self.__append_filename_to_path(full_path,uuid_name) + + def file_exists(self, uuid_name): + return os.path.isfile(self.get_filename_for_uuid(uuid_name)) \ No newline at end of file diff --git a/rugivi/image_service/image_server.py b/rugivi/image_service/image_server.py index 66bb070..5847bca 100755 --- a/rugivi/image_service/image_server.py +++ b/rugivi/image_service/image_server.py @@ -29,6 +29,9 @@ ############################################################################################## # +import hashlib +import math +import os from typing import NoReturn import pygame @@ -37,6 +40,9 @@ from queue import Queue from rugivi.fap_table.fap_table import FapTable +from rugivi.image_service.image_cache import ImageCache +from rugivi.image_service.streamed_mockup import StreamedMockup +from rugivi.image_service.video_still_generator import VideoStillGenerator from .abstract_streamed_media import AbstractStreamedMedia from .streamed_image import StreamedImage @@ -56,6 +62,8 @@ def __init__(self, name, image_server) -> None: self.waiting = True self.image_server: ImageServer = image_server self.image_server_database: ImageServerDatabase = None # type: ignore + self.video_still_generator: VideoStillGenerator = None # type: ignore + self.image_cache: ImageCache = None # type: ignore def conduit_loader_loop(self) -> NoReturn: original_surface: pygame.surface.Surface = None # type: ignore @@ -113,30 +121,70 @@ def conduit_loader_loop(self) -> NoReturn: self.media.state == StreamedImage.STATE_NEW or self.media.state == StreamedImage.STATE_READY_AND_RELOADING ): - # load image from disk - try: - original_surface = pygame.image.load( - self.media.original_file_path - ).convert() + # image is a surrogate for a video file + if self.media.get_extended_attribute("video_still_uuid") != None: + # print("Creating cached file for",self.media.get_extended_attribute("video_file")) + uuid_name = self.media.get_extended_attribute( + "video_still_uuid" + ) - self.media.state = StreamedImage.STATE_LOADED + if not self.image_cache.file_exists(uuid_name): - self.image_server._total_disk_loaded += 1 - except pygame.error as message: - print( - "Conduit " + str(self.name) + " cannot load ", - self.media.original_file_path, - ) - self.media.state = StreamedImage.STATE_ERROR_ON_LOAD - self.waiting = True - except FileNotFoundError as e: - print( - "Conduit " + str(self.name) + " cannot load ", - self.media.original_file_path, - ) - self.media.state = StreamedImage.STATE_ERROR_ON_LOAD - self.waiting = True + # cache miss, create a video still frame + + # The cache name was already generated through the crawler (along with the uuid), + # so that the image already has original_file_path set to the cache file (the + # surrogate file). + # Here it is *again* calculated, to make absolutly sure that cache_filename + # will *always* contain a cache-dir filename. Writing over + # original_file_path can destroy original collection images if somehow + # there is a bug. + cache_filename = self.image_cache.get_filename_for_uuid( + uuid_name + ) + self.image_cache.generate_path_for_uuid( + uuid_name + ) # make sure path exists + video_still_created = ( + self.video_still_generator.create_and_write_still_image( + self.media.get_extended_attribute("video_file"), + self.media.get_extended_attribute("video_position"), + cache_filename, + ) + ) + if video_still_created: + self.media.original_file_path = cache_filename + else: + self.media.state = StreamedImage.STATE_ERROR_ON_LOAD + self.waiting = True + + if self.media.state != StreamedImage.STATE_ERROR_ON_LOAD: + + # load image from disk + + try: + original_surface = pygame.image.load( + self.media.original_file_path + ).convert() + + self.media.state = StreamedImage.STATE_LOADED + + self.image_server._total_disk_loaded += 1 + except pygame.error as message: + print( + "Conduit " + str(self.name) + " cannot load ", + self.media.original_file_path, + ) + self.media.state = StreamedImage.STATE_ERROR_ON_LOAD + self.waiting = True + except FileNotFoundError as e: + print( + "Conduit " + str(self.name) + " cannot load ", + self.media.original_file_path, + ) + self.media.state = StreamedImage.STATE_ERROR_ON_LOAD + self.waiting = True if not self.waiting and self.media.state == StreamedImage.STATE_LOADED: # gather info @@ -159,7 +207,7 @@ def conduit_loader_loop(self) -> NoReturn: h = int(w * self.media.aspect_ratio) self.media._surfaces[q][ StreamedImage.SURFACE_SURFACE - ] = pygame.transform.scale(original_surface, (w, h)) + ] = pygame.transform.smoothscale(original_surface, (w, h)) self.media._surfaces[q][StreamedImage.SURFACE_BYTES] = ( w * h * self.media.bytes_per_pixel ) @@ -191,7 +239,7 @@ def conduit_loader_loop(self) -> NoReturn: and self.media.get_available_quality() >= StreamedImage.QUALITY_THUMB ): - self.image_server_database.add_image_thumb(self.media) # type: ignore + self.image_server_database.add_image_thumb(self.media) # type: ignore self.waiting = True self.media.drawn_view_height = -1 @@ -228,13 +276,25 @@ class ImageServer: HOUSEKEEPING_EVERY_SECONDS = 300 MEMCHECK_EVERY_SECONDS = 120 HOUSEKEEPING_THRESHOLD = [-1, 5000, 150, 20] - HOUSEKEEPING_MAX_MEM_MB_THRESHOLD = 2000 # TODO needs to be implemented, see further down + HOUSEKEEPING_MAX_MEM_MB_THRESHOLD = ( + 2000 # TODO needs to be implemented, see further down + ) QUALITY_PIXEL_SIZE = [32, 128, 1000] # thumb, grid, screen - def __init__(self, number_of_conduits, thumbDbFile) -> None: + def __init__( + self, + number_of_conduits, + thumbDbFile, + cache_base_dir, + number_of_video_conduits=-1, + ) -> None: self.conduits: list[ImageServerConduit] = [] for c in range(0, number_of_conduits): self.conduits.append(ImageServerConduit(c, self)) + if number_of_video_conduits == -1: + self.number_of_video_conduits = number_of_conduits + else: + self.number_of_video_conduits = number_of_video_conduits self.thread_loader = None self.thread_view_fetcher = None self.media_queue: Queue[AbstractStreamedMedia] = Queue() @@ -257,9 +317,17 @@ def __init__(self, number_of_conduits, thumbDbFile) -> None: self.paused = False + self.image_cache = ImageCache(cache_base_dir) + + # self.video_still_generator = VideoStillGenerator(jpg_quality=65, max_dimension=(800,800),remove_letterbox=True) + self.video_still_generator = VideoStillGenerator() + # self.video_still_generator = VideoStillGenerator(jpg_quality=95,remove_letterbox=True) + self.image_server_database = ImageServerDatabase(thumbDbFile) for conduit in self.conduits: conduit.image_server_database = self.image_server_database + conduit.video_still_generator = self.video_still_generator + conduit.image_cache = self.image_cache def start(self) -> None: for conduit in self.conduits: @@ -289,10 +357,20 @@ def server_loader_loop(self) -> None: if not self.paused: if self.media_queue.qsize() > 0: - for conduit in self.conduits: + for conduit_number, conduit in enumerate(self.conduits): if not conduit.is_busy(): media = self.media_queue.get() + if ( + media.get_extended_attribute("video_still_uuid") != None + and not self.image_cache.file_exists( + media.get_extended_attribute("video_still_uuid") + ) + and conduit_number >= self.number_of_video_conduits + ): + self.media_queue.put(media) + continue + # TODO make dependend on ram media._load_quality = media._ordered_quality @@ -352,87 +430,100 @@ def server_view_fetcher_loop(self) -> None: self.fetcher_loop_running = True while self.running: sleep(0.2) + # print("server view fetcher loop") for view in self.views: sleep(1) - # did view move at all? - - view_chunk_x1_C = view.world.convert_S_to_C(view.world_x1_S) - view_chunk_y1_C = view.world.convert_S_to_C(view.world_y1_S) - view_chunk_x2_C = view.world.convert_S_to_C(view.world_x2_S) - view_chunk_y2_C = view.world.convert_S_to_C(view.world_y2_S) - - # based on height, fetch surrounding chunks also - - for x_C in range(view_chunk_x1_C, view_chunk_x2_C + 1): - for y_C in range(view_chunk_y1_C, view_chunk_y2_C + 1): - chunk: Chunk = view.world.get_chunk_at_C(x_C, y_C) + spot_width = math.ceil(World.SPOT_SIZE / view.height) - for x_SL in range(0, World.CHUNK_SIZE): - for y_SL in range(0, World.CHUNK_SIZE): - frame = chunk.get_frame_at_SL(x_SL, y_SL) - - if frame != None: - image = frame.image - - needed_quality = StreamedImage.QUALITY_THUMB + if ( + # view.height / 4 + # <= World.SPOT_SIZE + spot_width + >= ImageServer.QUALITY_PIXEL_SIZE[StreamedImage.QUALITY_GRID] + ): - if ( - view.height - < World.SPOT_SIZE - / ImageServer.QUALITY_PIXEL_SIZE[ - StreamedImage.QUALITY_THUMB - ] - ): + view_chunk_x1_C = view.world.convert_S_to_C(view.world_x1_S) + view_chunk_y1_C = view.world.convert_S_to_C(view.world_y1_S) + view_chunk_x2_C = view.world.convert_S_to_C(view.world_x2_S) + view_chunk_y2_C = view.world.convert_S_to_C(view.world_y2_S) + + # based on height, fetch surrounding chunks also + # print(view_chunk_x1_C,view_chunk_y1_C,"-",view_chunk_x2_C,view_chunk_y2_C) + for x_C in range(view_chunk_x1_C, view_chunk_x2_C + 1): + for y_C in range(view_chunk_y1_C, view_chunk_y2_C + 1): + chunk: Chunk = view.world.get_chunk_at_C(x_C, y_C) + # print("view fetcher: chunk",chunk.x_C,chunk.y_C) + # sleep(0.02) + for x_SL in range(0, World.CHUNK_SIZE): + # sleep(0.002) + for y_SL in range(0, World.CHUNK_SIZE): + # sleep(0.00001) + frame = chunk.get_frame_at_SL(x_SL, y_SL) + + if frame != None: + image = frame.image + + if isinstance(image, StreamedMockup): + continue + + # needed_quality = StreamedImage.QUALITY_THUMB needed_quality = StreamedImage.QUALITY_GRID - x_S = x_SL + chunk.top_spot_x_S - y_S = y_SL + chunk.top_spot_y_S - is_on_screen = ( - (view.world_x1_S <= x_S) - and (x_S <= view.world_x2_S) - and (view.world_y1_S <= y_S) - and (y_S <= view.world_y2_S) - ) - if ( - is_on_screen - and view.height - <= World.SPOT_SIZE - / ( - ImageServer.QUALITY_PIXEL_SIZE[ + + # if ( + # view.height / 4 + # < World.SPOT_SIZE + # / ImageServer.QUALITY_PIXEL_SIZE[ + # StreamedImage.QUALITY_THUMB + # ] + # ): + # needed_quality = StreamedImage.QUALITY_GRID + x_S = x_SL + chunk.top_spot_x_S + y_S = y_SL + chunk.top_spot_y_S + is_on_screen = ( + (view.world_x1_S <= x_S) + and (x_S <= view.world_x2_S) + and (view.world_y1_S <= y_S) + and (y_S <= view.world_y2_S) + ) + if ( + is_on_screen + and spot_width + > ImageServer.QUALITY_PIXEL_SIZE[ StreamedImage.QUALITY_SCREEN ] / 2 - ) - ): - needed_quality = StreamedImage.QUALITY_SCREEN - - if ( - is_on_screen - and view.height - < World.SPOT_SIZE - / ( - ImageServer.QUALITY_PIXEL_SIZE[ + ): + needed_quality = ( StreamedImage.QUALITY_SCREEN - ] - ) - ): - needed_quality = StreamedImage.QUALITY_ORIGINAL - - if ( - image.state == StreamedImage.STATE_READY - and image.get_available_quality() - < needed_quality - ): - image.set_ordered_quality(needed_quality) + ) if ( - image.get_ordered_quality() - > image.get_available_quality() + is_on_screen + and spot_width + > ImageServer.QUALITY_PIXEL_SIZE[ + StreamedImage.QUALITY_SCREEN + ] ): - image.state = ( - StreamedImage.STATE_READY_AND_RELOADING + needed_quality = ( + StreamedImage.QUALITY_ORIGINAL ) - self.media_queue.queue.insert(0, image) + + if ( + image.state == StreamedImage.STATE_READY + and image.get_available_quality() + < needed_quality + ): + image.set_ordered_quality(needed_quality) + + if ( + image.get_ordered_quality() + > image.get_available_quality() + ): + image.state = ( + StreamedImage.STATE_READY_AND_RELOADING + ) + self.media_queue.queue.insert(0, image) # load peek frame = view.world.get_frame_at_S( @@ -441,7 +532,8 @@ def server_view_fetcher_loop(self) -> None: if frame != None: image = frame.image if image != None: - + if isinstance(image, StreamedMockup): + continue if ( image.state == StreamedImage.STATE_READY and image.get_available_quality() @@ -459,7 +551,7 @@ def server_view_fetcher_loop(self) -> None: # load current Fap Table in HiRes for fapTable_view in self.fap_table_views: - current_fapTable : FapTable = fapTable_view.fap_table + current_fapTable: FapTable = fapTable_view.fap_table if current_fapTable != None and current_fapTable.is_displayed: for card in current_fapTable.cards: if card.image != None: @@ -521,6 +613,24 @@ def create_streamed_image( self.media_queue.put(image) return image + def create_streamed_mockup(self, path, color_string="", color=None): + image = StreamedMockup() + image.set_original_file_path(path) + # self.media_queue.put(image) + if color != None: + image.average_color = color + else: + md5 = hashlib.md5(str(color_string).encode()).hexdigest() + r = int(md5[0:2], 16) + g = int(md5[2:4], 16) + b = int(md5[4:6], 16) + image.average_color = (r, g, b) # type: ignore + + image.state = AbstractStreamedMedia.STATE_READY + self._available_quality = AbstractStreamedMedia.QUALITY_COLOR + self.streamed_images.append(image) + return image + def get_number_of_images(self) -> int: return self.streamed_images.length() diff --git a/rugivi/image_service/streamed_mockup.py b/rugivi/image_service/streamed_mockup.py new file mode 100644 index 0000000..20f4b9d --- /dev/null +++ b/rugivi/image_service/streamed_mockup.py @@ -0,0 +1,59 @@ +#!/usr/bin/python3 +# +############################################################################################## +# +# RuGiVi - Adult Media Landscape Browser +# +# For updates see git-repo at +# https://github.com/pronopython/rugivi +# +############################################################################################## +# +# Copyright (C) PronoPython +# +# Contact me at pronopython@proton.me +# +# This program is free software: you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the +# Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# +############################################################################################## +# + +from rugivi.image_service.abstract_streamed_media import AbstractStreamedMedia + + +class StreamedMockup(AbstractStreamedMedia): + def __init__(self): + super().__init__() + self._surface = None + self._scaledSurface = None + self.mode = None + + def get_surface(self, quality=None): + return None + + def get_memory_usage_in_bytes( + self, quality: int = AbstractStreamedMedia.QUALITY_ALL + ) -> int: + return 13 + + def get_number_of_surfaces_loaded_in_ram( + self, quality: int = AbstractStreamedMedia.QUALITY_ALL + ) -> int: + return 0 + + def increment_age(self) -> None: + pass + + def unload_except_thumb(self) -> None: + pass diff --git a/rugivi/image_service/video_still_generator.py b/rugivi/image_service/video_still_generator.py new file mode 100755 index 0000000..d8a3d5b --- /dev/null +++ b/rugivi/image_service/video_still_generator.py @@ -0,0 +1,285 @@ +#!/usr/bin/python3 +# +############################################################################################## +# +# RuGiVi - Adult Media Landscape Browser +# +# For updates see git-repo at +# https://github.com/pronopython/rugivi +# +############################################################################################## +# +# Copyright (C) PronoPython +# +# Contact me at pronopython@proton.me +# +# This program is free software: you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the +# Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# +############################################################################################## +# + +import gc +import math +import os +import time +import cv2 +from threading import Lock + +class VideoStillGenerator: + + def __init__(self, jpg_quality=75, max_dimension=(None, None), remove_letterbox=False) -> None: # type: ignore + self.jpg_quality = jpg_quality # 0-100, default of cv2 = 90 + self.max_dimension = max_dimension + self.remove_letterbox = remove_letterbox + #self._lock = Lock() + #self._lock2 = Lock() + + #self.vcs = {} + #self.vcs_time = {} + #self.vcs_queue = [] + + ''' + def get_vc_for_videofile(self, videofile): + with self._lock2: + if videofile in self.vcs: + self.vcs_queue.remove(videofile) + self.vcs_queue.insert(0,videofile) + self.vcs_time[videofile] = time.time() + print("VC Cache:",len(self.vcs_queue),videofile) + return self.vcs[videofile] + else: + if len(self.vcs_queue) >= 40: + videofile_to_remove = self.vcs_queue.pop() + the_vcs = self.vcs.pop(videofile_to_remove, None) + the_vcs.release() + videofile_to_remove.release() + print("VC destr:",len(self.vcs_queue),videofile_to_remove) + + + #to_be_removed = [] + #for vf, last_access in self.vcs_time: + # if last_access + 60*5 < time.time(): + # to_be_removed.append(vf) + #for vf in to_be_removed: + # the_vcs = self.vcs.pop(vf, None) + # the_vcs.release() + # self.vcs_time.pop(vf,None) + # self.vcs_queue.remove(vf) + # vf.release() + # print("VC destr:",len(self.vcs_queue),vf) + + + new_vcs = cv2.VideoCapture(videofile) + self.vcs_queue.insert(0,videofile) + self.vcs[videofile] = new_vcs + self.vcs_time[videofile] = time.time() + print("VC New :",len(self.vcs_queue),videofile) + return new_vcs + ''' + + + def __image_resize( + self, image, width=None, height=None, inter=cv2.INTER_AREA, upscale=True + ): # -> Any: + dim = None + (h, w) = image.shape[:2] + if width is None and height is None: + return image + elif width is None: + if h <= height: + return image + r = height / float(h) # type: ignore + dim = (int(w * r), height) + elif height is None: + if w <= width: + return image + r = width / float(w) # type: ignore + dim = (width, int(h * r)) + else: + if w <= width and h <= height: + return image + aspect_source = h / w + aspect_dest = height / width + if aspect_source > aspect_dest: + r = height / float(h) + dim = (int(w * r), height) + else: + r = width / float(w) + dim = (width, int(h * r)) + + resized = cv2.resize(image, dim, interpolation=inter) # type: ignore + return resized + + + def analyze_and_get_positions(self, videofile): + + #print("Video:" + videofile) + #with self._lock: + try: + #video = self.get_vc_for_videofile(videofile) + video = cv2.VideoCapture(videofile) + if not video.isOpened(): + #video.release # save some memory + #del video + #gc.collect() + + return [] + + fps = video.get(cv2.CAP_PROP_FPS) + frame_count = int(video.get(cv2.CAP_PROP_FRAME_COUNT)) + duration = frame_count / fps + + #print("Duration:" + str(duration)) + + positions = [] + + + FINAL_DISTANCE = 15 # seconds + REACH_EXPONENT = -0.04 # to lim + OVERALL_OFFSET = 0 + distance = FINAL_DISTANCE - FINAL_DISTANCE * math.exp( + REACH_EXPONENT * (duration - OVERALL_OFFSET) + ) + local_offset = distance / 2 + + current_position = OVERALL_OFFSET + local_offset + position_number = 0 + while current_position < duration: + positions.append(float(current_position)) + current_position += distance + position_number += 1 + + #video.release + #del video + #gc.collect() + return positions + + except cv2.error as e: + #print("cv2.error:", e) + return [] + except Exception as e: + #print("Exception:", e) + return [] + + + + def create_and_write_still_image(self, videofile, position, output_file): + + + video = None + #buffer = None + image = None + #with self._lock: + try: + #print(videofile,"1",position,time.time()) + #video = self.get_vc_for_videofile(videofile) + video = cv2.VideoCapture(videofile, apiPreference=cv2.CAP_FFMPEG) # TODO use ffmpeg faster? how to install? + if not video.isOpened(): + video.release # save some memory + #del video + #gc.collect() + #cv2.destroyAllWindows() + return [] + #print(videofile,"2",position,time.time()) + #fps = video.get(cv2.CAP_PROP_FPS) + #print(videofile,"3a",position,time.time()) + #frame_count = int(video.get(cv2.CAP_PROP_FRAME_COUNT)) + #print(videofile,"3b",position,time.time()) + #duration = frame_count / fps + #print(videofile,"3c",position,time.time()) + video.set(cv2.CAP_PROP_POS_MSEC, position * 1000) # this is slow! (~2-3 sec) + #print(videofile,"3d",position,time.time()) + hasFrames, image = video.read() + #print(videofile,"4",position,time.time()) + if hasFrames: + + image_resized = self.__image_resize( + image, + width=self.max_dimension[0], + height=self.max_dimension[1], + upscale=False, + ) + #del image + image = image_resized + + if self.remove_letterbox: + # remove black borders / letterbox + gray = cv2.cvtColor(image,cv2.COLOR_BGR2GRAY) + _,thresh = cv2.threshold(gray,10,255,cv2.THRESH_BINARY) + x,y,w,h = cv2.boundingRect(thresh) + if w > 10 and h > 10: # prevent cropping of black images + image = image[y:y+h,x:x+w] + + + + # using buffer and imencode to avoid naming the file ".jpg" + #success, buffer = cv2.imencode( + # ".jpg", image, [cv2.IMWRITE_JPEG_QUALITY, self.jpg_quality] + #) + + # never overwrite! + if not os.path.isfile(output_file): + #buffer.tofile(output_file) + + # will fail to write when path does not exist! + image_written = cv2.imwrite(output_file, image, [int(cv2.IMWRITE_JPEG_QUALITY), self.jpg_quality]) + #print("wrote",output_file, success,"from","'"+videofile+"'") + #video.release # save some memory + #image.release + #cv2.destroyAllWindows() + #del video + #del image + #del buffer + #gc.collect() + #print(videofile,"5",position,time.time()) + return image_written + else: + print("Cache file exists! Panic!") + + + #video.release # save some memory + #image.release + #del image + #del video + #del buffer + #gc.collect() + return False + #video.release + return False + except cv2.error as e: + #print("cv2.error:", e) + + #if video != None: + #video.release # save some memory + #if image != None: + # image.release + #del video + #del image + #del buffer + #gc.collect() + + + return False + except Exception as e: + #print("Exception:", e) + #if video != None: + # video.release # save some memory + #if image != None: + # image.release + #del video + #del image + #del buffer + #gc.collect() + return False + diff --git a/rugivi/rugivi.conf b/rugivi/rugivi.conf index 1061a1e..7a12649 100755 --- a/rugivi/rugivi.conf +++ b/rugivi/rugivi.conf @@ -1,10 +1,30 @@ [world] crawlerRootDir=~ +crawlerExcludeDirList= worldDB=~/.local/share/rugivi/chunks.sqlite +crawlvideos=False +crossshapegrow=False +nodiagonalgrow=True +organicgrow=True +reachoutantmode=True [thumbs] thumbDB=~/.local/share/rugivi/thumbs.sqlite +[cache] +cacherootdir=~/.local/share/rugivi/cache + +[videoframe] +jpgquality=65 +maxsizeenabled=False +maxsize=800 +removeletterbox=True + +[videoplayback] +vlcbinary=vlc +vlcenabled=False +vlcseekposition=True + [control] reverseScrollWheelZoom=False pythonexecutable=python3 @@ -21,3 +41,8 @@ statusFontSize=24 [configuration] configured = False + +[debug] +vlcverbose=False +cv2verbose=False +mockupimages=False \ No newline at end of file diff --git a/rugivi/rugivi.py b/rugivi/rugivi.py index f2c7a38..72dcea3 100755 --- a/rugivi/rugivi.py +++ b/rugivi/rugivi.py @@ -30,6 +30,7 @@ # import os +import pathlib import sys @@ -99,6 +100,7 @@ def __init__(self) -> None: self.thumbDbFile = "thumbs.sqlite" self.crawlDir = "." self.tagDir = "" + self.cache_base_dir = "./cache" self.configDir = dir_helper.get_config_dir("RuGiVi") self.configParser = config_file_handler.ConfigFileHandler( @@ -116,6 +118,8 @@ def __init__(self) -> None: ) sys.exit(0) + + self.worldDbFile = self.configParser.get_directory_path( "world", "worldDB", self.worldDbFile ) @@ -125,6 +129,14 @@ def __init__(self) -> None: self.crawlDir = self.configParser.get_directory_path( "world", "crawlerRootDir", self.crawlDir ) + self.cache_base_dir = self.configParser.get_directory_path( + "cache", "cacherootdir", self.cache_base_dir + ) + dir_list_str = self.configParser.get("world", "crawlerExcludeDirList") + if len(dir_list_str) > 0: + self.excludeDirList = dir_list_str.split(sep=";") + else: + self.excludeDirList = [] self.show_info = self.configParser.get_int("control", "showinfo") @@ -141,6 +153,47 @@ def __init__(self) -> None: else: self.zoomDirection = 1 + self.cross_shape_grow = self.configParser.get_boolean("world", "crossshapegrow") + self.no_diagonal_grow = self.configParser.get_boolean("world", "nodiagonalgrow") + self.organic_grow = self.configParser.get_boolean("world", "organicgrow") + self.reach_out_ant_mode = self.configParser.get_boolean("world", "reachoutantmode") + + self.vlc_binary = self.configParser.get("videoplayback", "vlcbinary") + self.video_crawl_enabled = self.configParser.get_boolean("world", "crawlvideos") + self.video_vlc_enabled = self.configParser.get_boolean( + "videoplayback", "vlcenabled" + ) + self.video_vlc_seek_position = self.configParser.get_boolean( + "videoplayback", "vlcseekposition" + ) + self.debug_vlc_verbose = self.configParser.get_boolean( + "debug", "vlcverbose" + ) + self.debug_cv2_verbose = self.configParser.get_boolean( + "debug", "cv2verbose" + ) + self.debug_mockupimages = self.configParser.get_boolean( + "debug", "mockupimages" + ) + + if not self.debug_cv2_verbose: + # prevent open cv error messages when video files make problems + os.environ['OPENCV_LOG_LEVEL'] = 'OFF' + os.environ['OPENCV_FFMPEG_LOGLEVEL'] = "-8" + + self.video_still_generator_jpg_quality = self.configParser.get_int("videoframe", "jpgquality" + ) + self.video_still_generator_resize_enabled = self.configParser.get_boolean( + "videoframe", "maxsizeenabled" + ) + self.video_still_generator_max_dimension = self.configParser.get_int( + "videoframe", "maxsize" + ) + self.video_still_generator_remove_letterbox = self.configParser.get_boolean( + "videoframe", "removeletterbox" + ) + + def parseCommandlineArgs(self) -> None: try: options, args = getopt.getopt( @@ -170,6 +223,16 @@ def open_file(self, path) -> None: else: subprocess.Popen(["xdg-open", path]) + def open_video_with_vlc(self, path, position: float = 0.0): + if position > 2: + position -= 2 + # position_str = "--start-time=" + str(math.floor(position)) + position_str = "--start-time=" + str(position) + if not self.debug_vlc_verbose: + p = subprocess.Popen([self.vlc_binary, position_str, path],shell=False,stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL ) + else: + p = subprocess.Popen([self.vlc_binary, position_str, path]) + def run(self) -> NoReturn: pygame.init() @@ -183,7 +246,12 @@ def run(self) -> NoReturn: self.size, pygame.RESIZABLE | pygame.HWSURFACE | pygame.DOUBLEBUF ) - self.image_server = ImageServer(32, self.thumbDbFile) + self.image_server = ImageServer(32, self.thumbDbFile, self.cache_base_dir, number_of_video_conduits = 8) + # reconfigure video still generatorl + self.image_server.video_still_generator.jpg_quality=self.video_still_generator_jpg_quality + if self.video_still_generator_resize_enabled: + self.image_server.video_still_generator.max_dimension=(self.video_still_generator_max_dimension,self.video_still_generator_max_dimension) + self.image_server.video_still_generator.remove_letterbox=self.video_still_generator_remove_letterbox self.world = World() @@ -195,7 +263,7 @@ def run(self) -> NoReturn: World.SPOT_SIZE / ImageServer.QUALITY_PIXEL_SIZE[StreamedImage.QUALITY_THUMB] ) - view = View(self.world, initial_height) + view : View = View(self.world, initial_height) fap_table_view = FapTableView() @@ -228,7 +296,17 @@ def run(self) -> NoReturn: path = self.crawlDir self.crawler = OrganicCrawlerFirstEdition( - self.world, self.image_server, path, self.worldDbFile + self.world, + self.image_server, + path, + self.worldDbFile, + crawl_videos=self.video_crawl_enabled, + excludeDirList=self.excludeDirList, + mockup_mode=self.debug_mockupimages, + cross_shape_grow = self.cross_shape_grow, + no_diagonal_grow = self.no_diagonal_grow, + organic_grow = self.organic_grow, + reach_out_ant_mode = self.reach_out_ant_mode, ) self.crawler.run() @@ -549,6 +627,46 @@ def run(self) -> NoReturn: redraw = True elif event.key == pygame.K_g: xy_dialog = Dialog_xy() + elif event.key == pygame.K_c: + infotext = "" + if view.selection.get_selected_file() != None: + if view.selection.image != None: + if ( + view.selection.image.get_extended_attribute( + "video_file" + ) + != None + ): + video_filename = ( + view.selection.image.get_extended_attribute( + "video_file" + ) + ) + video_position = float(view.selection.image.get_extended_attribute("video_position")) # type: ignore + infotext = infotext + "Video file:\n" + infotext = infotext + video_filename + "\n" # type: ignore + infotext = infotext + "Parent directory of video file:\n" + infotext = infotext + str(pathlib.Path(video_filename).parent.resolve()) + "\n" # type: ignore + infotext = infotext + "Position (sec):\n" + infotext = infotext + str(video_position) + "\n" + infotext = infotext + "Still image in cache:\n" + infotext = infotext + view.selection.image.original_file_path + "\n" + infotext = infotext + "Parent directory of still image in cache:\n" + infotext = infotext + str(pathlib.Path(view.selection.image.original_file_path).parent.resolve()) + "\n" # type: ignore + else: + infotext = infotext + "Image file:\n" + infotext = infotext + view.selection.get_selected_file() + "\n" + infotext = infotext + "Parent directory of image file:\n" + infotext = infotext + str(pathlib.Path(view.selection.get_selected_file()).parent.resolve()) + "\n" # type: ignore + + infotext = infotext +"Selected Spot:\n" + infotext = infotext + str(view.selection.x_S) + infotext = infotext + "," + infotext = infotext + str(view.selection.y_S) + "\n" + infotext = infotext + "Ordered Quality: " + str(view.selection.image.get_ordered_quality()) + ", Available Quality: " + str(view.selection.image.get_available_quality()) + "\n" + infotext = infotext + "State: " + view.selection.image.state + "\n" + + clipboard_dialog = Dialog_copy_clipboard("Info about selection",infotext) elif event.key == pygame.K_j: last_x_S = math.floor( view.current_center_world_pos_x_P / World.SPOT_SIZE @@ -584,7 +702,31 @@ def run(self) -> NoReturn: elif event.key == pygame.K_n: if view.selection.get_selected_file() != None: - self.open_file(view.selection.get_selected_file()) + if view.selection.image != None: + if ( + view.selection.image.get_extended_attribute( + "video_file" + ) + != None + ): + video_filename = ( + view.selection.image.get_extended_attribute( + "video_file" + ) + ) + video_position = float(view.selection.image.get_extended_attribute("video_position")) # type: ignore + if self.video_vlc_enabled: + if self.video_vlc_seek_position: + self.open_video_with_vlc( + video_filename, position=video_position + ) + else: + self.open_video_with_vlc(video_filename) + else: + self.open_file(video_filename) + else: + self.open_file(view.selection.get_selected_file()) + elif event.key == pygame.K_t: if view.selection.get_selected_file() != None: fapelsystem_dir = dir_helper.get_module_dir("fapelsystem") @@ -772,7 +914,7 @@ def run(self) -> NoReturn: status.writeln("Crawler Dir: " + self.crawler.current_dir) if view.selection.get_selected_file() != None: status.writeln( - "Selected Image: " + view.selection.get_selected_file() # type: ignore + "Selected Media: " + view.selection.get_selected_file() # type: ignore ) else: status.writeln("Selected Image: -none-") @@ -789,20 +931,19 @@ def run(self) -> NoReturn: status.draw(self.display) - - if self.image_server._total_database_loaded + self.image_server._total_disk_loaded < 10: + if ( + self.image_server._total_database_loaded + + self.image_server._total_disk_loaded + < 10 and not self.debug_mockupimages + ): font = pygame.font.SysFont("monospace", 40) - label = font.render( - "starting... please wait!", True, (255, 255, 255) - ) + label = font.render("starting... please wait!", True, (255, 255, 255)) self.display.blit(label, (30, 30)) mp = int(time() % 4) - label2 = font.render((" "*mp) + ".", True, (255, 80, 207)) + label2 = font.render((" " * mp) + ".", True, (255, 80, 207)) self.display.blit(label2, (30, 80)) - #pygame.display.flip() - - + # pygame.display.flip() if self.running == False: self.display.fill((100, 100, 100, 0)) diff --git a/rugivi/rugivi_configurator.py b/rugivi/rugivi_configurator.py index 1c709df..020d4c3 100755 --- a/rugivi/rugivi_configurator.py +++ b/rugivi/rugivi_configurator.py @@ -30,6 +30,8 @@ # import abc +import configparser +import platform from tkinter import Tk from tkinter import Label @@ -49,7 +51,8 @@ from tkinter import Scrollbar import tkinter as tk -#import tkinter.ttk as ttk + +# import tkinter.ttk as ttk from tkinter import filedialog import os from typing import NoReturn @@ -60,9 +63,15 @@ class SelectionSingleItem: __metaclass__ = abc.ABCMeta - - def __init__(self,configParser:config_file_handler.ConfigFileHandler,description,configGroup,configItem) -> None: - self.configParser:config_file_handler.ConfigFileHandler = configParser + + def __init__( + self, + configParser: config_file_handler.ConfigFileHandler, + description, + configGroup, + configItem, + ) -> None: + self.configParser: config_file_handler.ConfigFileHandler = configParser self.description = description self.configGroup = configGroup self.configItem = configItem @@ -75,23 +84,31 @@ def loadInitValue() -> None: pass - class SelectionFolder(SelectionSingleItem): - - def __init__(self,parent,configParser:config_file_handler.ConfigFileHandler,description,configGroup, configItem) -> None: - super().__init__(configParser,description, configGroup, configItem) + def __init__( + self, + parent, + configParser: config_file_handler.ConfigFileHandler, + description, + configGroup, + configItem, + ) -> None: + super().__init__(configParser, description, configGroup, configItem) self.frame = Frame(parent) - self.frame.columnconfigure(1,weight=1) - Label(self.frame, text=description).grid(column=0, row=0,sticky=W) - self.dirText = StringVar(self.frame,self.initValue) - self.dirText.trace_add("write",self.valueChanged) - Entry(self.frame,textvariable=self.dirText).grid(column=1, row=0,sticky=W+E) - Button(self.frame, text="Select", command=self.buttonClicked).grid(column=2,row=0,sticky=E) + self.frame.columnconfigure(1, weight=1) + Label(self.frame, text=description).grid(column=0, row=0, sticky=W) + self.dirText = StringVar(self.frame, self.initValue) + self.dirText.trace_add("write", self.valueChanged) + Entry(self.frame, textvariable=self.dirText).grid(column=1, row=0, sticky=W + E) + Button(self.frame, text="Select", command=self.buttonClicked).grid( + column=2, row=0, sticky=E + ) def loadInitValue(self) -> None: - self.initValue = self.configParser.get_directory_path(self.configGroup,self.configItem, "") - + self.initValue = self.configParser.get_directory_path( + self.configGroup, self.configItem, "" + ) def buttonClicked(self) -> None: selected_folder = filedialog.askdirectory() @@ -99,124 +116,144 @@ def buttonClicked(self) -> None: selected_folder = os.path.abspath(selected_folder) self.dirText.set(selected_folder) - - def valueChanged(self,*args) -> None: - self.configParser.change_config()[self.configGroup][self.configItem] = self.dirText.get() - - + def valueChanged(self, *args) -> None: + self.configParser.change_config()[self.configGroup][ + self.configItem + ] = self.dirText.get() def getFrame(self) -> Frame: return self.frame - class SelectionFile(SelectionSingleItem): - - def __init__(self,parent,configParser,description,configGroup, configItem,filetypes) -> None: - super().__init__(configParser,description, configGroup, configItem) + def __init__( + self, parent, configParser, description, configGroup, configItem, filetypes + ) -> None: + super().__init__(configParser, description, configGroup, configItem) self.frame = Frame(parent) - self.frame.columnconfigure(1,weight=1) + self.frame.columnconfigure(1, weight=1) self.filetypes = filetypes - Label(self.frame, text=description).grid(column=0, row=0,ipadx=5,sticky=W) - self.dirText = StringVar(self.frame,self.initValue) - self.dirText.trace_add("write",self.valueChanged) - Entry(self.frame,textvariable=self.dirText).grid(column=1, row=0,ipadx=5,sticky=W+E) - Button(self.frame, text="Select", command=self.buttonClicked).grid(column=2,row=0,ipadx=5,sticky=E) + Label(self.frame, text=description).grid(column=0, row=0, ipadx=5, sticky=W) + self.dirText = StringVar(self.frame, self.initValue) + self.dirText.trace_add("write", self.valueChanged) + Entry(self.frame, textvariable=self.dirText).grid( + column=1, row=0, ipadx=5, sticky=W + E + ) + Button(self.frame, text="Select", command=self.buttonClicked).grid( + column=2, row=0, ipadx=5, sticky=E + ) def loadInitValue(self) -> None: - self.initValue = self.configParser.get_directory_path(self.configGroup,self.configItem, "") + self.initValue = self.configParser.get_directory_path( + self.configGroup, self.configItem, "" + ) def buttonClicked(self) -> None: - filename = filedialog.askopenfilename(filetypes = self.filetypes) + filename = filedialog.askopenfilename(filetypes=self.filetypes) self.dirText.set(filename) self.configParser.change_config()[self.configGroup][self.configItem] = filename - def valueChanged(self,*args) -> None: - self.configParser.change_config()[self.configGroup][self.configItem] = self.dirText.get() + def valueChanged(self, *args) -> None: + self.configParser.change_config()[self.configGroup][ + self.configItem + ] = self.dirText.get() def getFrame(self) -> Frame: return self.frame - class SelectionBoolean(SelectionSingleItem): - - def __init__(self,parent,configParser,description,configGroup, configItem) -> None: - super().__init__(configParser,description, configGroup, configItem) + def __init__( + self, parent, configParser, description, configGroup, configItem + ) -> None: + super().__init__(configParser, description, configGroup, configItem) self.frame = Frame(parent) - self.frame.columnconfigure(1,weight=1) - Label(self.frame, text=description).grid(column=0, row=0,ipadx=5,sticky=W) - self.button = Button(self.frame, text=str(self.initValue), command=self.buttonClicked) - self.button.grid(column=2,row=0,ipadx=5,sticky=E) + self.frame.columnconfigure(1, weight=1) + Label(self.frame, text=description).grid(column=0, row=0, ipadx=5, sticky=W) + self.button = Button( + self.frame, text=str(self.initValue), command=self.buttonClicked + ) + self.button.grid(column=2, row=0, ipadx=5, sticky=E) self.value = self.initValue def loadInitValue(self) -> None: - self.initValue = self.configParser.get_boolean(self.configGroup,self.configItem) + self.initValue = self.configParser.get_boolean( + self.configGroup, self.configItem + ) def buttonClicked(self) -> None: self.value = not self.value self.button.configure(text=str(self.value)) - self.configParser.change_config()[self.configGroup][self.configItem] = str(self.value) - + self.configParser.change_config()[self.configGroup][self.configItem] = str( + self.value + ) def getFrame(self) -> Frame: return self.frame - class SelectionInteger(SelectionSingleItem): - - def __init__(self,parent,configParser,description,configGroup, configItem) -> None: - super().__init__(configParser,description, configGroup, configItem) + def __init__( + self, parent, configParser, description, configGroup, configItem + ) -> None: + super().__init__(configParser, description, configGroup, configItem) self.frame = Frame(parent) - self.frame.columnconfigure(1,weight=1) - Label(self.frame, text=description).grid(column=0, row=0,ipadx=5,sticky=W) - self.dirText = StringVar(self.frame,str(self.initValue)) - self.dirText.trace_add("write",self.valueChanged) - Entry(self.frame,textvariable=self.dirText).grid(column=1, row=0,ipadx=5,sticky=W+E) + self.frame.columnconfigure(1, weight=1) + Label(self.frame, text=description).grid(column=0, row=0, ipadx=5, sticky=W) + self.dirText = StringVar(self.frame, str(self.initValue)) + self.dirText.trace_add("write", self.valueChanged) + Entry(self.frame, textvariable=self.dirText).grid( + column=1, row=0, ipadx=5, sticky=W + E + ) def loadInitValue(self) -> None: - self.initValue = self.configParser.get_int(self.configGroup,self.configItem) + self.initValue = self.configParser.get_int(self.configGroup, self.configItem) - def valueChanged(self,*args) -> None: - self.configParser.change_config()[self.configGroup][self.configItem] = self.dirText.get() + def valueChanged(self, *args) -> None: + self.configParser.change_config()[self.configGroup][ + self.configItem + ] = self.dirText.get() def getFrame(self) -> Frame: return self.frame - class SelectionMultiItem(SelectionSingleItem): - - def __init__(self,parent,configParser,description,configGroup,configItem) -> None: - super().__init__(configParser,description, configGroup, configItem) + def __init__( + self, parent, configParser, description, configGroup, configItem + ) -> None: + super().__init__(configParser, description, configGroup, configItem) self.frame = Frame(parent) - self.frame.columnconfigure(1,weight=1) + self.frame.columnconfigure(1, weight=1) - Label(self.frame, text=description).grid(column=0, row=0,rowspan=2,sticky=W) + Label(self.frame, text=description).grid(column=0, row=0, rowspan=2, sticky=W) listboxframe = Frame(self.frame) - self.listbox = Listbox(listboxframe,height=5) - self.listbox.pack(side=LEFT,fill=BOTH,expand=YES) - #self.listbox.grid(column=1,row=0, rowspan=2,sticky=W+E) + self.listbox = Listbox(listboxframe, height=5) + self.listbox.pack(side=LEFT, fill=BOTH, expand=YES) + # self.listbox.grid(column=1,row=0, rowspan=2,sticky=W+E) scrollbar = Scrollbar(listboxframe) - scrollbar.pack(side=RIGHT,fill=BOTH) - self.listbox.config(yscrollcommand = scrollbar.set) - scrollbar.config(command = self.listbox.yview) + scrollbar.pack(side=RIGHT, fill=BOTH) + self.listbox.config(yscrollcommand=scrollbar.set) + scrollbar.config(command=self.listbox.yview) - listboxframe.grid(column=1,row=0, rowspan=2,sticky=W+E) - - Button(self.frame, text="+", command=self.buttonClickedAdd).grid(column=2,row=0,sticky=E) - Button(self.frame, text="-", command=self.buttonClickedRemove).grid(column=2,row=1,sticky=E) + listboxframe.grid(column=1, row=0, rowspan=2, sticky=W + E) + + Button(self.frame, text="+", command=self.buttonClickedAdd).grid( + column=2, row=0, sticky=E + ) + Button(self.frame, text="-", command=self.buttonClickedRemove).grid( + column=2, row=1, sticky=E + ) for item in self.items: - self.listbox.insert(tk.END,item) + self.listbox.insert(tk.END, item) def loadInitValue(self) -> None: if len(self.configParser.items(self.configGroup)) > 0: @@ -236,89 +273,166 @@ def getFrame(self) -> Frame: def valueChanged(self) -> None: self.configParser.change_config()[self.configGroup].clear() - for pos, item in enumerate(self.listbox.get(0,tk.END)): - self.configParser.change_config()[self.configGroup][self.configItem+str(pos)] = item - - + for pos, item in enumerate(self.listbox.get(0, tk.END)): + self.configParser.change_config()[self.configGroup][ + self.configItem + str(pos) + ] = item class SelectionMultiFolder(SelectionMultiItem): - - - def __init__(self,parent,configParser,description,configGroup,configItem) -> None: - super().__init__(parent,configParser,description, configGroup, configItem) - + def __init__( + self, parent, configParser, description, configGroup, configItem + ) -> None: + super().__init__(parent, configParser, description, configGroup, configItem) def buttonClickedAdd(self) -> None: selected_folder = filedialog.askdirectory() if selected_folder != (): selected_folder = os.path.abspath(selected_folder) - self.listbox.insert(tk.END,selected_folder) + self.listbox.insert(tk.END, selected_folder) self.valueChanged() -class ConfigApp: - +class ConfigApp: def __init__(self) -> None: print("RuGiVi Configurator") - self.configDir = dir_helper.get_config_dir("RuGiVi") - self.configParser : config_file_handler.ConfigFileHandler = config_file_handler.ConfigFileHandler(os.path.join(self.configDir,"rugivi.conf")) - + self.configParser: config_file_handler.ConfigFileHandler = ( + config_file_handler.ConfigFileHandler( + os.path.join(self.configDir, "rugivi.conf") + ) + ) + + self.config_migrated = False + self.migrate_old_conf(self.configParser) self.root = Tk() - self.root.geometry("800x480") + self.root.geometry("800x650") self.root.title("RuGiVi Configurator") - frm = Frame(self.root) - frm.grid(sticky=N+S+W+E) - frm.columnconfigure(0,weight =1) - - Label(frm, text="Crawler settings").grid(column=0, row=0,sticky=W) - - #ttk.Button(frm, text="Quit", command=root.destroy).grid(column=1, row=0) - itfo = SelectionFolder(frm,self.configParser,"Crawler root directory","world","crawlerRootDir") - itfo.getFrame().grid(column=0,row=1,ipadx=10,padx=10,sticky=W+E) - - - filetypes = (("SQLite files", "*.sqlite"),("All files", "*.*") ) - itfo = SelectionFile(frm,self.configParser,"Crawler World DB File","world","worldDB",filetypes) - itfo.getFrame().grid(column=0,row=2,ipadx=10,padx=10,sticky=W+E) - - Label(frm, text="You must use a new World DB File or delete the old one when changing root directory", fg="red").grid(column=0, row=3,sticky=W) - - Label(frm, text="Thumb Database settings").grid(column=0, row=4,sticky=W) - - itfo = SelectionFile(frm,self.configParser,"Thumb DB File","thumbs","thumbDB",filetypes) - itfo.getFrame().grid(column=0,row=5,ipadx=10,padx=10,sticky=W+E) - - - Label(frm, text="GUI settings").grid(column=0, row=6,sticky=W) - - itfo = SelectionBoolean(frm,self.configParser,"Reverse Scroll Wheel Zoom","control","reversescrollwheelzoom") - itfo.getFrame().grid(column=0,row=7,ipadx=10,padx=10,sticky=W+E) - - itfo = SelectionInteger(frm,self.configParser,"Status font size","fonts","statusfontsize") - itfo.getFrame().grid(column=0,row=8,ipadx=10,padx=10,sticky=W+E) - - - Label(frm, text="FapTables").grid(column=0, row=9,sticky=W) - - itfo = SelectionMultiFolder(frm,self.configParser,"FapTable parent dirs","fapTableParentDirs","dir") - itfo.getFrame().grid(column=0,row=10,ipadx=10,padx=9,sticky=W+E) + frm.grid(sticky=N + S + W + E) + frm.columnconfigure(0, weight=1) + + row = 0 + + Label(frm, text="Crawler settings").grid(column=0, row=row, sticky=W) + row += 1 + + # ttk.Button(frm, text="Quit", command=root.destroy).grid(column=1, row=0) + itfo = SelectionFolder( + frm, self.configParser, "Crawler root directory", "world", "crawlerRootDir" + ) + itfo.getFrame().grid(column=0, row=row, ipadx=10, padx=10, sticky=W + E) + row += 1 + + filetypes = (("SQLite files", "*.sqlite"), ("All files", "*.*")) + itfo = SelectionFile( + frm, + self.configParser, + "Crawler World DB File", + "world", + "worldDB", + filetypes, + ) + itfo.getFrame().grid(column=0, row=row, ipadx=10, padx=10, sticky=W + E) + row += 1 + + Label( + frm, + text="You must use a new World DB File or delete the old one when changing root directory", + fg="red", + ).grid(column=0, row=row, sticky=W) + row += 1 + + Label(frm, text="Thumb Database settings").grid(column=0, row=row, sticky=W) + row += 1 + + itfo = SelectionFile( + frm, self.configParser, "Thumb DB File", "thumbs", "thumbDB", filetypes + ) + itfo.getFrame().grid(column=0, row=row, ipadx=10, padx=10, sticky=W + E) + row += 1 + + Label(frm, text="Video settings").grid(column=0, row=row, sticky=W) + row += 1 + + itfo = SelectionBoolean( + frm, self.configParser, "Enable video crawling", "world", "crawlvideos" + ) + itfo.getFrame().grid(column=0, row=row, ipadx=10, padx=10, sticky=W + E) + row += 1 + + itfo = SelectionFolder( + frm, + self.configParser, + "Video still cache directory", + "cache", + "cacherootdir", + ) + itfo.getFrame().grid(column=0, row=row, ipadx=10, padx=10, sticky=W + E) + row += 1 + + itfo = SelectionBoolean( + frm, self.configParser, "Play video with VLC", "videoplayback", "vlcenabled" + ) + itfo.getFrame().grid(column=0, row=row, ipadx=10, padx=10, sticky=W + E) + row += 1 + + Label(frm, text="GUI settings").grid(column=0, row=row, sticky=W) + row += 1 + + itfo = SelectionBoolean( + frm, + self.configParser, + "Reverse Scroll Wheel Zoom", + "control", + "reversescrollwheelzoom", + ) + itfo.getFrame().grid(column=0, row=row, ipadx=10, padx=10, sticky=W + E) + row += 1 + + itfo = SelectionInteger( + frm, self.configParser, "Status font size", "fonts", "statusfontsize" + ) + itfo.getFrame().grid(column=0, row=row, ipadx=10, padx=10, sticky=W + E) + row += 1 + + Label(frm, text="FapTables").grid(column=0, row=row, sticky=W) + row += 1 + + itfo = SelectionMultiFolder( + frm, self.configParser, "FapTable parent dirs", "fapTableParentDirs", "dir" + ) + itfo.getFrame().grid(column=0, row=row, ipadx=10, padx=9, sticky=W + E) + row += 1 + + itfo = SelectionMultiFolder( + frm, self.configParser, "FapTable single dirs", "fapTableSingleDirs", "dir" + ) + itfo.getFrame().grid(column=0, row=row, ipadx=10, padx=10, sticky=W + E) + row += 1 + + if self.config_migrated: + Label( + frm, + text="Config migrated & new items added. Make sure to save the config!", + fg="red", + ).grid(column=0, row=row, sticky=E) + row += 1 - itfo = SelectionMultiFolder(frm,self.configParser,"FapTable single dirs","fapTableSingleDirs","dir") - itfo.getFrame().grid(column=0,row=11,ipadx=10,padx=10,sticky=W+E) bframe = Frame(frm) - Button(bframe, text="Save and exit", command=self.actionSaveAndExit).grid(column=0,row=0,ipadx=5,sticky=E) - Button(bframe, text="Exit without save", command=self.actionExitWithoutSave).grid(column=1,row=0,ipadx=5,sticky=E) - bframe.grid(column=0,row=12,sticky=E) - - + Button(bframe, text="Save and exit", command=self.actionSaveAndExit).grid( + column=0, row=0, ipadx=5, sticky=E + ) + Button( + bframe, text="Exit without save", command=self.actionExitWithoutSave + ).grid(column=1, row=0, ipadx=5, sticky=E) + bframe.grid(column=0, row=row, sticky=E) + row += 1 frm.pack(expand=True, fill=BOTH) @@ -334,7 +448,70 @@ def actionExitWithoutSave(self) -> NoReturn: exit() + def is_windows(self) -> bool: + if platform.system() == "Windows": + return True + else: + return False + + def _migrate_check_and_add_entry(self, configParser, group, key, initvalue): + entry_present = True + section_missing = False + try: + configParser.config_parser.get(group, key) + except configparser.NoOptionError: + entry_present = False + except configparser.NoSectionError: + entry_present = False + section_missing = True + if not entry_present: + print("Migrate old config to new one: adding [",group,"]",key,"=",initvalue) + if section_missing: + self.configParser.change_config().add_section(group) + self.configParser.change_config()[group][key] = initvalue + self.config_migrated = True + + def migrate_old_conf(self, configParser): + + # older => v0.4.0 + + self._migrate_check_and_add_entry(configParser, "world","crawlerExcludeDirList","") + self._migrate_check_and_add_entry(configParser, "world","crawlvideos","False") + + self._migrate_check_and_add_entry(configParser, "world","crossshapegrow","False") + self._migrate_check_and_add_entry(configParser, "world","nodiagonalgrow","True") + self._migrate_check_and_add_entry(configParser, "world","organicgrow","True") + self._migrate_check_and_add_entry(configParser, "world","reachoutantmode","True") + + if self.is_windows(): + self._migrate_check_and_add_entry(configParser, "cache","cacherootdir","~\\AppData\\Roaming\\RuGiVi\\cache") + else: + self._migrate_check_and_add_entry(configParser, "cache","cacherootdir","~/.local/share/rugivi/cache") + + self._migrate_check_and_add_entry(configParser, "videoframe","jpgquality","65") + self._migrate_check_and_add_entry(configParser, "videoframe","maxsizeenabled","False") + self._migrate_check_and_add_entry(configParser, "videoframe","maxsize","800") + self._migrate_check_and_add_entry(configParser, "videoframe","removeletterbox","True") + + if self.is_windows(): + self._migrate_check_and_add_entry(configParser, "videoplayback","vlcbinary","C:/Program Files/VideoLAN/VLC/vlc.exe") + else: + self._migrate_check_and_add_entry(configParser, "videoplayback","vlcbinary","vlc") + + self._migrate_check_and_add_entry(configParser, "videoplayback","vlcenabled","False") + self._migrate_check_and_add_entry(configParser, "videoplayback","vlcseekposition","True") + + if self.is_windows(): + self._migrate_check_and_add_entry(configParser, "control","pythonexecutable","python") + else: + self._migrate_check_and_add_entry(configParser, "control","pythonexecutable","python3") + + self._migrate_check_and_add_entry(configParser, "debug","vlcverbose","False") + self._migrate_check_and_add_entry(configParser, "debug","cv2verbose","False") + self._migrate_check_and_add_entry(configParser, "debug","mockupimages","False") + + + def main() -> None: app = ConfigApp() app.run() - diff --git a/rugivi/rugivi_image_cache_maintenance.py b/rugivi/rugivi_image_cache_maintenance.py new file mode 100644 index 0000000..324c33d --- /dev/null +++ b/rugivi/rugivi_image_cache_maintenance.py @@ -0,0 +1,227 @@ +#!/usr/bin/python3 +# +############################################################################################## +# +# RuGiVi - Adult Media Landscape Browser +# +# For updates see git-repo at +# https://github.com/pronopython/rugivi +# +############################################################################################## +# +# Copyright (C) PronoPython +# +# Contact me at pronopython@proton.me +# +# This program is free software: you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the +# Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# +############################################################################################## +# + + +import time +import os + +# Import pygame, hide welcome message because it messes with +# status output +os.environ["PYGAME_HIDE_SUPPORT_PROMPT"] = "hide" + +import pathlib +import sys +from typing import NoReturn +from sqlitedict import SqliteDict + +from rugivi.world_database_service.world_database import ChunkSaveObject +from rugivi import config_file_handler as config_file_handler +from rugivi import dir_helper as dir_helper + +class ImageCacheMaintenance(): + + + + def __init__(self) -> None: + self.db : SqliteDict= None # type: ignore + self.worldDbFile = "chunks.sqlite" + self.cache_base_dir = "./cache" + self.cache_files_total_size = 0 + + self.worldDbFiles = [] + + self.cache_files = [] + + self.orphant_files = [] + + + + + def run(self): + self.configDir = dir_helper.get_config_dir("RuGiVi") + self.configParser = config_file_handler.ConfigFileHandler( + os.path.join(self.configDir, "rugivi.conf") + ) + + self.configured = self.configParser.get_boolean("configuration", "configured") + + if not self.configured: + print("Not configured") + print("Please configure RuGiVi and save the settings with the RuGiVi Configurator before running it") + sys.exit(0) + + + self.worldDbFile = self.configParser.get_directory_path( + "world", "worldDB", self.worldDbFile + ) + + self.cache_base_dir = self.configParser.get_directory_path( + "cache", "cacherootdir", self.cache_base_dir + ) + + + + + + + + self.open_world_db(self.worldDbFile) + + self.generate_world_db_file_list() + self.generate_cache_file_list() + self.calculate_cache_size() + + print("World DB files:",len(self.worldDbFiles)) + print("Cache files :",len(self.cache_files)) + print("Cache size:",int(self.cache_files_total_size/(1024*1024)),"MB") + + self.generate_orphant_file_list() + + print("Orphant files :",len(self.orphant_files)) + + if len(self.orphant_files) > 0: + if len(sys.argv) > 1 and sys.argv[1] == "-c": + self.delete_orphant_files() + else: + print("NOTHING WAS CHANGED YET!") + print("RUN THIS TOOL AGAIN with '-c' to move orphant files and clean up the cache!") + else: + print("No orphant files. Nothing to clean up. Cache is healthy.") + + def open_world_db(self,crawler_db_file): + self.db = SqliteDict(crawler_db_file) + + + + def generate_world_db_file_list(self): + + print("Reading Chunks from Database:",end="",flush=True) + + for item in self.db.items(): # type: ignore + if str(item[0]).startswith("(") and str(item[0]).endswith(")"): + tupel = item[1] + if tupel == None: + continue + + print(".",end="",flush=True) + chunk_save_object = ChunkSaveObject() + chunk_save_object.from_tupel(tupel) + + for filepath in chunk_save_object.spots_filepath_list: + if len(filepath) > 0: + #print(filepath) + self.worldDbFiles.append(filepath) + print("done") + + + + def generate_cache_file_list(self): + print("Scanning files of cache:",end="",flush=True) + every = 5000 + for root, d_names, f_names in os.walk(self.cache_base_dir): + if root.find("trash_") > -1: + print("") + print("Warning: There is already a trash folder present:",root) + print("") + continue + for f_name in f_names: + if f_name.endswith(".jpg"): + file = os.path.join(root, f_name) + #print(file) + self.cache_files.append(file) + every -= 1 + if every <= 0: + print(".",end="",flush=True) + every = 5000 + print("done") + + def calculate_cache_size(self): + print("Gathering information on files in cache:",end="",flush=True) + every = 5000 + self.cache_files_total_size = 0 + for file in self.cache_files: + self.cache_files_total_size += os.path.getsize(file) + every -= 1 + if every <= 0: + print(".",end="",flush=True) + every = 5000 + print("done") + + + def generate_orphant_file_list(self): + print("Finding orphant cache files:",end="",flush=True) + every = 5000 + for file in self.cache_files: + if file not in self.worldDbFiles: + self.orphant_files.append(file) + every -= 1 + if every <= 0: + print(".",end="",flush=True) + every = 5000 + print("done") + + + def delete_orphant_files(self): + + foldername = "trash_" + time.strftime("%Y%m%d_%H%M%S") + + trashbase = os.path.abspath(os.path.join(self.cache_base_dir, foldername)) + + pathlib.Path(trashbase).mkdir(parents=True, exist_ok=True) + + moved_files = 0 + + for file in self.orphant_files: + # Precaution: Check if file path lies within cache dir + if os.path.exists(file) and file.startswith(self.cache_base_dir): + filename = (os.path.basename(file)) + newname = os.path.join(trashbase, filename) + # Precaution: never overwrite + if os.path.exists(newname): + print("Panic: Destination file",newname,"already exists! Abort") + sys.exit() + os.rename(file,newname) + moved_files += 1 + else: + print("Panic: File",file,"not found or not in cache! Abort") + sys.exit() + + print(moved_files, "file(s) moved to trash:",trashbase) + print("Check these files if they are ok to be deleted and then delete the 'trash_*' folder manually.") + +if __name__ == "__main__": + app = ImageCacheMaintenance() + app.run() + + +def main() -> NoReturn: # type: ignore + app = ImageCacheMaintenance() + app.run() diff --git a/rugivi/rugivi_windows.conf b/rugivi/rugivi_windows.conf index c297ab7..21d0da4 100755 --- a/rugivi/rugivi_windows.conf +++ b/rugivi/rugivi_windows.conf @@ -1,10 +1,30 @@ [world] crawlerRootDir=~\Pictures\ +crawlerExcludeDirList= worldDB=~\AppData\Roaming\RuGiVi\chunks.sqlite +crawlvideos=False +crossshapegrow=False +nodiagonalgrow=True +organicgrow=True +reachoutantmode=True [thumbs] thumbDB=~\AppData\Roaming\RuGiVi\thumbs.sqlite +[cache] +cacherootdir=~\AppData\Roaming\RuGiVi\cache + +[videoframe] +jpgquality=65 +maxsizeenabled=False +maxsize=800 +removeletterbox=True + +[videoplayback] +vlcbinary=C:/Program Files/VideoLAN/VLC/vlc.exe +vlcenabled=False +vlcseekposition=True + [control] reverseScrollWheelZoom=False pythonexecutable=python @@ -21,3 +41,8 @@ statusFontSize=24 [configuration] configured = False + +[debug] +vlcverbose=False +cv2verbose=False +mockupimages=False \ No newline at end of file diff --git a/rugivi/selection.py b/rugivi/selection.py index 2d54b75..af42d14 100755 --- a/rugivi/selection.py +++ b/rugivi/selection.py @@ -61,9 +61,14 @@ def update_selected_spot(self) -> None: self.frame = self.world.get_frame_at_S(self.x_S, self.y_S) if self.frame != None: self.image = self.frame.image # type: ignore + else: + self.image = None # type: ignore def get_selected_file(self) -> Optional[str]: if self.image != None: - return self.image.original_file_path + if self.image.get_extended_attribute("video_file") != None: + return self.image.get_extended_attribute("video_file") + else: + return self.image.original_file_path else: return None diff --git a/rugivi/view.py b/rugivi/view.py index aa5e72d..64e01d0 100755 --- a/rugivi/view.py +++ b/rugivi/view.py @@ -272,43 +272,51 @@ def draw_view( spot_height_P, ), ) - if spot_width_P < 16: + if spot_width_P < 5: pass elif spot_width_P <= 32: surface :Surface = image.get_surface(AbstractStreamedMedia.QUALITY_THUMB) # type: ignore - scaled_image_surface = pygame.transform.scale( - surface, - (image_drawing_width_P, image_drawing_height_P), - ) - display.blit( - scaled_image_surface, - ( - current_screen_x_PL + image_offset_x_P, - current_screen_y_PL + image_offset_y_P, - ), - ) - self.performance_images_drawn = ( - self.performance_images_drawn + 1 - ) + if surface != None: + scaled_image_surface = pygame.transform.smoothscale( + surface, + (image_drawing_width_P, image_drawing_height_P), + ) + display.blit( + scaled_image_surface, + ( + current_screen_x_PL + image_offset_x_P, + current_screen_y_PL + image_offset_y_P, + ), + ) + self.performance_images_drawn = ( + self.performance_images_drawn + 1 + ) else: surface :Surface = image.get_surface() # type: ignore - - scaled_image_surface = pygame.transform.scale( - surface, - (image_drawing_width_P, image_drawing_height_P), - ) - display.blit( - scaled_image_surface, - ( - current_screen_x_PL + image_offset_x_P, - current_screen_y_PL + image_offset_y_P, - ), - ) - self.performance_images_drawn = ( - self.performance_images_drawn + 1 - ) + if surface != None: + # only smoothscale to small destination size, big dest. sizes take too long + if spot_width_P < 256: # TODO hard coded pixels + scaled_image_surface = pygame.transform.smoothscale( + surface, + (image_drawing_width_P, image_drawing_height_P), + ) + else: + scaled_image_surface = pygame.transform.scale( + surface, + (image_drawing_width_P, image_drawing_height_P), + ) + display.blit( + scaled_image_surface, + ( + current_screen_x_PL + image_offset_x_P, + current_screen_y_PL + image_offset_y_P, + ), + ) + self.performance_images_drawn = ( + self.performance_images_drawn + 1 + ) elapsed_time = int((time_ns() - start_time) / 1000000) @@ -431,19 +439,20 @@ def draw_view( ) surface: pygame.surface.Surface = image.get_surface() # type: ignore - scaled_image_surface = pygame.transform.scale( - surface, (image_drawing_width_P, image_drawing_height_P) - ) - display.blit( - scaled_image_surface, - ( - current_screen_x_PL + spot_width_P + (thickness * 2), - current_screen_y_PL, - ), - ) - self.performance_images_drawn = ( - self.performance_images_drawn + 1 - ) + if surface != None: + scaled_image_surface = pygame.transform.smoothscale( + surface, (image_drawing_width_P, image_drawing_height_P) + ) + display.blit( + scaled_image_surface, + ( + current_screen_x_PL + spot_width_P + (thickness * 2), + current_screen_y_PL, + ), + ) + self.performance_images_drawn = ( + self.performance_images_drawn + 1 + ) pygame.draw.rect( display, diff --git a/rugivi/world_database_service/world_database.py b/rugivi/world_database_service/world_database.py index 32dde25..765b7ea 100755 --- a/rugivi/world_database_service/world_database.py +++ b/rugivi/world_database_service/world_database.py @@ -45,7 +45,7 @@ class ChunkSaveObject: def __init__(self) -> None: - self.spots_filepath_list : list = [] + self.spots_filepath_list: list = [] """ a list of spots filepath flattened from matrix row by row """ self.number_of_empty_spots = 0 @@ -54,6 +54,8 @@ def __init__(self) -> None: self.top_spot_x_S = 0 self.top_spot_y_S = 0 + self.extended_dictionaries: list = [] + def __save_numpy_spots(self, numpy_spots_matrix) -> None: self.spots_filepath_list = [] for y_SL in range(0, World.CHUNK_SIZE): @@ -65,15 +67,47 @@ def __save_numpy_spots(self, numpy_spots_matrix) -> None: continue self.spots_filepath_list.append("") + def __save_extended_dictionaries(self, numpy_spots_matrix) -> None: + self.extended_dictionaries = [] + has_extended_dictionary = False + for y_SL in range(0, World.CHUNK_SIZE): + for x_SL in range(0, World.CHUNK_SIZE): + frame: Frame = numpy_spots_matrix[x_SL, y_SL] + if frame != None: + if frame.image != None: + if ( + frame.image._extended_dictionary != None + and len(frame.image._extended_dictionary) > 0 + ): + has_extended_dictionary = True + self.extended_dictionaries.append( + frame.image._extended_dictionary + ) + continue + self.extended_dictionaries.append({}) + if not has_extended_dictionary: + self.extended_dictionaries = [] + def to_tuple(self) -> tuple: - return ( - self.x_C, - self.y_C, - self.top_spot_x_S, - self.top_spot_y_S, - self.number_of_empty_spots, - self.spots_filepath_list, - ) + if len(self.extended_dictionaries) > 0: + return ( + self.x_C, + self.y_C, + self.top_spot_x_S, + self.top_spot_y_S, + self.number_of_empty_spots, + self.spots_filepath_list, + self.extended_dictionaries, + ) + else: + return ( + self.x_C, + self.y_C, + self.top_spot_x_S, + self.top_spot_y_S, + self.number_of_empty_spots, + self.spots_filepath_list, + ) def from_tupel(self, tupel) -> None: self.spots_filepath_list = tupel[5] @@ -82,10 +116,13 @@ def from_tupel(self, tupel) -> None: self.y_C = tupel[1] self.top_spot_x_S = tupel[2] self.top_spot_y_S = tupel[3] + if len(tupel) > 6: # an extended dictionary was saved + self.extended_dictionaries = tupel[6] def from_chunk(self, chunk: Chunk) -> None: - self.spots_filepath_list = None # type: ignore + self.spots_filepath_list = None # type: ignore self.__save_numpy_spots(chunk._spots_matrix) + self.__save_extended_dictionaries(chunk._spots_matrix) self.number_of_empty_spots = chunk._number_of_empty_spots self.x_C = chunk.x_C self.y_C = chunk.y_C @@ -144,11 +181,21 @@ def get_chunk_at_C(self, x_C, y_C) -> Chunk: newChunk = Chunk(self.world, x_C, y_C) for y_SL in range(0, World.CHUNK_SIZE): for x_SL in range(0, World.CHUNK_SIZE): - original_file_path = chunk_save_object.spots_filepath_list[(y_SL * World.CHUNK_SIZE) + x_SL] + original_file_path = chunk_save_object.spots_filepath_list[ + (y_SL * World.CHUNK_SIZE) + x_SL + ] if original_file_path != "": streamed_image = self.image_server.create_streamed_image( original_file_path, StreamedImage.QUALITY_THUMB ) + + if len(chunk_save_object.extended_dictionaries) > 0: + streamed_image._extended_dictionary = ( + chunk_save_object.extended_dictionaries[ + (y_SL * World.CHUNK_SIZE) + x_SL + ] + ) + newChunk.set_frame_at_SL(Frame(streamed_image), x_SL, y_SL) return newChunk @@ -160,4 +207,4 @@ def save_world_to_database(self) -> None: chunk_save_object = ChunkSaveObject() chunk_save_object.from_chunk(chunk) self.db[str((chunk.x_C, chunk.y_C))] = chunk_save_object.to_tuple() - self.commit_db() \ No newline at end of file + self.commit_db() diff --git a/setup.py b/setup.py index 2b23be2..42f485c 100755 --- a/setup.py +++ b/setup.py @@ -29,7 +29,7 @@ from setuptools import setup setup(name='rugivi', - version='0.3.1-alpha', + version='0.4.0-alpha', description='RuGiVi - Adult Media Landscape Browser', url='https://github.com/pronopython/rugivi', author='pronopython', @@ -39,14 +39,15 @@ package_data={'rugivi':['*']}, include_package_data=True, zip_safe=False, - install_requires=['pygame','psutil','numpy','sqlitedict','pyshortcuts','Pillow'], + install_requires=['pygame','psutil','numpy','sqlitedict','pyshortcuts','Pillow','opencv-python-headless'], entry_points={ 'gui_scripts': [ 'rugivi_configurator=rugivi.rugivi_configurator:main' ], 'console_scripts': [ 'rugivi_printModuleDir=rugivi.print_module_dir:printModuleDir', - 'rugivi=rugivi.rugivi:main' + 'rugivi=rugivi.rugivi:main', + 'rugivi_image_cache_maintenance=rugivi.rugivi_image_cache_maintenance:main' ] } )