• 4 Posts
  • 13 Comments
Joined 2 months ago
Cake day: April 23rd, 2024
  • cyberluna@programming.devOPtoLinux@lemmy.ml•hello! i made a versatile file syncing tool in C++ it's called lunas
    4·
    2 months ago

    quick overview of the syncing algo

    • a simple overview

    1- list all input directories content first

    2- a table is made as a map, the rows are the files/dirs, the column numbers are the input directories in a specific order, what’s inside each cell of each row, is the mtime, modification time, of that file in different input directory if -diff option was used

    3- and it loops through the table to check what is a SRC AND has a newer mtime than another DEST, if so, it removes the dest and resyncs it

    without the option -diff what gets filled in the cells of each row are true/false of whether that file exist in this input directory or not. and it just sync based on file name, and which SRC dir it detects first that would be the src of what is missing in the DEST

    the ordering of the input directories in the table’s columns, are as the user input them, but the local ones has a priority, they get listed first in these columns

    so, the conflicts with the -diff option is resolve based on newer/older mtime and src/dest and the newer src updates the older dest. without it just based on file names which is more random, as explained

    • a more techincal overview

    1- list all input directories content first

    2- in each input directory listing, add the content to vector A ‘all_content’, the type to vector B ‘types’, the input_dir_number of the file to vector C ‘track_existence’, and if option -diff is enabled, push the mtime to vector D ‘track_existence_mtime’

    i’m gonna explain with the option -diff first which let’s it check for mtime, modification time, difference between files and sync based on that

    -diff option enables ‘–attributes mtime’ by default which makes sure if re-run it only resync the files if they were changed

    ‘–mtime off’ can be used as mentioned in the man page to avoid syncing the mtime

    3- the all_content gets sorted using quick sort, and the other vectors follow its sorting order

    4- a 2D vector/a table is made as an existence map, the rows are the files/dirs, the column numbers are the input directories in a specific order, what’s inside each cell of each row, is the mtime of that file in different input directory

    5- the track_existence vector should be cleared after that

    6- and it loops through the 2D vector to check what is a SRC AND has a newer mtime than another DEST, if so, it removes the dest and resyncs it

    without the option -diff the vector D ‘track_existence_mtime’ don’t get filled, and what gets filled in the cells of each row are true/false of whether that file exist in this input directory or not. and it just sync based on file name, and which SRC dir it detects first that would be the src of what is missing in the DEST

    the ordering of the input directories in the existence map columns, are as the user input them, but the local ones has a priority they get listed first in these columns

    so, the conflicts with the -diff option is resolved based on newer/older mtime and src/dest and the newer src updates the older dest. without it just based on file names which is more random, as explained

    in the copying, or filesystem in general, functions, C++ provide methods to check if certain operations were faulty or not, i use these methods combined with checking the return of remote reading/writing if successful or not, if a write or read to a buffer produced an error, syncing to that file stops and it goes to the next file. that file stays named as file.ls.part

    lunas doesn’t have checksum option, not yet at least i might add it later. so if that is a problem for someone, they could avoid using lunas for now

    but i made a seperate program that checks recursively checksums of many input directories which i usually use when needed to check if lunas is working correctly or not

    btw just to be clear, as mentioned in the license, –> This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License version 3 for more details. -> https://www.gnu.org/licenses/gpl-3.0.en.html

  • cyberluna@programming.devOPtoLinux@lemmy.ml•hello! i made a versatile file syncing tool in C++ it's called lunas
    10·
    1 month ago

    i’m not familiar with rclone. I’m kinda familiar with rsync, but not much so excuse me if i got something wrong,

    3 main things that are in lunas but not in rsync

    • multi directory syncing with src and/or dest for each input directory

    u can sync as many directories, locally and remotely, ig as your ram can handle

    src input directories -> files are not synced to them, but they sync to other input directories

    dest input directories -> files ARE synced to them, but they DON’T sync to other input directories

    src+dest input directories -> they do both, files are synced to them, and from them to other input directories

    lunas -p dir1 -p dir2 -p dir3 -p dir4 -p dir5 -r user@ip:dir6

    all of these dirs should be the same at the end, since -p is src and dest at the same time, same for -r

    • config file -> which many presets could be defined and used very conveniently " lunas -c preset_name"

    path : ~/.config/lunas/lunas.conf

    global{

    resume = on  # resume file syncing if it was interrupted

   progress = on # show progress bar for file syncing

   update = on # check for file mtimes and resync them based on mtime difference. Newer to old

   mkdir = on # make input directories if not there

   compression = on # in remote transfer, zlib is used

}

cool{

   p = /path/to/dir

   r = user@ip:/path/to/dir

   verbose = on

   attributes = atime

}

    command > lunas -c cool

    using this command lunas would load what’s in the preset named “cool”, and tries to sync the paths inside them and enable the options. And there is >the special preset called “global” which can always run while using -c preset, or normally using the cli “lunas -p dir -p dir2” this basically lets the user >defined their default options that make sense for them instead of rewriting them in the cli each time

    btw lunas can load multiple presets at once

    lunas -c cool -c other_preset -c etc

    • sync removal -> to sync remove files from other directories

    in case someone wanted to delete a directory or file and don’t want them to be synced back to other backup directories for instance, or want them >removed, they could use sync removal. By default it acts as excluding, unless -cr Y is used, then it should remove them from other input directories, >if found

    or --exclude can be used to exclude files without removing them, i’m thinking of maybe adding a .lunasignore file similar to .gitignore to make this >extra convenient

    lunas doesn’t have as many options as rsync, it still has many tho, u could take a quick look at both of them to compare if you want. But i’m still developing lunas and i wanna make it even more cooler, it’s been like over 6 months since i started it, so hopefully more cool stuff come in the future