EonaCat.Sync - Synchronization Service

Synchronizing files and databases between computers with advanced features including cloud storage, compression, intelligent merging, incremental backups, and REST API support.

Features

• File Synchronization

  • Change detection and incremental sync
  • SHA256 file integrity verification
  • Chunked file transfers for large files
  • Configurable exclusion patterns
  • Timestamp preservation
  • Recursive directory sync
  • Retry logic with exponential backoff
  • ** File compression (GZip, Deflate, Brotli)**

• Database Synchronization

  • Entity change tracking and detection
  • Batch operations for performance
  • Transaction support
  • Multi-database support (SQL Server, SQLite, etc.)
  • Change logging and tracking
  • Index rebuild capabilities
  • Foreign key constraint control
  • ** Entity Framework Core support (optional)**

• Cloud Storage Integration

  • Azure Blob Storage support
  • AWS S3 integration
  • Generic cloud provider interface
  • File upload/download with metadata
  • Shared access link generation
  • Documentation: Cloud providers are available as optional extensions

• Advanced Merging

  • Line-based 3-way merge for text files
  • XML/config file intelligent merging
  • Automatic conflict detection and resolution
  • Multiple merge strategies

• Backup & Versioning

  • Full and incremental backup support
  • Automatic change detection
  • Version history tracking
  • Point-in-time restore capability
  • Retention policies

• Conflict Resolution

  • Last-write-wins strategy
  • Source/Target wins strategies
  • Manual resolution support
  • Merge capabilities
  • Customizable conflict handling

• Monitoring & Logging

  • Performance metrics collection
  • System health checks
  • Operation profiling
  • Structured logging
  • Docker container support

• Session Management

  • Tracked sync sessions
  • Progress notifications
  • Error handling and reporting
  • Change history tracking

• REST API

  • Async sync operations
  • Cloud storage management
  • Backup operations
  • System health monitoring

Installation

Add to your project:

dotnet add package EonaCat.Sync

Optional packages:

dotnet add package Azure.Storage.Blobs          # For Azure support
dotnet add package AWSSDK.S3                    # For AWS S3 support
dotnet add package Microsoft.EntityFrameworkCore # For EF Core support

Quick Start

Basic File Synchronization

using EonaCat.Sync;
using EonaCat.Sync.Services;

// Create sync service
var syncService = SyncServiceFactory.CreateSyncService();

// Sync files
var result = await syncService.SyncFilesAsync(
    @"C:\SourceFolder",
    @"C:\TargetFolder");

Console.WriteLine($"Changes applied: {result.ChangesApplied}");

File Compression

using EonaCat.Sync.Compression;

var compressionService = new CompressionService();

// Compress a file
await compressionService.CompressFileAsync(
    @"C:\large-file.dat", 
    @"C:\compressed.gz",
    CompressionAlgorithm.GZip);

// Get compression ratio
var ratio = await compressionService.GetCompressionRatioAsync(
    @"C:\large-file.dat",
    @"C:\compressed.gz");

Console.WriteLine($"Compression ratio: {ratio:F2}%");

Advanced Merging

using EonaCat.Sync.Merge;

var mergeService = new MergeService();

// Perform 3-way merge
var result = await mergeService.MergeFilesAsync(
    @"C:\base.txt",
    @"C:\source.txt", 
    @"C:\target.txt");

if (result.Conflicts.Count > 0)
{
    Console.WriteLine($"Merge conflicts found: {result.Conflicts.Count}");
}

Console.WriteLine(result.MergedContent);

Incremental Backups

using EonaCat.Sync.Backup;

var backupService = new IncrementalBackupService();

// Create full backup
var fullBackup = await backupService.CreateFullBackupAsync(
    @"C:\ImportantData",
    @"D:\Backups");

// Later: Create incremental backup
var incrementalBackup = await backupService.CreateIncrementalBackupAsync(
    @"C:\ImportantData",
    @"D:\Backups");

Console.WriteLine($"Changed files: {incrementalBackup.ChangedFiles.Count}");

// List all versions
var versions = await backupService.ListVersionsAsync(@"D:\Backups");

// Restore a specific version
await backupService.RestoreVersionAsync(
    @"D:\Backups",
    fullBackup.VersionId,
    @"C:\Restored");

Monitoring & Metrics

using EonaCat.Sync.Monitoring;

var metricsCollector = new SyncMetricsCollector();
var logger = new ConsoleStructuredLogger();

// Start operation
var metric = metricsCollector.StartOperation("FileSyncOperation");

// Do work...
var result = await syncService.SyncFilesAsync(source, target);

metric.FilesProcessed = result.ChangesApplied;
metric.ErrorCount = result.Errors.Count;
metricsCollector.EndOperation(metric);

// Get metrics
var aggregateMetrics = metricsCollector.GetAggregateMetrics();
logger.LogInformation($"Success Rate: {aggregateMetrics.SuccessRate:F2}%");

Advanced Configuration

Enable Compression During Sync

var fileSyncOptions = new FileSyncOptions
{
    CompressTransfers = true,
    CompressionThreshold = 1048576, // 1MB
    ChunkSize = 5242880 // 5MB
};

var syncService = SyncServiceFactory.CreateFileSyncService(fileSyncOptions);

Database Sync with EF Core

// When Microsoft.EntityFrameworkCore is installed
using EonaCat.Sync.EntityFramework;

var sourceContext = new MyDbContext(sourceConnection);
var targetContext = new MyDbContext(targetConnection);

var result = await EFCoreSynchronizer.SyncDbContextAsync(
    sourceContext, 
    targetContext);

Custom Cloud Provider

public class CustomCloudProvider : CloudProviderBase
{
    public override CloudProviderType ProviderType => CloudProviderType.Generic;

    // Implement abstract methods
}

// Register with factory
CloudProviderFactory.RegisterProvider(provider);

Custom Merge Strategy

public class CustomMergeStrategy : IMergeStrategy
{
    public string Name => "Custom Merge";
    public bool CanHandle(string filePath) => filePath.EndsWith(".custom");

    public async Task<MergeResult> MergeAsync(string baseFile, string sourceFile, string targetFile)
    {
        // Custom merge logic
    }
}

// Use with service
var mergeService = new MergeService();
mergeService.RegisterStrategy(new CustomMergeStrategy());

Installation

Add to your project:

dotnet add package EonaCat.Sync

Quick Start

Basic File Synchronization

using EonaCat.Sync;
using EonaCat.Sync.Services;

// Create sync service
var syncService = SyncServiceFactory.CreateSyncService();

// Sync files
var result = await syncService.SyncFilesAsync(
	@"C:\SourceFolder",
	@"C:\TargetFolder");

Console.WriteLine($"Changes applied: {result.ChangesApplied}");
Console.WriteLine($"Success: {result.Success}");

Basic Database Synchronization

var options = new DatabaseSyncOptions
{
	UseTransactions = true,
	DatabaseBatchSize = 1000
};

var syncService = SyncServiceFactory.CreateDatabaseSyncService(options);

var result = await syncService.SyncDatabaseAsync(
	"Server=SourceServer;Database=SourceDB;Integrated Security=true;",
	"Server=TargetServer;Database=TargetDB;Integrated Security=true;",
	new[] { "Users", "Products", "Orders" });

Console.WriteLine($"Database sync complete. Success: {result.Success}");

Full Synchronization (Files + Database)

var options = new SyncOptions
{
	ConflictResolutionStrategy = "LastWriteWins",
	DeleteMissingFilesInTarget = true,
	ValidateAfterSync = true
};

var syncService = SyncServiceFactory.CreateSyncService();

var result = await syncService.SyncFullAsync(
	@"C:\System1\Data",
	@"C:\System2\Data",
	"Server=System1;Database=AppDB;",
	"Server=System2;Database=AppDB;",
	new[] { "Users", "Settings", "Documents" },
	options);

Bidirectional Synchronization

var options = new SyncOptions
{
	BidirectionalSync = true,
	ConflictResolutionStrategy = "LastWriteWins"
};

var result = await syncService.SyncBidirectionalAsync(
	@"C:\Location1",
	@"C:\Location2",
	options);

Configuration

FileSyncOptions

var fileSyncOptions = new FileSyncOptions
{
	// Conflict resolution: "LastWriteWins", "SourceWins", "TargetWins", "Manual"
	ConflictResolutionStrategy = "LastWriteWins",

	// Delete files from target if they don't exist in source
	DeleteMissingFilesInTarget = false,

	// Validate data after sync
	ValidateAfterSync = true,

	// Maximum retry attempts for failed operations
	MaxRetries = 3,

	// Timeout in seconds
	TimeoutSeconds = 300,

	// File patterns to exclude (semicolon-separated)
	ExcludeFilePatterns = "*.tmp;*.temp;*.bak;*.lock",

	// Buffer size for file operations (bytes)
	BufferSizeBytes = 65536,

	// Preserve original file timestamps
	PreserveTimestamps = true,

	// Verify file integrity using SHA256
	VerifyFileIntegrity = true,

	// Compress files during transfer
	CompressTransfers = false,

	// Minimum file size for compression (bytes)
	CompressionThreshold = 1048576,

	// Chunk size for large file transfers (bytes)
	ChunkSize = 5242880 // 5MB
};

var syncService = SyncServiceFactory.CreateFileSyncService(fileSyncOptions);

DatabaseSyncOptions

var dbSyncOptions = new DatabaseSyncOptions
{
	// Use transactions for each batch
	UseTransactions = true,

	// Batch size for database operations
	DatabaseBatchSize = 1000,

	// Comma-separated list of entities to sync
	EntitiesToSync = "Users,Products,Orders",

	// Clear target database before sync
	ClearTargetBeforeSync = false,

	// Rebuild indexes after sync
	RebuildIndexesAfterSync = false,

	// Bypass foreign key constraints during sync
	BypassForeignKeyConstraints = false,

	// Conflict resolution strategy
	ConflictResolutionStrategy = "LastWriteWins"
};

var syncService = SyncServiceFactory.CreateDatabaseSyncService(dbSyncOptions);

Advanced Usage

Custom Notifications

public class CustomSyncNotifier : ISyncNotifier
{
	public void NotifyProgress(string message, int percentage)
	{
		Console.WriteLine($"Progress: {percentage}% - {message}");
	}

	public void NotifyError(string errorMessage)
	{
		Console.WriteLine($"Error: {errorMessage}");
	}

	public void NotifyCompletion(SyncResult result)
	{
		Console.WriteLine($"Sync complete. Changes: {result.ChangesApplied}");
	}

	public void NotifyConflict(SyncConflict conflict)
	{
		Console.WriteLine($"Conflict detected in {conflict.EntityIdentifier}");
	}
}

var notifier = new CustomSyncNotifier();
var syncService = SyncServiceFactory.CreateSyncService(notifier);

Session Management

// Start a sync session
var session = await syncService.StartSyncSessionAsync("System1", "System2");
Console.WriteLine($"Session started: {session.SessionId}");

// Perform sync operations
var result = await syncService.SyncFilesAsync(source, target);

// Complete session
await syncService.CompleteSyncSessionAsync(session.SessionId);

// Get session status
var sessionStatus = await syncService.GetSyncSessionAsync(session.SessionId);

Change Tracking

// Create a file synchronizer
var fileSynchronizer = new FileSynchronizer();

// Detect changes since a specific time
var changes = await fileSynchronizer.DetectFileChangesAsync(
	@"C:\MyFolder", 
	DateTime.UtcNow.AddHours(-1));

foreach (var change in changes)
{
	Console.WriteLine($"Change: {change.SourceId} - {change.Type}");
}

File Metadata

var fileSynchronizer = new FileSynchronizer();

// Get file metadata
var metadata = await fileSynchronizer.GetFileMetadataAsync(@"C:\MyFile.dat");

Console.WriteLine($"File: {metadata.FilePath}");
Console.WriteLine($"Size: {metadata.FileSize} bytes");
Console.WriteLine($"Hash: {metadata.FileHash}");
Console.WriteLine($"Modified: {metadata.LastModified}");

Custom Conflict Resolution

public class CustomConflictResolver : IConflictResolver
{
	public async Task<object> ResolveConflictAsync(SyncConflict conflict)
	{
		// Custom logic to resolve conflicts
		if (conflict.SourceLastModified > conflict.TargetLastModified)
			return conflict.SourceValue;
		else
			return conflict.TargetValue;
	}

	public string GetResolutionStrategy(ChangeType changeType)
	{
		return "CustomLogic";
	}
}

Error Handling

try
{
	var result = await syncService.SyncFilesAsync(source, target);

	if (!result.Success)
	{
		foreach (var error in result.Errors)
		{
			Console.WriteLine($"Error: {error}");
		}
	}
}
catch (Exception ex)
{
	Console.WriteLine($"Sync failed: {ex.Message}");
}

Best Practices

1. Always validate connection strings before starting database sync
2. Use transactions for database operations to ensure consistency
3. Enable file integrity verification for critical files
4. Set appropriate timeouts based on network conditions
5. Monitor progress for long-running syncs
6. Test conflict resolution strategies before production use
7. Backup target systems before destructive sync operations
8. Use bidirectional sync cautiously to avoid infinite loops