EonaCat.Sync/README.md

# EonaCat.Sync - Synchronization Service

Synchronizing files and databases between computers with advanced features including cloud storage, compression, intelligent merging, incremental backups, and REST API support.

## Features

- **File Synchronization**
  - Change detection and incremental sync
  - SHA256 file integrity verification
  - Chunked file transfers for large files
  - Configurable exclusion patterns
  - Timestamp preservation
  - Recursive directory sync
  - Retry logic with exponential backoff
  - ** File compression (GZip, Deflate, Brotli)**

- **Database Synchronization**
  - Entity change tracking and detection
  - Batch operations for performance
  - Transaction support
  - Multi-database support (SQL Server, SQLite, etc.)
  - Change logging and tracking
  - Index rebuild capabilities
  - Foreign key constraint control
  - ** Entity Framework Core support (optional)**

- **Cloud Storage Integration**
  - Azure Blob Storage support
  - AWS S3 integration
  - Generic cloud provider interface
  - File upload/download with metadata
  - Shared access link generation
  - **Documentation: Cloud providers are available as optional extensions**

- **Advanced Merging**
  - Line-based 3-way merge for text files
  - XML/config file intelligent merging
  - Automatic conflict detection and resolution
  - Multiple merge strategies

- **Backup & Versioning**
  - Full and incremental backup support
  - Automatic change detection
  - Version history tracking
  - Point-in-time restore capability
  - Retention policies

- **Conflict Resolution**
  - Last-write-wins strategy
  - Source/Target wins strategies
  - Manual resolution support
  - Merge capabilities
  - Customizable conflict handling

- **Monitoring & Logging**
  - Performance metrics collection
  - System health checks
  - Operation profiling
  - Structured logging
  - Docker container support

- **Session Management**
  - Tracked sync sessions
  - Progress notifications
  - Error handling and reporting
  - Change history tracking

- **REST API**
  - Async sync operations
  - Cloud storage management
  - Backup operations
  - System health monitoring

## Installation

Add to your project:
```bash
dotnet add package EonaCat.Sync
```

Optional packages:
```bash
dotnet add package Azure.Storage.Blobs          # For Azure support
dotnet add package AWSSDK.S3                    # For AWS S3 support
dotnet add package Microsoft.EntityFrameworkCore # For EF Core support
```

## Quick Start

### Basic File Synchronization

```csharp
using EonaCat.Sync;
using EonaCat.Sync.Services;

// Create sync service
var syncService = SyncServiceFactory.CreateSyncService();

// Sync files
var result = await syncService.SyncFilesAsync(
    @"C:\SourceFolder",
    @"C:\TargetFolder");

Console.WriteLine($"Changes applied: {result.ChangesApplied}");
```

### File Compression

```csharp
using EonaCat.Sync.Compression;

var compressionService = new CompressionService();

// Compress a file
await compressionService.CompressFileAsync(
    @"C:\large-file.dat", 
    @"C:\compressed.gz",
    CompressionAlgorithm.GZip);

// Get compression ratio
var ratio = await compressionService.GetCompressionRatioAsync(
    @"C:\large-file.dat",
    @"C:\compressed.gz");

Console.WriteLine($"Compression ratio: {ratio:F2}%");
```

### Advanced Merging

```csharp
using EonaCat.Sync.Merge;

var mergeService = new MergeService();

// Perform 3-way merge
var result = await mergeService.MergeFilesAsync(
    @"C:\base.txt",
    @"C:\source.txt", 
    @"C:\target.txt");

if (result.Conflicts.Count > 0)
{
    Console.WriteLine($"Merge conflicts found: {result.Conflicts.Count}");
}

Console.WriteLine(result.MergedContent);
```

### Incremental Backups

```csharp
using EonaCat.Sync.Backup;

var backupService = new IncrementalBackupService();

// Create full backup
var fullBackup = await backupService.CreateFullBackupAsync(
    @"C:\ImportantData",
    @"D:\Backups");

// Later: Create incremental backup
var incrementalBackup = await backupService.CreateIncrementalBackupAsync(
    @"C:\ImportantData",
    @"D:\Backups");

Console.WriteLine($"Changed files: {incrementalBackup.ChangedFiles.Count}");

// List all versions
var versions = await backupService.ListVersionsAsync(@"D:\Backups");

// Restore a specific version
await backupService.RestoreVersionAsync(
    @"D:\Backups",
    fullBackup.VersionId,
    @"C:\Restored");
```

### Monitoring & Metrics

```csharp
using EonaCat.Sync.Monitoring;

var metricsCollector = new SyncMetricsCollector();
var logger = new ConsoleStructuredLogger();

// Start operation
var metric = metricsCollector.StartOperation("FileSyncOperation");

// Do work...
var result = await syncService.SyncFilesAsync(source, target);

metric.FilesProcessed = result.ChangesApplied;
metric.ErrorCount = result.Errors.Count;
metricsCollector.EndOperation(metric);

// Get metrics
var aggregateMetrics = metricsCollector.GetAggregateMetrics();
logger.LogInformation($"Success Rate: {aggregateMetrics.SuccessRate:F2}%");
```

## Advanced Configuration

### Enable Compression During Sync

```csharp
var fileSyncOptions = new FileSyncOptions
{
    CompressTransfers = true,
    CompressionThreshold = 1048576, // 1MB
    ChunkSize = 5242880 // 5MB
};

var syncService = SyncServiceFactory.CreateFileSyncService(fileSyncOptions);
```

### Database Sync with EF Core

```csharp
// When Microsoft.EntityFrameworkCore is installed
using EonaCat.Sync.EntityFramework;

var sourceContext = new MyDbContext(sourceConnection);
var targetContext = new MyDbContext(targetConnection);

var result = await EFCoreSynchronizer.SyncDbContextAsync(
    sourceContext, 
    targetContext);
```

### Custom Cloud Provider

```csharp
public class CustomCloudProvider : CloudProviderBase
{
    public override CloudProviderType ProviderType => CloudProviderType.Generic;

    // Implement abstract methods
}

// Register with factory
CloudProviderFactory.RegisterProvider(provider);
```

### Custom Merge Strategy

```csharp
public class CustomMergeStrategy : IMergeStrategy
{
    public string Name => "Custom Merge";
    public bool CanHandle(string filePath) => filePath.EndsWith(".custom");

    public async Task<MergeResult> MergeAsync(string baseFile, string sourceFile, string targetFile)
    {
        // Custom merge logic
    }
}

// Use with service
var mergeService = new MergeService();
mergeService.RegisterStrategy(new CustomMergeStrategy());
```

## Installation

Add to your project:
```bash
dotnet add package EonaCat.Sync
```

## Quick Start

### Basic File Synchronization

```csharp
using EonaCat.Sync;
using EonaCat.Sync.Services;

// Create sync service
var syncService = SyncServiceFactory.CreateSyncService();

// Sync files
var result = await syncService.SyncFilesAsync(
	@"C:\SourceFolder",
	@"C:\TargetFolder");

Console.WriteLine($"Changes applied: {result.ChangesApplied}");
Console.WriteLine($"Success: {result.Success}");
```

### Basic Database Synchronization

```csharp
var options = new DatabaseSyncOptions
{
	UseTransactions = true,
	DatabaseBatchSize = 1000
};

var syncService = SyncServiceFactory.CreateDatabaseSyncService(options);

var result = await syncService.SyncDatabaseAsync(
	"Server=SourceServer;Database=SourceDB;Integrated Security=true;",
	"Server=TargetServer;Database=TargetDB;Integrated Security=true;",
	new[] { "Users", "Products", "Orders" });

Console.WriteLine($"Database sync complete. Success: {result.Success}");
```

### Full Synchronization (Files + Database)

```csharp
var options = new SyncOptions
{
	ConflictResolutionStrategy = "LastWriteWins",
	DeleteMissingFilesInTarget = true,
	ValidateAfterSync = true
};

var syncService = SyncServiceFactory.CreateSyncService();

var result = await syncService.SyncFullAsync(
	@"C:\System1\Data",
	@"C:\System2\Data",
	"Server=System1;Database=AppDB;",
	"Server=System2;Database=AppDB;",
	new[] { "Users", "Settings", "Documents" },
	options);
```

### Bidirectional Synchronization

```csharp
var options = new SyncOptions
{
	BidirectionalSync = true,
	ConflictResolutionStrategy = "LastWriteWins"
};

var result = await syncService.SyncBidirectionalAsync(
	@"C:\Location1",
	@"C:\Location2",
	options);
```

## Configuration

### FileSyncOptions

```csharp
var fileSyncOptions = new FileSyncOptions
{
	// Conflict resolution: "LastWriteWins", "SourceWins", "TargetWins", "Manual"
	ConflictResolutionStrategy = "LastWriteWins",

	// Delete files from target if they don't exist in source
	DeleteMissingFilesInTarget = false,

	// Validate data after sync
	ValidateAfterSync = true,

	// Maximum retry attempts for failed operations
	MaxRetries = 3,

	// Timeout in seconds
	TimeoutSeconds = 300,

	// File patterns to exclude (semicolon-separated)
	ExcludeFilePatterns = "*.tmp;*.temp;*.bak;*.lock",

	// Buffer size for file operations (bytes)
	BufferSizeBytes = 65536,

	// Preserve original file timestamps
	PreserveTimestamps = true,

	// Verify file integrity using SHA256
	VerifyFileIntegrity = true,

	// Compress files during transfer
	CompressTransfers = false,

	// Minimum file size for compression (bytes)
	CompressionThreshold = 1048576,

	// Chunk size for large file transfers (bytes)
	ChunkSize = 5242880 // 5MB
};

var syncService = SyncServiceFactory.CreateFileSyncService(fileSyncOptions);
```

### DatabaseSyncOptions

```csharp
var dbSyncOptions = new DatabaseSyncOptions
{
	// Use transactions for each batch
	UseTransactions = true,

	// Batch size for database operations
	DatabaseBatchSize = 1000,

	// Comma-separated list of entities to sync
	EntitiesToSync = "Users,Products,Orders",

	// Clear target database before sync
	ClearTargetBeforeSync = false,

	// Rebuild indexes after sync
	RebuildIndexesAfterSync = false,

	// Bypass foreign key constraints during sync
	BypassForeignKeyConstraints = false,

	// Conflict resolution strategy
	ConflictResolutionStrategy = "LastWriteWins"
};

var syncService = SyncServiceFactory.CreateDatabaseSyncService(dbSyncOptions);
```

## Advanced Usage

### Custom Notifications

```csharp
public class CustomSyncNotifier : ISyncNotifier
{
	public void NotifyProgress(string message, int percentage)
	{
		Console.WriteLine($"Progress: {percentage}% - {message}");
	}

	public void NotifyError(string errorMessage)
	{
		Console.WriteLine($"Error: {errorMessage}");
	}

	public void NotifyCompletion(SyncResult result)
	{
		Console.WriteLine($"Sync complete. Changes: {result.ChangesApplied}");
	}

	public void NotifyConflict(SyncConflict conflict)
	{
		Console.WriteLine($"Conflict detected in {conflict.EntityIdentifier}");
	}
}

var notifier = new CustomSyncNotifier();
var syncService = SyncServiceFactory.CreateSyncService(notifier);
```

### Session Management

```csharp
// Start a sync session
var session = await syncService.StartSyncSessionAsync("System1", "System2");
Console.WriteLine($"Session started: {session.SessionId}");

// Perform sync operations
var result = await syncService.SyncFilesAsync(source, target);

// Complete session
await syncService.CompleteSyncSessionAsync(session.SessionId);

// Get session status
var sessionStatus = await syncService.GetSyncSessionAsync(session.SessionId);
```

### Change Tracking

```csharp
// Create a file synchronizer
var fileSynchronizer = new FileSynchronizer();

// Detect changes since a specific time
var changes = await fileSynchronizer.DetectFileChangesAsync(
	@"C:\MyFolder", 
	DateTime.UtcNow.AddHours(-1));

foreach (var change in changes)
{
	Console.WriteLine($"Change: {change.SourceId} - {change.Type}");
}
```

### File Metadata

```csharp
var fileSynchronizer = new FileSynchronizer();

// Get file metadata
var metadata = await fileSynchronizer.GetFileMetadataAsync(@"C:\MyFile.dat");

Console.WriteLine($"File: {metadata.FilePath}");
Console.WriteLine($"Size: {metadata.FileSize} bytes");
Console.WriteLine($"Hash: {metadata.FileHash}");
Console.WriteLine($"Modified: {metadata.LastModified}");
```

### Custom Conflict Resolution

```csharp
public class CustomConflictResolver : IConflictResolver
{
	public async Task<object> ResolveConflictAsync(SyncConflict conflict)
	{
		// Custom logic to resolve conflicts
		if (conflict.SourceLastModified > conflict.TargetLastModified)
			return conflict.SourceValue;
		else
			return conflict.TargetValue;
	}

	public string GetResolutionStrategy(ChangeType changeType)
	{
		return "CustomLogic";
	}
}
```

## Error Handling

```csharp
try
{
	var result = await syncService.SyncFilesAsync(source, target);

	if (!result.Success)
	{
		foreach (var error in result.Errors)
		{
			Console.WriteLine($"Error: {error}");
		}
	}
}
catch (Exception ex)
{
	Console.WriteLine($"Sync failed: {ex.Message}");
}
```

## Best Practices

1. **Always validate connection strings** before starting database sync
2. **Use transactions** for database operations to ensure consistency
3. **Enable file integrity verification** for critical files
4. **Set appropriate timeouts** based on network conditions
5. **Monitor progress** for long-running syncs
6. **Test conflict resolution strategies** before production use
7. **Backup target systems** before destructive sync operations
8. **Use bidirectional sync cautiously** to avoid infinite loops